nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "lufeng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NUTCH-1647) protocol-http throws unzipBestEffort returned null for some pages
Date Wed, 01 Jan 2014 14:18:52 GMT

    [ https://issues.apache.org/jira/browse/NUTCH-1647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859880#comment-13859880
] 

lufeng commented on NUTCH-1647:
-------------------------------

This is cause by return content length is 0. and the unzipBestEffort method return null.

{code:java}
content = GZIPUtils.unzipBestEffort(compressed);
{code}

{code:bash}
lemo@debian:~/Workspace/java/apache-workspace/nutch-svn/runtime/local$ wget --verbose --server-response
http://www.provinciegroningen.nl/actueel/dossiers/rwe-centrale
--2014-01-01 21:47:06--  http://www.provinciegroningen.nl/actueel/dossiers/rwe-centrale
Resolving www.provinciegroningen.nl (www.provinciegroningen.nl)... 194.13.8.20
Connecting to www.provinciegroningen.nl (www.provinciegroningen.nl)|194.13.8.20|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 301 TYPO3 RealURL redirect
  Date: Wed, 01 Jan 2014 13:47:22 GMT
  Server: Apache
  Location: http://www.provinciegroningen.nl/actueel/dossiers/rwe-centrale/
  Cache-Control: max-age=3600
  Expires: Wed, 01 Jan 2014 14:47:22 GMT
  Vary: Accept-Encoding
  Content-Length: 0
  Content-Type: text/html; charset=UTF-8
  Connection: Keep-Alive
  Set-Cookie: fe_typo_user=56acbb2f413742a928a94ebf51a51bcd; path=/
  Age: 0
Location: http://www.provinciegroningen.nl/actueel/dossiers/rwe-centrale/ [following]
--2014-01-01 21:47:13--  http://www.provinciegroningen.nl/actueel/dossiers/rwe-centrale/
Reusing existing connection to www.provinciegroningen.nl:80.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Wed, 01 Jan 2014 13:47:22 GMT
  Server: Apache
  Cache-Control: max-age=3600
  Expires: Wed, 01 Jan 2014 14:47:22 GMT
  Vary: Accept-Encoding
  Transfer-Encoding: chunked
  Content-Type: text/html; charset=utf-8
  Connection: Keep-Alive
  Set-Cookie: fe_typo_user=29b705c75f2c6ff9cf495577efd727dd; path=/
  Age: 0
Length: unspecified [text/html]
Saving to: `rwe-centrale.2'

    [                                         <=>                                  
                              ] 51,728      2.92K/s   in 34s     

2014-01-01 21:47:48 (1.49 KB/s) - `rwe-centrale.2' saved [51728]
{code}

if you use httpclient protocol plugin and open follow redirects option, it will download the
page correctly.

{code:java}
lemo@debian:~/Workspace/java/apache-workspace/nutch-svn/runtime/local$ bin/nutch indexchecker
http://www.provinciegroningen.nl/actueel/dossiers/rwe-centrale
fetching: http://www.provinciegroningen.nl/actueel/dossiers/rwe-centrale
parsing: http://www.provinciegroningen.nl/actueel/dossiers/rwe-centrale
contentType: application/xhtml+xml
content :	Provincie Groningen: RWE-centrale Provincie Groningen  >  Actueel  > 
Dossiers  > RWE-centrale RWE-c
title :	Provincie Groningen: RWE-centrale
host :	www.provinciegroningen.nl
tstamp :	Wed Jan 01 22:03:40 CST 2014
url :	http://www.provinciegroningen.nl/actueel/dossiers/rwe-centrale
{code}

but this option is always false setting in HttpBase class.

{code:java}
  public ProtocolOutput getProtocolOutput(Text url, CrawlDatum datum) {
    
    String urlString = url.toString();
    try {
      URL u = new URL(urlString);
      Response response = getResponse(u, datum, false); // make a request
{code}

so current solution
1. get that option in Configuration file and get that option in getProtocolOuput interface

but for protocol-http plugin, we need to write some code to handler url redirect.

> protocol-http throws unzipBestEffort returned null for some pages
> -----------------------------------------------------------------
>
>                 Key: NUTCH-1647
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1647
>             Project: Nutch
>          Issue Type: Bug
>          Components: protocol
>    Affects Versions: 1.7
>            Reporter: Markus Jelsma
>             Fix For: 1.8
>
>
> bin/nutch indexchecker http://www.provinciegroningen.nl/actueel/dossiers/rwe-centrale
 
> Fetch failed with protocol status: exception(16), lastModified=0: java.io.IOException:
unzipBestEffort returned null
> {code}
> 2013-10-01 13:44:55,612 INFO  http.Http - http.proxy.host = null
> 2013-10-01 13:44:55,612 INFO  http.Http - http.proxy.port = 8080
> 2013-10-01 13:44:55,612 INFO  http.Http - http.timeout = 12000
> 2013-10-01 13:44:55,612 INFO  http.Http - http.content.limit = 5242880
> 2013-10-01 13:44:55,612 INFO  http.Http - http.agent = Mozilla/5.0 (compatible; OpenindexSpider;
+http://www.openindex.io/en/webmasters/spider.html)
> 2013-10-01 13:44:55,612 INFO  http.Http - http.accept.language = en-us,en-gb,en;q=0.7,*;q=0.3
> 2013-10-01 13:44:55,613 INFO  http.Http - http.accept = text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> 2013-10-01 13:44:55,925 ERROR http.Http - Failed to get protocol output
> java.io.IOException: unzipBestEffort returned null
>         at org.apache.nutch.protocol.http.api.HttpBase.processGzipEncoded(HttpBase.java:317)
>         at org.apache.nutch.protocol.http.HttpResponse.<init>(HttpResponse.java:164)
>         at org.apache.nutch.protocol.http.Http.getResponse(Http.java:64)
>         at org.apache.nutch.protocol.http.api.HttpBase.getProtocolOutput(HttpBase.java:140)
>         at org.apache.nutch.indexer.IndexingFiltersChecker.run(IndexingFiltersChecker.java:86)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>         at org.apache.nutch.indexer.IndexingFiltersChecker.main(IndexingFiltersChecker.java:150)
> {code}
> Haven't got a clue yet as to what the exact issue is.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message