nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ferdy Galema (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NUTCH-1342) Read time out protocol-http
Date Wed, 13 Jun 2012 13:00:43 GMT

    [ https://issues.apache.org/jira/browse/NUTCH-1342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294429#comment-13294429
] 

Ferdy Galema commented on NUTCH-1342:
-------------------------------------

Do you have any clue as to why protocol-httpclient has a different behaviour?

Also, two suggestions for your patch:

Perhaps you could finegrain the mechanism by allowing a configurable amount of timeouts before
definitely failing. Something like:
if (++timeoutRetries>this.allowedNumberOfTimeoutRetries) throw e; //rethrow

Secondly, could you specifically catch SocketTimeoutException? (I'm not sure if there are
other IOExceptions that shouldn't be catched in any case.)
                
> Read time out protocol-http
> ---------------------------
>
>                 Key: NUTCH-1342
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1342
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher
>    Affects Versions: 1.4, 1.5
>            Reporter: Markus Jelsma
>            Assignee: Markus Jelsma
>            Priority: Critical
>             Fix For: 1.6
>
>         Attachments: NUTCH-1342-1.6-1.patch
>
>
> For some reason some URL's always time out with protocol-http but not protocol-httpclient.
The stack trace is always the same:
> {code}
> 2012-04-20 11:25:44,275 ERROR http.Http - Failed to get protocol output
> java.net.SocketTimeoutException: Read timed out
>         at java.net.SocketInputStream.socketRead0(Native Method)
>         at java.net.SocketInputStream.read(SocketInputStream.java:129)
>         at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
>         at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>         at java.io.FilterInputStream.read(FilterInputStream.java:116)
>         at java.io.PushbackInputStream.read(PushbackInputStream.java:169)
>         at java.io.FilterInputStream.read(FilterInputStream.java:90)
>         at org.apache.nutch.protocol.http.HttpResponse.readPlainContent(HttpResponse.java:228)
>         at org.apache.nutch.protocol.http.HttpResponse.<init>(HttpResponse.java:157)
>         at org.apache.nutch.protocol.http.Http.getResponse(Http.java:64)
>         at org.apache.nutch.protocol.http.api.HttpBase.getProtocolOutput(HttpBase.java:138)
> {code}
> Some example URL's:
> * 404 http://www.fcgroningen.nl/tribunenamen/stemmen/
> * 301 http://shop.fcgroningen.nl/aanbieding

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message