lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Read Time Out Exception while trying to upload a huge SOLR input xml
Date Thu, 01 Apr 2010 16:10:02 GMT
Don't do that. For many reasons <G>. By trying to batch so many docs
together, you're just *asking* for trouble. Quite apart from whether it'll
work once, having *any* HTTP-based protocol work reliably with 13G is
fragile...

For instance, I don't want to have my know whether the XML parsing in
SOLR parses the entire document into memory before processing or
not. But I sure don't want my application to change behavior if SOLR
changes it's mind and wants to process the other way. My perfectly
working application (assuming an event-driven parser) could
suddenly start requiring over 13G of memory... Oh my aching head!

Your specific error might even be dependent upon GCing, which will
cause it to break differently, sometimes, maybe......

So do break things up and transmit multiple documents. It'll save you
a world of hurt.

HTH
Erick

On Thu, Apr 1, 2010 at 4:34 AM, Mark Fletcher
<mark.fletcher2004@gmail.com>wrote:

> Hi,
>
> For the first time I tried uploading a huge input SOLR xml having about 1.2
> million *docs* (13GB in size). After some time I get the following
> exception:-
>
>  <u>The server encountered an internal error ([was class
> java.net.SocketTimeoutException] Read timed out
> java.lang.RuntimeException: [was class java.net.SocketTimeoutException]
> Read
> timed out
>  at
>
> com.ctc.wstx.util.ExceptionUtil.throwRuntimeException(ExceptionUtil.java:18)
>  at com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:731)
>  at
>
> com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3657)
>  at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:809)
>  at org.apache.solr.handler.XMLLoader.readDoc(XMLLoader.java:279)
>  at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:138)
>  at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
>  at
>
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
>  at
>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>  at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
>  at
>
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
>  at
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
>  at
>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>  at
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>  at
>
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>  at
>
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>  at
>
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
>  at
>
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>  at
>
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>  at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
>  at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
>  at
>
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
>  at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
>  at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.SocketTimeoutException: Read timed out
> ...
>
> Was the file I tried to upload too big and should I try reducing its
> size..?
>
> Thanks and Rgds,
> Mark.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message