lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yonik Seeley" <ysee...@gmail.com>
Subject Re: Doc add limit
Date Wed, 26 Jul 2006 17:42:07 GMT
So it looks like your client is hanging trying to send somethig over
the socket to the server and blocking... probably because Tomcat isn't
reading anything from the socket because it's busy trying to restart
the webapp.

What is the heap size of the server? try increasing it... maybe tomcat
could have detected low memory and tried to reload the webapp.

-Yonik

On 7/26/06, sangraal aiken <sangraal@gmail.com> wrote:
> Thanks for you help Yonik, I've responded to your questions below:
>
> On 7/26/06, Yonik Seeley <yseeley@gmail.com> wrote:
> >
> > It's possible it's not hanging, but just takes a long time on a
> > specific add.  This is because Lucene will occasionally merge
> > segments.  When very large segments are merged, it can take a long
> > time.
>
>
> I've left it running (hung) for up to a half hour at a time and I've
> verified that my cpu idles during the hang. I have witnessed much shorter
> hangs on the ramp up to my 6,144 limit but they have been more like 2 - 10
> seconds in length. Perhaps this is the Lucene merging you mentioned.
>
> In the log file, add commands are followed by the number of
> > milliseconds the operation took.  Next time Solr hangs, wait for a
> > number of minutes until you see the operation logged and note how long
> > it took.
>
>
> Here are the last 5 log entries before the hang the last one is doc #6,144.
> Also it looks like Tomcat is trying to redeploy the webapp those last tomcat
> entries repeat indefinitely every 10 seconds or so. Perhaps this is a Tomcat
> problem?
>
> Jul 26, 2006 1:25:28 PM org.apache.solr.core.SolrCore update
> INFO: add (id=110705) 0 36596
> Jul 26, 2006 1:25:28 PM org.apache.solr.core.SolrCore update
> INFO: add (id=110700) 0 36600
> Jul 26, 2006 1:25:28 PM org.apache.solr.core.SolrCore update
> INFO: add (id=110688) 0 36603
> Jul 26, 2006 1:25:28 PM org.apache.solr.core.SolrCore update
> INFO: add (id=110690) 0 36608
> Jul 26, 2006 1:25:28 PM org.apache.solr.core.SolrCore update
> INFO: add (id=110686) 0 36611
> Jul 26, 2006 1:25:36 PM org.apache.catalina.startup.HostConfigcheckResources
> FINE: Checking context[] redeploy resource /source/solr/apache-tomcat-5.5.17
> /webapps/ROOT
> Jul 26, 2006 1:25:36 PM org.apache.catalina.startup.HostConfigcheckResources
> FINE: Checking context[] redeploy resource /source/solr/apache-tomcat-5.5.17
> /webapps/ROOT/META-INF/context.xml
> Jul 26, 2006 1:25:36 PM org.apache.catalina.startup.HostConfigcheckResources
> FINE: Checking context[] reload resource /source/solr/apache-tomcat-5.5.17
> /webapps/ROOT/WEB-INF/web.xml
> Jul 26, 2006 1:25:36 PM org.apache.catalina.startup.HostConfigcheckResources
> FINE: Checking context[] reload resource /source/solr/apache-tomcat-5.5.17
> /webapps/ROOT/META-INF/context.xml
> Jul 26, 2006 1:25:36 PM org.apache.catalina.startup.HostConfigcheckResources
> FINE: Checking context[] reload resource /source/solr/apache-tomcat-5.5.17
> /conf/context.xml
>
> How many documents are in the index before you do a batch that causes
> > a hang?  Does it happen on the first batch?  If so, you might be
> > seeing some other bug.  What appserver are you using?  Do the admin
> > pages respond when you see this hang?  If so, what does a stack trace
> > look like?
>
>
> I actually don't think I had the problem on the first batch, in fact my
> first batch contained very close to 6,144 documents so perhaps there is a
> relation there. Right now, I'm adding to an index with close to 90,000
> documents in it.
> I'm running Tomcat 5.5.17 and the admin pages respond just fine when it's
> hung... I did a thread dump and this is the trace of my update:
>
> "http-8080-Processor25" Id=33 in RUNNABLE (running in native) total cpu
> time=6330.7360ms user time=5769.5920ms
>      at java.net.SocketOutputStream.socketWrite0(Native Method)
>      at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
>      at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
>      at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
>      at java.io.PrintStream.write(PrintStream.java:412)
>      at java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java
> :112)
>      at sun.net.www.http.HttpClient.writeRequests(HttpClient.java:533)
>      at sun.net.www.protocol.http.HttpURLConnection.writeRequests(
> HttpURLConnection.java:410)
>      at sun.net.www.protocol.http.HttpURLConnection.getInputStream(
> HttpURLConnection.java:934)
>      at com.gawker.solr.update.GanjaUpdate.doUpdate(GanjaUpdate.java:169)
>      at com.gawker.solr.update.GanjaUpdate.update(GanjaUpdate.java:62)
>      at org.apache.jsp.update_jsp._jspService(update_jsp.java:57)
>      at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
>      at org.apache.jasper.servlet.JspServletWrapper.service(
> JspServletWrapper.java:332)
>      at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java
> :314)
>      at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)
>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
>      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
> ApplicationFilterChain.java:252)
>      at org.apache.catalina.core.ApplicationFilterChain.doFilter(
> ApplicationFilterChain.java:173)
>      at org.apache.catalina.core.StandardWrapperValve.invoke(
> StandardWrapperValve.java:213)
>      at org.apache.catalina.core.StandardContextValve.invoke(
> StandardContextValve.java:178)
>      at org.apache.catalina.core.StandardHostValve.invoke(
> StandardHostValve.java:126)
>      at org.apache.catalina.valves.ErrorReportValve.invoke(
> ErrorReportValve.java:105)
>      at org.apache.catalina.core.StandardEngineValve.invoke(
> StandardEngineValve.java:107)
>      at org.apache.catalina.connector.CoyoteAdapter.service(
> CoyoteAdapter.java:148)
>      at org.apache.coyote.http11.Http11Processor.process(
> Http11Processor.java:869)
>      at
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection
> (Http11BaseProtocol.java:664)
>      at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(
> PoolTcpEndpoint.java:527)
>      at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(
> LeaderFollowerWorkerThread.java:80)
>      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(
> ThreadPool.java:684)
>      at java.lang.Thread.run(Thread.java:613)
>
>
> -Yonik
> >
> >
> > On 7/26/06, sangraal aiken <sangraal@gmail.com> wrote:
> > > Hey there... I'm having an issue with large doc updates on my solr
> > > installation. I'm adding in batches between 2-20,000 docs at a time and
> > I've
> > > noticed Solr seems to hang at 6,144 docs every time. Breaking the adds
> > into
> > > smaller batches works just fine, but I was wondering if anyone knew why
> > this
> > > would happen. I've tried doubling memory as well as tweaking various
> > config
> > > options but nothing seems to let me break the 6,144 barrier.
> > >
> > > This is the output from Solr admin. Any help would be greatly
> > appreciated.
> > >
> > >
> > > *name: * updateHandler  *class: *
> > > org.apache.solr.update.DirectUpdateHandler2  *version: * 1.0
> >   *description:
> > > * Update handler that efficiently directly updates the on-disk main
> > lucene
> > > index  *stats: *commits : 0
> > > optimizes : 0
> > > docsPending : 6144
> > > deletesPending : 6144
> > > adds : 6144
> > > deletesById : 0
> > > deletesByQuery : 0
> > > errors : 0
> > > cumulative_adds : 6144
> > > cumulative_deletesById : 0
> > > cumulative_deletesByQuery : 0
> > > cumulative_errors : 0
> > > docsDeleted : 0

Mime
View raw message