lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "sangraal aiken" <sangr...@gmail.com>
Subject Re: Doc add limit
Date Wed, 26 Jul 2006 19:01:57 GMT
I removed everything from the Add xml so the docs looked like this:

<doc>
<field name="id">187880</field>
</doc>
<doc>
<field name="id">187852</field>
</doc>

and it still hung at 6,144...

-S


On 7/26/06, Yonik Seeley <yseeley@gmail.com> wrote:
>
> If you narrow the docs down to just the "id" field, does it still
> happen at the same place?
>
> -Yonik
>
> On 7/26/06, sangraal aiken <sangraal@gmail.com> wrote:
> > I see the problem on Mac OS X/JDK: 1.5.0_06 and Debian/JDK: 1.5.0_07.
> >
> > I don't think it's a socket problem, because I can initiate additional
> > updates while the server is hung... weird I know.
> >
> > Thanks for all your help, I'll send a post if/when I find a solution.
> >
> > -S
> >
> > On 7/26/06, Yonik Seeley <yseeley@gmail.com> wrote:
> > >
> > > Tomcat problem, or a Solr problem that is only manifesting on your
> > > platform, or a JVM or libc problem, or even a client update problem...
> > > (possibly you might be exhausting the number of sockets in the server
> > > by using persistent connections with a long timeout and never reusing
> > > them?)
> > >
> > > What is your OS/JVM?
> > >
> > > -Yonik
> > >
> > > On 7/26/06, sangraal aiken <sangraal@gmail.com> wrote:
> > > > Right now the heap is set to 512M but I've increased it up to 2GB
> and
> > > yet it
> > > > still hangs at the same number 6,144...
> > > >
> > > > Here's something interesting... I pushed this code over to a
> different
> > > > server and tried an update. On that server it's hanging on #5,267.
> Then
> > > > tomcat seems to try to reload the webapp... indefinitely.
> > > >
> > > > So I guess this is looking more like a tomcat problem more than a
> > > > lucene/solr problem huh?
> > > >
> > > > -Sangraal
> > > >
> > > > On 7/26/06, Yonik Seeley <yseeley@gmail.com> wrote:
> > > > >
> > > > > So it looks like your client is hanging trying to send somethig
> over
> > > > > the socket to the server and blocking... probably because Tomcat
> isn't
> > > > > reading anything from the socket because it's busy trying to
> restart
> > > > > the webapp.
> > > > >
> > > > > What is the heap size of the server? try increasing it... maybe
> tomcat
> > > > > could have detected low memory and tried to reload the webapp.
> > > > >
> > > > > -Yonik
> > > > >
> > > > > On 7/26/06, sangraal aiken <sangraal@gmail.com> wrote:
> > > > > > Thanks for you help Yonik, I've responded to your questions
> below:
> > > > > >
> > > > > > On 7/26/06, Yonik Seeley <yseeley@gmail.com> wrote:
> > > > > > >
> > > > > > > It's possible it's not hanging, but just takes a long time
on
> a
> > > > > > > specific add.  This is because Lucene will occasionally
merge
> > > > > > > segments.  When very large segments are merged, it can
take a
> long
> > > > > > > time.
> > > > > >
> > > > > >
> > > > > > I've left it running (hung) for up to a half hour at a time
and
> I've
> > > > > > verified that my cpu idles during the hang. I have witnessed
> much
> > > > > shorter
> > > > > > hangs on the ramp up to my 6,144 limit but they have been more
> like
> > > 2 -
> > > > > 10
> > > > > > seconds in length. Perhaps this is the Lucene merging you
> mentioned.
> > > > > >
> > > > > > In the log file, add commands are followed by the number of
> > > > > > > milliseconds the operation took.  Next time Solr hangs,
wait
> for a
> > > > > > > number of minutes until you see the operation logged and
note
> how
> > > long
> > > > > > > it took.
> > > > > >
> > > > > >
> > > > > > Here are the last 5 log entries before the hang the last one
is
> doc
> > > > > #6,144.
> > > > > > Also it looks like Tomcat is trying to redeploy the webapp those
> > > last
> > > > > tomcat
> > > > > > entries repeat indefinitely every 10 seconds or so. Perhaps
this
> is
> > > a
> > > > > Tomcat
> > > > > > problem?
> > > > > >
> > > > > > Jul 26, 2006 1:25:28 PM org.apache.solr.core.SolrCore update
> > > > > > INFO: add (id=110705) 0 36596
> > > > > > Jul 26, 2006 1:25:28 PM org.apache.solr.core.SolrCore update
> > > > > > INFO: add (id=110700) 0 36600
> > > > > > Jul 26, 2006 1:25:28 PM org.apache.solr.core.SolrCore update
> > > > > > INFO: add (id=110688) 0 36603
> > > > > > Jul 26, 2006 1:25:28 PM org.apache.solr.core.SolrCore update
> > > > > > INFO: add (id=110690) 0 36608
> > > > > > Jul 26, 2006 1:25:28 PM org.apache.solr.core.SolrCore update
> > > > > > INFO: add (id=110686) 0 36611
> > > > > > Jul 26, 2006 1:25:36 PM
> > > > > org.apache.catalina.startup.HostConfigcheckResources
> > > > > > FINE: Checking context[] redeploy resource /source/solr/apache-
> > > > > tomcat-5.5.17
> > > > > > /webapps/ROOT
> > > > > > Jul 26, 2006 1:25:36 PM
> > > > > org.apache.catalina.startup.HostConfigcheckResources
> > > > > > FINE: Checking context[] redeploy resource /source/solr/apache-
> > > > > tomcat-5.5.17
> > > > > > /webapps/ROOT/META-INF/context.xml
> > > > > > Jul 26, 2006 1:25:36 PM
> > > > > org.apache.catalina.startup.HostConfigcheckResources
> > > > > > FINE: Checking context[] reload resource /source/solr/apache-
> > > > > tomcat-5.5.17
> > > > > > /webapps/ROOT/WEB-INF/web.xml
> > > > > > Jul 26, 2006 1:25:36 PM
> > > > > org.apache.catalina.startup.HostConfigcheckResources
> > > > > > FINE: Checking context[] reload resource /source/solr/apache-
> > > > > tomcat-5.5.17
> > > > > > /webapps/ROOT/META-INF/context.xml
> > > > > > Jul 26, 2006 1:25:36 PM
> > > > > org.apache.catalina.startup.HostConfigcheckResources
> > > > > > FINE: Checking context[] reload resource /source/solr/apache-
> > > > > tomcat-5.5.17
> > > > > > /conf/context.xml
> > > > > >
> > > > > > How many documents are in the index before you do a batch that
> > > causes
> > > > > > > a hang?  Does it happen on the first batch?  If so, you
might
> be
> > > > > > > seeing some other bug.  What appserver are you using? 
Do the
> > > admin
> > > > > > > pages respond when you see this hang?  If so, what does
a
> stack
> > > trace
> > > > > > > look like?
> > > > > >
> > > > > >
> > > > > > I actually don't think I had the problem on the first batch,
in
> fact
> > > my
> > > > > > first batch contained very close to 6,144 documents so perhaps
> there
> > > is
> > > > > a
> > > > > > relation there. Right now, I'm adding to an index with close
to
> > > 90,000
> > > > > > documents in it.
> > > > > > I'm running Tomcat 5.5.17 and the admin pages respond just fine
> when
> > > > > it's
> > > > > > hung... I did a thread dump and this is the trace of my update:
> > > > > >
> > > > > > "http-8080-Processor25" Id=33 in RUNNABLE (running in native)
> total
> > > cpu
> > > > > > time=6330.7360ms user time=5769.5920ms
> > > > > >      at java.net.SocketOutputStream.socketWrite0(Native Method)
> > > > > >      at java.net.SocketOutputStream.socketWrite(
> > > SocketOutputStream.java
> > > > > :92)
> > > > > >      at java.net.SocketOutputStream.write(
> SocketOutputStream.java
> > > :136)
> > > > > >      at java.io.BufferedOutputStream.write(
> BufferedOutputStream.java
> > > > > :105)
> > > > > >      at java.io.PrintStream.write(PrintStream.java:412)
> > > > > >      at java.io.ByteArrayOutputStream.writeTo(
> > > ByteArrayOutputStream.java
> > > > > > :112)
> > > > > >      at sun.net.www.http.HttpClient.writeRequests(
> HttpClient.java
> > > :533)
> > > > > >      at
> sun.net.www.protocol.http.HttpURLConnection.writeRequests(
> > > > > > HttpURLConnection.java:410)
> > > > > >      at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(
> > > > > > HttpURLConnection.java:934)
> > > > > >      at com.gawker.solr.update.GanjaUpdate.doUpdate(
> GanjaUpdate.java
> > > > > :169)
> > > > > >      at com.gawker.solr.update.GanjaUpdate.update(
> GanjaUpdate.java
> > > :62)
> > > > > >      at org.apache.jsp.update_jsp._jspService
> (update_jsp.java:57)
> > > > > >      at org.apache.jasper.runtime.HttpJspBase.service(
> > > HttpJspBase.java
> > > > > :97)
> > > > > >      at javax.servlet.http.HttpServlet.service(HttpServlet.java
> :802)
> > > > > >      at org.apache.jasper.servlet.JspServletWrapper.service(
> > > > > > JspServletWrapper.java:332)
> > > > > >      at org.apache.jasper.servlet.JspServlet.serviceJspFile(
> > > > > JspServlet.java
> > > > > > :314)
> > > > > >      at org.apache.jasper.servlet.JspServlet.service(
> JspServlet.java
> > > > > :264)
> > > > > >      at javax.servlet.http.HttpServlet.service(HttpServlet.java
> :802)
> > > > > >      at
> > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter
> > > > > (
> > > > > > ApplicationFilterChain.java:252)
> > > > > >      at org.apache.catalina.core.ApplicationFilterChain.doFilter
> (
> > > > > > ApplicationFilterChain.java:173)
> > > > > >      at org.apache.catalina.core.StandardWrapperValve.invoke(
> > > > > > StandardWrapperValve.java:213)
> > > > > >      at org.apache.catalina.core.StandardContextValve.invoke(
> > > > > > StandardContextValve.java:178)
> > > > > >      at org.apache.catalina.core.StandardHostValve.invoke(
> > > > > > StandardHostValve.java:126)
> > > > > >      at org.apache.catalina.valves.ErrorReportValve.invoke(
> > > > > > ErrorReportValve.java:105)
> > > > > >      at org.apache.catalina.core.StandardEngineValve.invoke(
> > > > > > StandardEngineValve.java:107)
> > > > > >      at org.apache.catalina.connector.CoyoteAdapter.service(
> > > > > > CoyoteAdapter.java:148)
> > > > > >      at org.apache.coyote.http11.Http11Processor.process(
> > > > > > Http11Processor.java:869)
> > > > > >      at
> > > > > >
> > > > >
> > >
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection
> > > > > > (Http11BaseProtocol.java:664)
> > > > > >      at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket
> (
> > > > > > PoolTcpEndpoint.java:527)
> > > > > >      at
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(
> > > > > > LeaderFollowerWorkerThread.java:80)
> > > > > >      at
> > > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(
> > > > > > ThreadPool.java:684)
> > > > > >      at java.lang.Thread.run(Thread.java:613)
> > > > > >
> > > > > >
> > > > > > -Yonik
> > > > > > >
> > > > > > >
> > > > > > > On 7/26/06, sangraal aiken <sangraal@gmail.com> wrote:
> > > > > > > > Hey there... I'm having an issue with large doc updates
on
> my
> > > solr
> > > > > > > > installation. I'm adding in batches between 2-20,000
docs at
> a
> > > time
> > > > > and
> > > > > > > I've
> > > > > > > > noticed Solr seems to hang at 6,144 docs every time.
> Breaking
> > > the
> > > > > adds
> > > > > > > into
> > > > > > > > smaller batches works just fine, but I was wondering
if
> anyone
> > > knew
> > > > > why
> > > > > > > this
> > > > > > > > would happen. I've tried doubling memory as well as
tweaking
> > > various
> > > > > > > config
> > > > > > > > options but nothing seems to let me break the 6,144
barrier.
> > > > > > > >
> > > > > > > > This is the output from Solr admin. Any help would
be
> greatly
> > > > > > > appreciated.
> > > > > > > >
> > > > > > > >
> > > > > > > > *name: * updateHandler  *class: *
> > > > > > > > org.apache.solr.update.DirectUpdateHandler2  *version:
* 1.0
> > > > > > >   *description:
> > > > > > > > * Update handler that efficiently directly updates
the
> on-disk
> > > main
> > > > > > > lucene
> > > > > > > > index  *stats: *commits : 0
> > > > > > > > optimizes : 0
> > > > > > > > docsPending : 6144
> > > > > > > > deletesPending : 6144
> > > > > > > > adds : 6144
> > > > > > > > deletesById : 0
> > > > > > > > deletesByQuery : 0
> > > > > > > > errors : 0
> > > > > > > > cumulative_adds : 6144
> > > > > > > > cumulative_deletesById : 0
> > > > > > > > cumulative_deletesByQuery : 0
> > > > > > > > cumulative_errors : 0
> > > > > > > > docsDeleted : 0
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message