lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Solr jam after all my jvm thread pool hang in blocked state
Date Wed, 25 Aug 2010 01:17:47 GMT
Has anyone changed your tomcat settings? You're logging
information at the INFO level, I wonder if you used to be
logging at WARN.

I'd also take a look at your log directory to see if you've got
a bazillion log files, and/or how big your log file is, has it
been accumulating log messages forever? In other words,
are you absolutely sure that your log file handling is correct?

Take a look at how much free disk space you have at this
point. Like I said above, this is a long shot but often one
of the things that doesn't show up for a very long time on
a production system.

And why are you logging to console anyway? Has that
changed from file-only?

Or am I on a completely wrong tangent???

Best
Erick

On Tue, Aug 24, 2010 at 6:41 AM, AlexxelA <alexandre.boudreault@canoe.ca>wrote:

>
> Thread dump
>
> Got like 240 thread like this :
>
>
> "http-8080-Processor222" daemon prio=10 tid=0x00007fe36c010c00 nid=0x1e94
> waiting for monitor entry [0x000000004caa6000..0x000000004caa6d20]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at java.util.logging.StreamHandler.publish(StreamHandler.java:174)
>        - waiting to lock <0x00007fe37e72b340> (a
> java.util.logging.ConsoleHandler)
>        at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:88)
>        at java.util.logging.Logger.log(Logger.java:472)
>        at java.util.logging.Logger.doLog(Logger.java:494)
>        at java.util.logging.Logger.log(Logger.java:517)
>        at java.util.logging.Logger.info(Logger.java:1036)
>        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1212)
>        at
>
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
>        at
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
>        at
>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
>        at
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
>        at
>
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
>        at
>
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
>        at
>
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>        at
>
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
>        at
>
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
>        at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
>        at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)
>        at
>
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
>        at
>
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
>        at
>
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
>        at
>
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
>        at java.lang.Thread.run(Thread.java:619)
>
>
> Here the running thread :
>
> "http-8080-Processor156" daemon prio=10 tid=0x0000000000df2000 nid=0x1e52
> runnable [0x0000000044521000..0x0000000044521c20]
>   java.lang.Thread.State: RUNNABLE
>        at java.io.FileOutputStream.writeBytes(Native Method)
>        at java.io.FileOutputStream.write(FileOutputStream.java:260)
>        at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
>        - locked <0x00007fe37e3abcd8> (a java.io.BufferedOutputStream)
>        at java.io.PrintStream.write(PrintStream.java:430)
>        - locked <0x00007fe37e3abca0> (a java.io.PrintStream)
>        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
>        at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)
>        at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:276)
>        at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122)
>        - locked <0x00007fe37e72cd90> (a java.io.OutputStreamWriter)
>        at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)
>        at java.util.logging.StreamHandler.flush(StreamHandler.java:225)
>        - locked <0x00007fe37e72b340> (a java.util.logging.ConsoleHandler)
>        at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:89)
>        at java.util.logging.Logger.log(Logger.java:472)
>        at java.util.logging.Logger.doLog(Logger.java:494)
>        at java.util.logging.Logger.log(Logger.java:517)
>        at java.util.logging.Logger.info(Logger.java:1036)
>        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1212)
>        at
>
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
>        at
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
>        at
>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
>        at
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
>        at
>
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
>        at
>
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
>        at
>
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>        at
>
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
>        at
>
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
>        at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
>        at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)
>        at
>
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
>        at
>
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
>        at
>
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
>        at
>
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
>        at java.lang.Thread.run(Thread.java:619)
>
> And the system thread
>
> "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=10
> tid=0x0000000000b1c800 nid=0x192 waiting on condition
> [0x000000004183b000..0x000000004183bba0]
>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>        at java.lang.Thread.sleep(Native Method)
>        at
>
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1548)
>        at java.lang.Thread.run(Thread.java:619)
>
> "pool-1-thread-1" prio=10 tid=0x00007fe374220400 nid=0x191 waiting for
> monitor entry [0x0000000040bca000..0x0000000040bcaa20]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at java.util.logging.StreamHandler.publish(StreamHandler.java:174)
>        - waiting to lock <0x00007fe37e72b340> (a
> java.util.logging.ConsoleHandler)
>        at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:88)
>        at java.util.logging.Logger.log(Logger.java:472)
>        at java.util.logging.Logger.doLog(Logger.java:494)
>        at java.util.logging.Logger.log(Logger.java:517)
>        at java.util.logging.Logger.info(Logger.java:1036)
>        at
> org.apache.solr.search.SolrIndexSearcher.warm(SolrIndexSearcher.java:1519)
>        at org.apache.solr.core.SolrCore$3.call(SolrCore.java:1018)
>        at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>        at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
>        at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>        at java.lang.Thread.run(Thread.java:619)
>
> "Low Memory Detector" daemon prio=10 tid=0x00007fe374005000 nid=0x18e
> runnable [0x0000000000000000..0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
> "CompilerThread1" daemon prio=10 tid=0x00007fe374002800 nid=0x18d waiting
> on
> condition [0x0000000000000000..0x00000000403684d0]
>   java.lang.Thread.State: RUNNABLE
>
> "CompilerThread0" daemon prio=10 tid=0x00007fe374000c00 nid=0x18c waiting
> on
> condition [0x0000000000000000..0x0000000040267540]
>   java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=10 tid=0x00000000006bb000 nid=0x18b
> runnable
> [0x0000000000000000..0x00000000401677b0]
>   java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=10 tid=0x0000000000690c00 nid=0x18a in
> Object.wait()
> [0x000000004152d000..0x000000004152dba0]
>   java.lang.Thread.State: WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0x00007fe37e3a21d0> (a
> java.lang.ref.ReferenceQueue$Lock)
>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>        - locked <0x00007fe37e3a21d0> (a java.lang.ref.ReferenceQueue$Lock)
>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> "Reference Handler" daemon prio=10 tid=0x000000000068f400 nid=0x189 in
> Object.wait() [0x000000004142c000..0x000000004142ca20]
>   java.lang.Thread.State: WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0x00007fe37e3a2208> (a java.lang.ref.Reference$Lock)
>        at java.lang.Object.wait(Object.java:485)
>        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>        - locked <0x00007fe37e3a2208> (a java.lang.ref.Reference$Lock)
>
> "main" prio=10 tid=0x0000000000622400 nid=0x16e runnable
> [0x0000000000000000..0x00007fffcd94d9b0]
>   java.lang.Thread.State: RUNNABLE
>
> "VM Thread" prio=10 tid=0x000000000068a000 nid=0x188 runnable
>
> "GC task thread#0 (ParallelGC)" prio=10 tid=0x000000000062cc00 nid=0x178
> runnable
>
> "GC task thread#1 (ParallelGC)" prio=10 tid=0x000000000062e000 nid=0x179
> runnable
>
> "GC task thread#2 (ParallelGC)" prio=10 tid=0x000000000062f000 nid=0x17a
> runnable
>
> "GC task thread#3 (ParallelGC)" prio=10 tid=0x0000000000630400 nid=0x17b
> runnable
>
> "GC task thread#4 (ParallelGC)" prio=10 tid=0x0000000000631800 nid=0x17c
> runnable
>
> "GC task thread#5 (ParallelGC)" prio=10 tid=0x0000000000632c00 nid=0x17e
> runnable
>
> "GC task thread#6 (ParallelGC)" prio=10 tid=0x0000000000634000 nid=0x17f
> runnable
>
> "GC task thread#7 (ParallelGC)" prio=10 tid=0x0000000000635400 nid=0x180
> runnable
>
> "VM Periodic Task Thread" prio=10 tid=0x00007fe374007400 nid=0x18f waiting
> on condition
>
> JNI global references: 1151
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Solr-jam-after-all-my-jvm-thread-pool-hang-in-blocked-state-tp1303361p1311364.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message