lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From AlexxelA <alexandre.boudrea...@canoe.ca>
Subject Re: Solr jam after all my jvm thread pool hang in blocked state
Date Tue, 24 Aug 2010 13:41:41 GMT

Thread dump 

Got like 240 thread like this :


"http-8080-Processor222" daemon prio=10 tid=0x00007fe36c010c00 nid=0x1e94
waiting for monitor entry [0x000000004caa6000..0x000000004caa6d20]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at java.util.logging.StreamHandler.publish(StreamHandler.java:174)
	- waiting to lock <0x00007fe37e72b340> (a java.util.logging.ConsoleHandler)
	at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:88)
	at java.util.logging.Logger.log(Logger.java:472)
	at java.util.logging.Logger.doLog(Logger.java:494)
	at java.util.logging.Logger.log(Logger.java:517)
	at java.util.logging.Logger.info(Logger.java:1036)
	at org.apache.solr.core.SolrCore.execute(SolrCore.java:1212)
	at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
	at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
	at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
	at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
	at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
	at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
	at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
	at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
	at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)
	at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
	at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
	at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
	at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
	at java.lang.Thread.run(Thread.java:619)


Here the running thread : 

"http-8080-Processor156" daemon prio=10 tid=0x0000000000df2000 nid=0x1e52
runnable [0x0000000044521000..0x0000000044521c20]
   java.lang.Thread.State: RUNNABLE
	at java.io.FileOutputStream.writeBytes(Native Method)
	at java.io.FileOutputStream.write(FileOutputStream.java:260)
	at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
	- locked <0x00007fe37e3abcd8> (a java.io.BufferedOutputStream)
	at java.io.PrintStream.write(PrintStream.java:430)
	- locked <0x00007fe37e3abca0> (a java.io.PrintStream)
	at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
	at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)
	at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:276)
	at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122)
	- locked <0x00007fe37e72cd90> (a java.io.OutputStreamWriter)
	at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)
	at java.util.logging.StreamHandler.flush(StreamHandler.java:225)
	- locked <0x00007fe37e72b340> (a java.util.logging.ConsoleHandler)
	at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:89)
	at java.util.logging.Logger.log(Logger.java:472)
	at java.util.logging.Logger.doLog(Logger.java:494)
	at java.util.logging.Logger.log(Logger.java:517)
	at java.util.logging.Logger.info(Logger.java:1036)
	at org.apache.solr.core.SolrCore.execute(SolrCore.java:1212)
	at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
	at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
	at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
	at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
	at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
	at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
	at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
	at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
	at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)
	at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
	at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
	at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
	at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
	at java.lang.Thread.run(Thread.java:619)

And the system thread

"ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=10
tid=0x0000000000b1c800 nid=0x192 waiting on condition
[0x000000004183b000..0x000000004183bba0]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1548)
	at java.lang.Thread.run(Thread.java:619)

"pool-1-thread-1" prio=10 tid=0x00007fe374220400 nid=0x191 waiting for
monitor entry [0x0000000040bca000..0x0000000040bcaa20]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at java.util.logging.StreamHandler.publish(StreamHandler.java:174)
	- waiting to lock <0x00007fe37e72b340> (a java.util.logging.ConsoleHandler)
	at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:88)
	at java.util.logging.Logger.log(Logger.java:472)
	at java.util.logging.Logger.doLog(Logger.java:494)
	at java.util.logging.Logger.log(Logger.java:517)
	at java.util.logging.Logger.info(Logger.java:1036)
	at
org.apache.solr.search.SolrIndexSearcher.warm(SolrIndexSearcher.java:1519)
	at org.apache.solr.core.SolrCore$3.call(SolrCore.java:1018)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
	at java.lang.Thread.run(Thread.java:619)

"Low Memory Detector" daemon prio=10 tid=0x00007fe374005000 nid=0x18e
runnable [0x0000000000000000..0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=10 tid=0x00007fe374002800 nid=0x18d waiting on
condition [0x0000000000000000..0x00000000403684d0]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x00007fe374000c00 nid=0x18c waiting on
condition [0x0000000000000000..0x0000000040267540]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x00000000006bb000 nid=0x18b runnable
[0x0000000000000000..0x00000000401677b0]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x0000000000690c00 nid=0x18a in Object.wait()
[0x000000004152d000..0x000000004152dba0]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00007fe37e3a21d0> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
	- locked <0x00007fe37e3a21d0> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x000000000068f400 nid=0x189 in
Object.wait() [0x000000004142c000..0x000000004142ca20]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00007fe37e3a2208> (a java.lang.ref.Reference$Lock)
	at java.lang.Object.wait(Object.java:485)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
	- locked <0x00007fe37e3a2208> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x0000000000622400 nid=0x16e runnable
[0x0000000000000000..0x00007fffcd94d9b0]
   java.lang.Thread.State: RUNNABLE

"VM Thread" prio=10 tid=0x000000000068a000 nid=0x188 runnable 

"GC task thread#0 (ParallelGC)" prio=10 tid=0x000000000062cc00 nid=0x178
runnable 

"GC task thread#1 (ParallelGC)" prio=10 tid=0x000000000062e000 nid=0x179
runnable 

"GC task thread#2 (ParallelGC)" prio=10 tid=0x000000000062f000 nid=0x17a
runnable 

"GC task thread#3 (ParallelGC)" prio=10 tid=0x0000000000630400 nid=0x17b
runnable 

"GC task thread#4 (ParallelGC)" prio=10 tid=0x0000000000631800 nid=0x17c
runnable 

"GC task thread#5 (ParallelGC)" prio=10 tid=0x0000000000632c00 nid=0x17e
runnable 

"GC task thread#6 (ParallelGC)" prio=10 tid=0x0000000000634000 nid=0x17f
runnable 

"GC task thread#7 (ParallelGC)" prio=10 tid=0x0000000000635400 nid=0x180
runnable 

"VM Periodic Task Thread" prio=10 tid=0x00007fe374007400 nid=0x18f waiting
on condition 

JNI global references: 1151


-- 
View this message in context: http://lucene.472066.n3.nabble.com/Solr-jam-after-all-my-jvm-thread-pool-hang-in-blocked-state-tp1303361p1311364.html
Sent from the Solr - User mailing list archive at Nabble.com.

Mime
View raw message