lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fuad Efendi <f...@efendi.ca>
Subject Re: Solr stops responding
Date Tue, 15 Jul 2008 16:17:35 GMT
Following lines are strange, looks like SOLR deals with OOM and  
rethrows own exception (so that in some cases JVM simply hangs instead  
of exit):
   Apr 4, 2008 1:20:53 PM org.apache.solr.common.SolrException log
   SEVERE: java.lang.OutOfMemoryError: Java heap space



This is full Thread Dump after OOM, made in April with Tomcat 6.  
Deadlock at Tomcat? Looks like some queries succeed, but I was forced  
KILL -9.
=================


SEVERE: Error allocating socket processor
java.lang.OutOfMemoryError: Java heap space
Apr 4, 2008 1:57:36 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space

Exception in thread "catalina-exec-4" java.lang.OutOfMemoryError: Java  
heap space
Apr 4, 2008 1:58:18 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space

Apr 4, 2008 1:59:01 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space

Apr 4, 2008 1:59:01 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space

Apr 4, 2008 1:59:01 PM org.apache.tomcat.util.net.AprEndpoint$Acceptor run
SEVERE: Socket accept failed
java.lang.OutOfMemoryError: Java heap space
Apr 4, 2008 1:59:39 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space

Apr 4, 2008 1:59:53 PM org.apache.solr.core.SolrCore execute
INFO: /select  
wt=xml&facet.limit=100&rows=100&start=0&facet=true&facet.mincount=1&fl=id,item_name,category,price,price_txt,url,host,country&q=webcam&qt=dismax&version=2.2&facet.field=country&facet.field=host&fq=host:"www.excaliberpc.com"&hl=true
0  
18
Apr 4, 2008 2:00:51 PM org.apache.solr.core.SolrCore execute
INFO: /select  
wt=xml&facet.limit=100&rows=100&start=0&facet=true&facet.mincount=1&fl=id,item_name,category,price,price_txt,url,host,country&q=pepe+jeans&qt=dismax&version=2.2&facet.field=country&facet.field=host&hl=true
0  
38544
Apr 4, 2008 2:02:11 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space

Apr 4, 2008 2:02:11 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space

Apr 4, 2008 2:02:11 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space

Apr 4, 2008 2:02:11 PM org.apache.solr.core.SolrCore execute
INFO: /select  
wt=xml&facet.limit=100&rows=10&start=0&facet=true&facet.mincount=1&fl=id,item_name,category,price,price_txt,url,host,country&q=category:"core"&qt=standard&version=2.2&facet.field=country&facet.field=host&hl=true
0  
79439
Apr 4, 2008 2:02:21 PM org.apache.solr.core.SolrCore execute
INFO: /select  
wt=xml&facet.limit=100&rows=100&start=0&facet=true&facet.mincount=1&fl=id,item_name,category,price,price_txt,url,host,country&q=robot&qt=dismax&version=2.2&facet.field=country&facet.field=host&fq=host:"www.clickonit.com"&hl=true
0  
17
Apr 4, 2008 2:02:35 PM org.apache.solr.core.SolrCore execute
INFO: /select  
wt=xml&facet.limit=100&rows=100&start=0&facet=true&facet.mincount=1&fl=id,item_name,category,price,price_txt,url,host,country&q=Cognac&qt=dismax&version=2.2&facet.field=country&facet.field=host&fq=host:"www.designersimports.com"&hl=true
0  
19
Apr 4, 2008 2:03:12 PM org.apache.solr.core.SolrCore execute
INFO: /select  
wt=xml&facet.limit=100&rows=100&start=0&facet=true&facet.mincount=1&fl=id,item_name,category,price,price_txt,url,host,country&q=prada&qt=dismax&version=2.2&facet.field=country&facet.field=host&fq=host:"www.theprincessescloset.com"&hl=true
0  
1
Apr 4, 2008 2:04:55 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space

Apr 4, 2008 2:04:55 PM org.apache.solr.core.SolrCore execute
INFO: /select  
wt=xml&facet.limit=100&rows=100&start=0&sort=price+desc&facet=true&facet.mincount=1&fl=id,item_name,category,price,price_txt,url,host,country&q=velodyne+DD+15&qt=dismax&version=2.2&facet.field=country&facet.field=host&hl=true
0  
53
Apr 4, 2008 2:05:21 PM org.apache.solr.core.SolrCore execute
INFO: /select  
wt=xml&facet.limit=100&rows=100&start=0&facet=true&facet.mincount=1&fl=id,item_name,category,price,price_txt,url,host,country&q=velodyne+DD+15&qt=dismax&version=2.2&facet.field=country&facet.field=host&fq=host:"www.hometheaterstore.com"&hl=true
0  
3
Apr 4, 2008 2:06:06 PM org.apache.solr.core.SolrCore execute
INFO: /select  
wt=xml&facet.limit=100&rows=100&start=0&sort=id+asc&facet=true&facet.mincount=1&fl=id,item_name,category,price,price_txt,url,host,country&q=sex&qt=dismax&version=2.2&facet.field=country&facet.field=host&fq=host:"www.moviesunlimited.com"&fq=category:"video"&hl=true
0  
39
Apr 4, 2008 2:06:24 PM org.apache.solr.core.SolrCore execute
INFO: /select  
wt=xml&facet.limit=100&rows=10&start=0&facet=true&facet.mincount=1&fl=id,item_name,category,price,price_txt,url,host,country&q=id:[*+TO+*]&qt=standard&version=2.2&facet.field=country&facet.field=host&hl=true
0  
859
Apr 4, 2008 2:07:03 PM org.apache.solr.core.SolrCore execute
INFO: /select  
wt=xml&facet.limit=100&rows=100&start=0&sort=host+asc&facet=true&facet.mincount=1&fl=id,item_name,category,price,price_txt,url,host,country&q=sex&qt=dismax&version=2.2&facet.field=country&facet.field=host&fq=host:"www.moviesunlimited.com"&fq=category:"video"&hl=true
0  
11
Apr 4, 2008 2:07:42 PM org.apache.solr.core.SolrCore execute
INFO: /select  
wt=xml&facet.limit=100&rows=100&start=100&facet=true&facet.mincount=1&fl=id,item_name,category,price,price_txt,url,host,country&q=www.cheapbats.com&qt=dismax&version=2.2&facet.field=country&facet.field=host&hl=true
0  
38441
Full thread dump Java HotSpot(TM) 64-Bit Server VM (1.5.0_14-b03 mixed mode):

"catalina-exec-52" daemon prio=1 tid=0x00002aaaaffff5d0 nid=0x732b  
waiting on condition [0x0000000041b6b000..0x0000000041b6bda0]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-51" daemon prio=1 tid=0x00002aaaab18c460 nid=0x729f  
waiting on condition [0x0000000043888000..0x0000000043888d20]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-50" daemon prio=1 tid=0x00002aaaad17d800 nid=0x5dcd  
waiting on condition [0x0000000044999000..0x0000000044999ca0]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-49" daemon prio=1 tid=0x00002aaaad17cbc0 nid=0x5dcc  
waiting on condition [0x0000000044898000..0x0000000044898c20]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-48" daemon prio=1 tid=0x00002aaaab121af0 nid=0x5dcb  
waiting on condition [0x0000000044797000..0x0000000044797ba0]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-47" daemon prio=1 tid=0x00002aaaac63e040 nid=0x5dca  
waiting on condition [0x0000000044696000..0x0000000044696b20]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-46" daemon prio=1 tid=0x00002aaaac63c1b0 nid=0x5dc9  
waiting on condition [0x0000000044595000..0x0000000044595ea0]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-45" daemon prio=1 tid=0x00002aaaab11fd60 nid=0x5dc8  
waiting on condition [0x0000000044494000..0x0000000044494e20]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-44" daemon prio=1 tid=0x00002aaaac63f390 nid=0x5dc7  
waiting on condition [0x0000000044393000..0x0000000044393da0]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-43" daemon prio=1 tid=0x00002aaaac63d300 nid=0x5dc6  
waiting on condition [0x0000000044292000..0x0000000044292d20]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-42" daemon prio=1 tid=0x00002aaaac63c6c0 nid=0x5dc5  
waiting on condition [0x0000000044191000..0x0000000044191ca0]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-41" daemon prio=1 tid=0x00002aaaac63b2d0 nid=0x5dc4  
waiting on condition [0x0000000044090000..0x0000000044090c20]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-40" daemon prio=1 tid=0x00002aaaac63a690 nid=0x5dc3  
waiting on condition [0x0000000043f8f000..0x0000000043f8fba0]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-39" daemon prio=1 tid=0x00002aaaac639a10 nid=0x5dc2  
waiting on condition [0x0000000043e8e000..0x0000000043e8eb20]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-38" daemon prio=1 tid=0x00002aaaab857210 nid=0x5dc1  
waiting on condition [0x0000000043d8d000..0x0000000043d8dea0]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-37" daemon prio=1 tid=0x00002aaaab8565d0 nid=0x5dc0  
waiting on condition [0x0000000043c8c000..0x0000000043c8ce20]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-36" daemon prio=1 tid=0x00002aaaab16ca10 nid=0x5dbf  
waiting on condition [0x0000000043b8b000..0x0000000043b8bda0]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-35" daemon prio=1 tid=0x00002aaaab168d30 nid=0x5dbe  
waiting on condition [0x0000000043a8a000..0x0000000043a8ad20]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-34" daemon prio=1 tid=0x00002aaaab1680f0 nid=0x5dbd  
waiting on condition [0x0000000043989000..0x0000000043989ca0]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-32" daemon prio=1 tid=0x00002aaaab166870 nid=0x5dbb  
waiting on condition [0x0000000043787000..0x0000000043787ba0]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-31" daemon prio=1 tid=0x00002aaaab165c30 nid=0x5dba  
waiting on condition [0x0000000043686000..0x0000000043686b20]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-30" daemon prio=1 tid=0x00002aaaaab00850 nid=0x5db9  
waiting on condition [0x0000000043585000..0x0000000043585ea0]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-29" daemon prio=1 tid=0x00002aaaaaaffc50 nid=0x5db8  
waiting on condition [0x0000000043484000..0x0000000043484e20]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-28" daemon prio=1 tid=0x00002aaaaaaff010 nid=0x5db7  
waiting on condition [0x0000000043383000..0x0000000043383da0]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-27" daemon prio=1 tid=0x00002aaaaaafe3d0 nid=0x5db6  
waiting on condition [0x0000000043282000..0x0000000043282d20]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-26" daemon prio=1 tid=0x00002aaaabf401e0 nid=0x5db5  
waiting on condition [0x0000000043181000..0x0000000043181ca0]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-25" daemon prio=1 tid=0x00002aaaabf3eca0 nid=0x5db4  
waiting on condition [0x0000000043080000..0x0000000043080c20]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-24" daemon prio=1 tid=0x00002aaaab8533c0 nid=0x5db3  
waiting on condition [0x0000000042f7f000..0x0000000042f7fba0]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-23" daemon prio=1 tid=0x00002aaaaf2b18a0 nid=0x5db2  
waiting on condition [0x0000000042e7e000..0x0000000042e7eb20]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-22" daemon prio=1 tid=0x00002aaaaf2b0d90 nid=0x5db1  
waiting on condition [0x0000000042d7d000..0x0000000042d7dea0]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-21" daemon prio=1 tid=0x00002aaaab557410 nid=0x5db0  
waiting on condition [0x0000000042c7c000..0x0000000042c7ce20]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-20" daemon prio=1 tid=0x00002aaaaaafcf40 nid=0x5daf  
waiting on condition [0x0000000042b7b000..0x0000000042b7bda0]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-19" daemon prio=1 tid=0x00002aaaaaafbfe0 nid=0x5dae  
waiting on condition [0x0000000042a7a000..0x0000000042a7ad20]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-18" daemon prio=1 tid=0x00002aaaae3d6c10 nid=0x5dad  
waiting on condition [0x0000000042979000..0x0000000042979ca0]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-17" daemon prio=1 tid=0x00002aaaae3d5fd0 nid=0x5dac  
waiting on condition [0x0000000042878000..0x0000000042878c20]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-16" daemon prio=1 tid=0x00002aaaae3d53c0 nid=0x5dab  
waiting on condition [0x0000000042777000..0x0000000042777ba0]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-15" daemon prio=1 tid=0x00002aaaabea4070 nid=0x5daa  
waiting on condition [0x0000000042676000..0x0000000042676b20]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-14" daemon prio=1 tid=0x00002aaaaf4bfab0 nid=0x5da9  
waiting on condition [0x0000000042575000..0x0000000042575ea0]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-13" daemon prio=1 tid=0x00002aaaae3d7bc0 nid=0x5da8  
waiting on condition [0x0000000042474000..0x0000000042474e20]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-12" daemon prio=1 tid=0x00002aaaadbe4130 nid=0x5da7  
waiting on condition [0x0000000042373000..0x0000000042373da0]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-11" daemon prio=1 tid=0x00002aaaae3902f0 nid=0x5da6  
waiting on condition [0x0000000042272000..0x0000000042272d20]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-10" daemon prio=1 tid=0x00002aaaae3910a0 nid=0x5da5  
waiting on condition [0x0000000042171000..0x0000000042171ca0]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-9" daemon prio=1 tid=0x00002aaaabecf790 nid=0x5da4  
waiting on condition [0x0000000042070000..0x0000000042070c20]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-8" daemon prio=1 tid=0x00002aaaabece030 nid=0x5da3  
waiting on condition [0x0000000041f6f000..0x0000000041f6fba0]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-7" daemon prio=1 tid=0x00002aaaab556c30 nid=0x5da1  
waiting on condition [0x0000000041e6e000..0x0000000041e6eb20]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-6" daemon prio=1 tid=0x00002aaaaf2adfd0 nid=0x5da0  
waiting on condition [0x0000000041d6d000..0x0000000041d6dea0]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-5" daemon prio=1 tid=0x00002aaaae38bb70 nid=0x5d9f  
waiting on condition [0x0000000041c6c000..0x0000000041c6ce20]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-3" daemon prio=1 tid=0x00002aaaab596170 nid=0x5d9d  
waiting on condition [0x0000000041a6a000..0x0000000041a6ad20]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-2" daemon prio=1 tid=0x00002aaaabf35340 nid=0x5d9c  
waiting on condition [0x0000000041969000..0x0000000041969ca0]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"ajp-18009-Acceptor-0" daemon prio=1 tid=0x00002aaaafb29d90 nid=0x5d9b  
runnable [0x0000000041868000..0x0000000041868c20]
	at org.apache.tomcat.jni.Socket.accept(Native Method)
	at org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:1110)
	at java.lang.Thread.run(Thread.java:595)

"ajp-18009-CometPoller-0" daemon prio=1 tid=0x00002aaaafb4d150  
nid=0x5d9a in Object.wait() [0x0000000041767000..0x0000000041767ba0]
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002b95478569d0> (a  
org.apache.tomcat.util.net.AprEndpoint$Poller)
	at java.lang.Object.wait(Object.java:474)
	at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1267)
	- locked <0x00002b95478569d0> (a  
org.apache.tomcat.util.net.AprEndpoint$Poller)
	at java.lang.Thread.run(Thread.java:595)

"ajp-18009-Poller-0" daemon prio=1 tid=0x00002aaaafb4cc40 nid=0x5d99  
in Object.wait() [0x0000000041666000..0x0000000041666b20]
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002b9547856c30> (a  
org.apache.tomcat.util.net.AprEndpoint$Poller)
	at java.lang.Object.wait(Object.java:474)
	at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1267)
	- locked <0x00002b9547856c30> (a  
org.apache.tomcat.util.net.AprEndpoint$Poller)
	at java.lang.Thread.run(Thread.java:595)

"catalina-exec-1" daemon prio=1 tid=0x00002aaaae38c5d0 nid=0x5d98  
waiting on condition [0x0000000041565000..0x0000000041565ea0]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"http-18080-Acceptor-0" daemon prio=1 tid=0x00002aaaafb010b0  
nid=0x5d97 runnable [0x0000000041464000..0x0000000041464e20]
	at org.apache.tomcat.jni.Socket.accept(Native Method)
	at org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:1110)
	at java.lang.Thread.run(Thread.java:595)

"http-18080-Sendfile-0" daemon prio=1 tid=0x00002aaaafb50b20  
nid=0x5d96 in Object.wait() [0x0000000041363000..0x0000000041363da0]
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002b954d8bca60> (a  
org.apache.tomcat.util.net.AprEndpoint$Sendfile)
	at java.lang.Object.wait(Object.java:474)
	at org.apache.tomcat.util.net.AprEndpoint$Sendfile.run(AprEndpoint.java:1725)
	- locked <0x00002b954d8bca60> (a  
org.apache.tomcat.util.net.AprEndpoint$Sendfile)
	at java.lang.Thread.run(Thread.java:595)

"http-18080-CometPoller-0" daemon prio=1 tid=0x00002aaaafb5e680  
nid=0x5d95 in Object.wait() [0x0000000041262000..0x0000000041262d20]
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002b954d8bcb78> (a  
org.apache.tomcat.util.net.AprEndpoint$Poller)
	at java.lang.Object.wait(Object.java:474)
	at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1267)
	- locked <0x00002b954d8bcb78> (a  
org.apache.tomcat.util.net.AprEndpoint$Poller)
	at java.lang.Thread.run(Thread.java:595)

"pool-1-thread-1" prio=1 tid=0x00002aaaafb03a40 nid=0x5d92 waiting on  
condition [0x0000000040f5f000..0x0000000040f5fba0]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at  
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at  
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at  
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
	at java.lang.Thread.run(Thread.java:595)

"Low Memory Detector" daemon prio=1 tid=0x00002aaaaaba0a50 nid=0x5d90  
runnable [0x0000000000000000..0x0000000000000000]

"CompilerThread1" daemon prio=1 tid=0x00002aaaaab9ef60 nid=0x5d8f  
waiting on condition [0x0000000000000000..0x0000000040c5b740]

"CompilerThread0" daemon prio=1 tid=0x00002aaaaab9d9b0 nid=0x5d8e  
waiting on condition [0x0000000000000000..0x0000000040b5a6f0]

"AdapterThread" daemon prio=1 tid=0x00002aaaaab9c390 nid=0x5d8d  
waiting on condition [0x0000000000000000..0x0000000000000000]

"Signal Dispatcher" daemon prio=1 tid=0x00002aaaaab9b030 nid=0x5d8c  
runnable [0x0000000000000000..0x0000000000000000]

"Finalizer" daemon prio=1 tid=0x00002aaaaab86840 nid=0x5d8b in  
Object.wait() [0x0000000040858000..0x0000000040858c20]
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002b954d994708> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
	- locked <0x00002b954d994708> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=1 tid=0x00002aaaaab85d60 nid=0x5d8a in  
Object.wait() [0x0000000040757000..0x0000000040757ba0]
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002b95477625d8> (a java.lang.ref.Reference$Lock)
	at java.lang.Object.wait(Object.java:474)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
	- locked <0x00002b95477625d8> (a java.lang.ref.Reference$Lock)

"main" prio=1 tid=0x0000000040116020 nid=0x5d83 runnable  
[0x00007fff70172000..0x00007fff70172d00]
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
	- locked <0x00002b954d994920> (a java.net.SocksSocketImpl)
	at java.net.ServerSocket.implAccept(ServerSocket.java:450)
	at java.net.ServerSocket.accept(ServerSocket.java:421)
	at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
	at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
	at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at  
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at  
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)

"VM Thread" prio=1 tid=0x00002aaaaab81870 nid=0x5d89 runnable

"GC task thread#0 (ParallelGC)" prio=1 tid=0x0000000040132b60  
nid=0x5d85 runnable

"GC task thread#1 (ParallelGC)" prio=1 tid=0x00000000401339e0  
nid=0x5d86 runnable

"GC task thread#2 (ParallelGC)" prio=1 tid=0x0000000040134860  
nid=0x5d87 runnable

"GC task thread#3 (ParallelGC)" prio=1 tid=0x00000000401356e0  
nid=0x5d88 runnable

"VM Periodic Task Thread" prio=1 tid=0x0000000040129ab0 nid=0x5d91  
waiting on condition












==============
http://www.linkedin.com/in/liferay


Quoting Noble Paul ??????? ?????? <noble.paul@gmail.com>:

> Can we collect more information. It would be nice to know what the
> threads are doing when it hangs.
> If you are using *nix  issue kill -3 <pid>
> it would print out the stacktrace of all the threads in the VM . That
> may tell us what is the state of each thread which could help us
> suggest something
>
>
> On Tue, Jul 15, 2008 at 8:59 PM, Fuad Efendi <fuad@efendi.ca> wrote:
>> I constantly have the same problem; sometimes I have OutOfMemoryError in
>> logs, sometimes
>> not. Not-predictable. I minimized all caches, it still happens even with
>> 8192M. CPU usage
>> is 375%-400% (two double-core Opterons), SUN Java 5. Moved to BEA JRockit 5
>> yesterday,
>> looks 30 times faster (25% CPU load with 4096M RAM); no any problem yet,
>> let's see...
>>
>> Strange: Tomcat simply hangs instead of exit(...)
>>
>> There are some posts related to OutOfMemoryError in solr-user list.
>>
>>
>> ==============
>> http://www.linkedin.com/in/liferay
>>
>> Quoting Doug Steigerwald <dsteigerwald@mcclatchyinteractive.com>:
>>
>>> Since we pushed Solr out to production a few weeks ago, we've seen a
>>> few issues with Solr not responding to requests (searches or admin
>>> pages).  There doesn't seem to be any reason for it from what we can
>>> tell.  We haven't seen it in QA or development.
>>>
>>> We're running Solr with basically the example Solr setup with Jetty
>>> (6.1.3).  We package our Solr install by using 'ant example' and
>>> replacing configs/etc.  Whenever Solr stops responding, there are no
>>> messages in the logs, nothing.  Requests just time out.
>>>
>>> We have also only seen this on our slaves.  The master doesn't seem to
>>> be hitting this issue.  All the boxes are the same, version of java is
>>> the same, etc.
>>>
>>> We don't have a stack trace and no JMX set up.  Once we see this issue,
>>> our support folks just stop and start Solr on that machine.
>>>
>>> Has anyone else run into anything like this with Solr?
>>>
>>> Thanks.
>>> Doug
>>
>>
>>
>>
>
>
>
> --
> --Noble Paul
>




Mime
View raw message