lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ravi Solr <ravis...@gmail.com>
Subject Server inconsistent state & Core Reload issue
Date Wed, 01 May 2013 21:14:07 GMT
We are using Solr 3.6.2 with a single core setup on a glassfish server,
every 4-5 hours the server gradually gets into a some kind of a
inconsistent state and stops accepting any queries giving back cached
results. Even the core reload fails giving the following. Has anybody
experienced such behavior ? Can anybody help me understand why this might
happen ?

http://searchserver:80/solr/admin/cores?action=RELOAD&core=core1

<response>
     <lst name="responseHeader">
          <int name="status">0</int>
          <int name="QTime">9</int>
     </lst>
     <lst name="status">
          <lst name="core1">
              <str name="name">core1</str>
              <str name="instanceDir">/data/solr/core1-home/</str>
              <str name="dataDir">/data/solr/core/core1-data/</str>
              <date name="startTime">2013-05-01T19:16:31.32Z</date>
              <long name="uptime">137850</long>
              <lst name="index">
                      <int name="numDocs">21479</int>
                      <int name="maxDoc">25170</int>
                      <long name="version">1367184551418</long>
                      <int name="segmentCount">4</int>
                      <bool name="current">true</bool>
                      <bool name="hasDeletions">true</bool>
                      <str
name="directory">org.apache.lucene.store.MMapDirectory:org.apache.lucene.store.MMapDirectory@/data/solr/core/core1-data/index
lockFactory=org.apache.lucene.store.NativeFSLockFactory@71d9673</str>
                     <date name="lastModified">2013-05-01T19:15:04Z</date>
               </lst>
              </lst>
           </lst>
</response>


During the inconsistent state any queries being issued to the server loose
the query parameters. We can see the proper queries in the container's http
access logs but solr somehow solr doesn't get the query params at all. Also
note that "content length" on the container's access logs is always 68935,
which implies its always giving the same docs irrespective of the query.

If we restart the server everything is back to normal and the same queries
run properly.

SOLR Log
--------------
[#|2013-05-01T15:20:02.031-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=20;_ThreadName=httpSSLWorkerThread-9001-1;|[core1]
webapp=/solr path=/select params={} hits=21479 status=0 QTime=17 |#]

[#|2013-05-01T15:20:02.034-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=24;_ThreadName=httpSSLWorkerThread-9001-4;|[core1]
webapp=/solr path=/select params={} hits=21479 status=0 QTime=13 |#]

[#|2013-05-01T15:20:02.055-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=23;_ThreadName=httpSSLWorkerThread-9001-3;|[core1]
webapp=/solr path=/select params={} hits=21479 status=0 QTime=13 |#]

[#|2013-05-01T15:20:02.081-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=25;_ThreadName=httpSSLWorkerThread-9001-5;|[core1]
webapp=/solr path=/select params={} hits=21479 status=0 QTime=14 |#]

[#|2013-05-01T15:20:02.106-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=19;_ThreadName=httpSSLWorkerThread-9001-0;|[core1]
webapp=/solr path=/select params={} hits=21479 status=0 QTime=14 |#]

[#|2013-05-01T15:20:02.136-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=22;_ThreadName=httpSSLWorkerThread-9001-2;|[core1]
webapp=/solr path=/select params={} hits=21479 status=0 QTime=16 |#]

[#|2013-05-01T15:20:02.161-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=20;_ThreadName=httpSSLWorkerThread-9001-1;|[core1]
webapp=/solr path=/select params={} hits=21479 status=0 QTime=15 |#]

[#|2013-05-01T15:20:02.185-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=24;_ThreadName=httpSSLWorkerThread-9001-4;|[core1]
webapp=/solr path=/select params={} hits=21479 status=0 QTime=14 |#]

[#|2013-05-01T15:20:02.209-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=23;_ThreadName=httpSSLWorkerThread-9001-3;|[core1]
webapp=/solr path=/select params={} hits=21479 status=0 QTime=14 |#]

[#|2013-05-01T15:20:02.241-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=25;_ThreadName=httpSSLWorkerThread-9001-5;|[core1]
webapp=/solr path=/select params={} hits=21479 status=0 QTime=16 |#]

[#|2013-05-01T15:20:02.266-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=19;_ThreadName=httpSSLWorkerThread-9001-0;|[core1]
webapp=/solr path=/select params={} hits=21479 status=0 QTime=15 |#]

[#|2013-05-01T15:20:02.288-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=22;_ThreadName=httpSSLWorkerThread-9001-2;|[core1]
webapp=/solr path=/select params={} hits=21479 status=0 QTime=14 |#]

[#|2013-05-01T15:20:02.291-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=20;_ThreadName=httpSSLWorkerThread-9001-1;|[core1]
webapp=/solr path=/select params={} hits=21479 status=0 QTime=15 |#]



Container Access Logs
---------------------------------
"xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET
/solr/core1/select?q=*%3A*&rows=250&start=0&facet=true&fq=source%3A%
22site.com%22&fq=categories%3A%28%22Music+Venues%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2
HTTP/1.1" 200 68935

"xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET
/solr/core1/select?q=*%3A*&rows=5&start=0&facet=true&fq=source%3A%22site.com%22&fq=categories%3A%28%22Exhibits%22%29&fq=types%3A%28%22Painting%2FDrawing%22%29&sort=closingdate+asc%2C+openingdate+desc%2C+namestring+asc&wt=javabin&version=2&wt=javabin&version=2
HTTP/1.1" 200 68935

"xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET
/solr/core1/select?q=*%3A*&rows=1&start=0&facet=true&fq=source%3A%22site.com%22&fq=categories%3A%28%22Music+Events%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2
HTTP/1.1" 200 68935

"xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET
/solr/core1/select?q=*%3A*&rows=250&start=0&facet=true&fq=source%3A%
22site.com%22&fq=categories%3A%28%22Museums%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2
HTTP/1.1" 200 68935

"xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET
/solr/core1/select?q=*%3A*&rows=250&start=0&facet=true&fq=source%3A%
22site.com%22&fq=categories%3A%28%22Exhibits%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2
HTTP/1.1" 200 68935

"xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET
/solr/core1/select?q=*%3A*&rows=250&start=0&facet=true&fq=source%3A%
22site.com%22&fq=categories%3A%28%22Performing+Arts%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2
HTTP/1.1" 200 68935

"xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET
/solr/core1/select?q=*%3A*&rows=250&start=0&facet=true&fq=source%3A%
22site.com%22&fq=categories%3A%28%22Theaters%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2
HTTP/1.1" 200 68935

"xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET
/solr/core1/select?q=*%3A*&rows=250&start=0&facet=true&fq=source%3A%
22site.com%22&fq=specialinterests%3A%28%22Kid-Friendly%22%29&fq=group%3A%22Events%22&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2
HTTP/1.1" 200 68935

"xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET
/solr/core1/select?q=*%3A*&rows=250&start=0&facet=true&fq=source%3A%
22site.com%22&fq=specialinterests%3A%28%22Kid-Friendly%22%29&fq=group%3A%22Locations%22&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2
HTTP/1.1" 200 68935

"xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET
/solr/core1/select?q=*%3A*&rows=1&start=0&facet=true&fq=source%3A%22site.com%22&fq=specialinterests%3A%28%22Kid-Friendly%22%29&fq=categories%3A%28%22Performing+Arts%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2
HTTP/1.1" 200 68935

"xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET
/solr/core1/select?q=*%3A*&rows=1&start=0&facet=true&fq=source%3A%22site.com%22&fq=specialinterests%3A%28%22Kid-Friendly%22%29&fq=categories%3A%28%22Music+Events%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2
HTTP/1.1" 200 68935

"xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET
/solr/core1/select?q=*%3A*&rows=1&start=0&facet=true&fq=source%3A%22site.com%22&fq=specialinterests%3A%28%22Kid-Friendly%22%29&fq=categories%3A%28%22Exhibits%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2
HTTP/1.1" 200 68935

"xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET
/solr/core1/select?q=*%3A*&rows=1&start=0&facet=true&fq=source%3A%22site.com%22&fq=specialinterests%3A%28%22Kid-Friendly%22%29&fq=categories%3A%28%22Restaurants%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2
HTTP/1.1" 200 68935

"xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET
/solr/core1/select?q=*%3A*&rows=1&start=0&facet=true&fq=source%3A%22site.com%22&fq=specialinterests%3A%28%22Kid-Friendly%22%29&fq=categories%3A%28%22Museums%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2
HTTP/1.1" 200 68935


Thanks,

Ravi

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message