lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thiru M <m.thirukuma...@gmail.com>
Subject Re: Spike in SOLR Process and Frequent GC
Date Wed, 31 Aug 2016 04:28:06 GMT
Thanks for your response *Shawn*.

Have responded your queries and shared the SOLR details here.

What precisely were you measuring during the night with no activity ?

·         Earlier we have enabled delta content indexing (content which got
updated on day to day basis) script which triggers every 30 minutes. We
monitored the load on SOLR when the delta indexing is performed.

·         CPU utilization by SOLR process,

·         Heap memory usage.

what precise methods were you using to measure it?

·         We used the “*top*” command to monitor the solr process,

·         We monitored the free space in the server during the delta change
indexing process,

·         We also enabled JMX remote monitoring in SOLR and used *jConsole*
& *jVisualVM* to monitor CPU, Heap & Thread utilizations.

What part of the information you obtained represents a problem in your mind?

·         Information obtained from top and the SOLR GC log in the *linux-a*
server

·         Allocated max heap size is 2 GB. Based on the jConsole monitor h*eap
usage is < 600 MB in linux-a server but the CPU utilization by SOLR process
is on top (in figures it consumes 0.5 to 5 %)*.


There is one solr instance per server i.e in linux-a one solr instance and
in linux-b one solr instance (stand alone) is available.



linux-a

linux-b

Number of Cores

49

14

Number of Documents

6145495 (~6M)

1923181  (~1M)

Overall index directory size

9.9 GB

14 GB

Min Heap Memory

256 MB

Max Heap Memory

2048 MB

Total # System Processors

40

Overall RAM size

125 GB

Java Version

1.8.0_65 64-Bit Server VM

SOLR Parameters

-Xms512m
-Xmx2048m
-XX:NewRatio=3
-XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90
-XX:MaxTenuringThreshold=8
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC
-XX:ConcGCThreads=4
-XX:ParallelGCThreads=4
-XX:+CMSScavengeBeforeRemark
-XX:PretenureSizeThreshold=64m
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
-XX:CMSMaxAbortablePrecleanTime=6000
-XX:+CMSParallelRemarkEnabled
-XX:+ParallelRefProcEnabled
-verbose:gc
-XX:+PrintHeapAtGC
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+PrintGCTimeStamps
-XX:+PrintTenuringDistribution
-XX:+PrintGCApplicationStoppedTime
-Xloggc:/app/solr/logs/solr_gc.log
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=17010
-Dcom.sun.management.jmxremote.rmi.port=17010
-Djetty.port=7010
-DSTOP.PORT=6010
-DSTOP.KEY=solrrocks
-Duser.timezone=UTC
-Djetty.home=/opt/solr/server
-Dsolr.solr.home=/local/solr/default/data
-Dsolr.install.dir=/opt/solr
-Dlog4j.configuration=file:/app/solr/log4j.properties
-Xss256k


Regards,

Thirukumaran M


On Sun, Aug 28, 2016 at 2:01 AM, Shawn Heisey <apache@elyograg.org> wrote:

> On 8/27/2016 9:08 AM, Thiru M wrote:
> > We are using Solr 5.4.0 - "stand-alone" mode in our production boxes
> > hosted in Red Hat Enterprise Linux (RHEL) OS.
> >
> > Each box have number of different cores. Have attached the screenshot
> > shot with the Solr core & system details.
> >
> > 1. Earlier indexing was performed every 30 minutes in both production
> > servers,
> >
> > 2. In linux-a server 30 (stand-alone) cores created on same day and
> > content indexed into it,
> >
> > 3. we then spotted unusual GC performing every 2 to 7 seconds in
> > linux-a server and the  Solr process spiked,
> >
> > 4. Then we removed the indexing in linux-a server for a week,
> > monitored the both Solr process and GC.(No indexing performed during
> > this time),
> >
> > 5. No one uses the system during night time which we ensured it from
> > our end. But both Solr process and GC were in its peak, even "during
> > non business hours",
> >
> > 6. Have restated Solr instance in linux-a server. GC started again
> > after Solr instance brought up,
> >
> > 7. In linux-b server no spike in Solr process and no issues with GC,
> >
>
> Indexing creates a LOT of garbage.  Queries also create garbage, but not
> nearly as fast as indexing.  Solr has some background processes, and
> these will create garbage too.  Java uses a garbage collection memory
> model, so this is completely normal for java applications.
>
> What precisely were you measuring during the night with no activity, and
> what precise methods were you using to measure it?  What part of the
> information you obtained represents a problem in your mind?
>
> We'll also need some details about these servers:
>
> * Total index size of all Solr cores on the server.
> * Total amount of memory installed in the server.
> * Total number of documents contained in all Solr cores.
> * How many Solr instances per server?  What is the max heap size of each
> instance?
>
> Attachments rarely make it to the list.  The screenshot you mentioned
> did not make it.  You'll need to put it somewhere on the Internet an
> provide a URL.  Sharing sites like drobox or imgur are good choices for
> image data.
>
> Since I don't have a clear idea of what the exact issue is here, I don't
> have any immediate suggestions, aside from possibly increasing your max
> heap size ... but depending on the answers to the questions above, that
> might make things worse.
>
> Thanks,
> Shawn
>
>


-- 
By
M.Thirukumaran

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message