lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Modassar Ather <modather1...@gmail.com>
Subject Re: Very high memory and CPU utilization.
Date Mon, 02 Nov 2015 11:57:27 GMT
Okay. I guess your observation of 400% for a single core is with top and
looking at that core's entry? If so, the 400% can be explained by
excessive garbage collection. You could turn GC-logging on to check
that. With a bit of luck GC would be the cause of the slow down.

Yes it is with top command. I will check GC activities and try to relate
with CPU usage.

The query q=network se* is quick enough in our system too. It takes around
3-4 seconds for around 8 million records.
The problem is with the same query as phrase. q="network se*".
Can you please share your experience with such query where the wild card
expansion is huge like in the query above?

I changed my SolrCloud setup from 12 shard to 8 shard and given each shard
30 GB of RAM on the same machine with same index size (re-indexed) but
could not see the significant improvement for the query given.

I will check the swap activity.

Also can you please share your experiences with respect to RAM, GC, solr
cache setup etc as it seems by your comment that the SolrCloud environment
you have is kind of similar to the one I work on?

Regards,
Modassar

On Mon, Nov 2, 2015 at 5:20 PM, Toke Eskildsen <te@statsbiblioteket.dk>
wrote:

> On Mon, 2015-11-02 at 16:25 +0530, Modassar Ather wrote:
> > The remaining size after you removed the heap usage should be reserved
> for
> > the index (not only the other system activities).
> > I am not able to get  the above point. So when I start Solr with 28g RAM,
> > for all the activities related to Solr it should not go beyond 28g. And
> the
> > remaining heap will be used for activities other than Solr. Please help
> me
> > understand.
>
> It is described here:
> https://wiki.apache.org/solr/SolrPerformanceProblems#OS_Disk_Cache
>
> I will be quick to add that I do not agree with Shawn (the primary
> author of the page) on the stated limits and find that the page in
> general ignores that performance requirements differ a great deal.
> Nevertheless, it is very true that Solr performance is tied to the
> amount of OS disk cache:
>
> You can have a machine with 10TB of RAM, but Solr performance will still
> be poor if you use it all for JVMs.
>
> Practically all modern operating system uses free memory for disk cache.
> Free memory is the memory not used for JVMs or other programs. It might
> be that you have a lot less than 30-40GB free: If you are on a Linux
> server, try calling 'top' and see what is says under 'cached'.
>
> Related, I support jim's suggestion to inspect the swap activity:
> In the past we had problem with a machine that insisted on swapping
> excessively, although there were high IO and free memory.
>
> > The disks are SSDs.
>
> That makes your observations stranger still.
>
>
> - Toke Eskildsen, State and University Library, Denmark
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message