lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chetas Joshi <chetas.jo...@gmail.com>
Subject Re: Solr Initialization failure
Date Thu, 05 Jan 2017 01:46:19 GMT
Hi Shawn

Thanks for the explanation!

I have slab count set to 20 and I did not have global block cache.
I have a follow up question. Does setting slab count=1 affect the
write/read performance of Solr while reading the indices from HDFS? Is this
setting just used while creating new cores?

Thanks!

On Wed, Jan 4, 2017 at 4:11 PM, Shawn Heisey <apache@elyograg.org> wrote:

> On 1/4/2017 1:43 PM, Chetas Joshi wrote:
> > while creating a new collection, it fails to spin up solr cores on some
> > nodes due to "insufficient direct memory".
> >
> > Here is the error:
> >
> >    - *3044_01_17_shard42_replica1:*
> > org.apache.solr.common.SolrException:org.apache.solr.
> common.SolrException:
> >    The max direct memory is likely too low. Either increase it (by adding
> >    -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers
> >    startup args) or disable direct allocation using
> >    solr.hdfs.blockcache.direct.memory.allocation=false in
> solrconfig.xml. If
> >    you are putting the block cache on the heap, your java heap size
> might not
> >    be large enough. Failed allocating ~2684.35456 MB.
> >
> > The error is self explanatory.
> >
> > My question is: why does it require around 2.7 GB of off-heap memory to
> > spin up a single core??
>
> This message comes from the HdfsDirectoryFactory class.  This is the
> calculation of the total amount of memory needed:
>
>     long totalMemory = (long) bankCount * (long) numberOfBlocksPerBank
>         * (long) blockSize;
>
> The numberOfBlocksPerBank variable can come from the configuration, the
> code defaults it to 16384.  The blockSize variable gets assigned by a
> convoluted method involving bit shifts, and defaults to 8192.   The
> bankCount variable seems to come from solr.hdfs.blockcache.slab.count,
> and apparently defaults to 1.  Looks like it's been set to 20 on your
> config.  If we assume the other two are at their defaults and you have
> 20 for the slab count, then this results in 2684354560 bytes, which
> would cause the exact output seen in the error message when the memory
> allocation fails.
>
> I know very little about HDFS or how the HDFS directory works, but
> apparently it needs a lot of memory if you want good performance.
> Reducing solr.hdfs.blockcache.slab.count sounds like it might result in
> less memory being required.
>
> You might want to review this page for info about how to set up HDFS,
> where it says that each slab requires 128MB of memory:
>
> https://cwiki.apache.org/confluence/display/solr/Running+Solr+on+HDFS
>
> The default settings for the HDFS directory cause the block cache to be
> global, so all cores use it, instead of spinning up another cache for
> every additional core.
>
> What I've seen sounds like one of these two problems:  1) You've turned
> off the global cache option.  2) This node doesn't yet have any HDFS
> cores, so your collection create is tryin to create the first core using
> HDFS.  That action is trying to allocate the global cache, which has
> been sized at 20 slabs.
>
> Thanks,
> Shawn
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message