lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Altamirano, Emmanuel" <Emmanuel.Altamir...@transunion.com.INVALID>
Subject SolrCloud crashing due to memory error - 'Cannot allocate memory' (errno=12)
Date Fri, 11 Dec 2020 03:55:23 GMT
Hello,

We have a SolrCloud(8.6) with 3 servers with the same characteristics and configuration. We
assigned 32GB for heap memory each, and after some short period of time sending 40 concurrent
requests to the SolrCloud using a load balancer, we are getting the following error that shutdown
each Solr Server and Zookeeper:

OpenJDK 64-Bit Server VM warning: Failed to reserve large pages memory req_addr: 0x0000000000000000
bytes: 536870912 (errno = 12).
OpenJDK 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007edd4d9da000, 12288, 0) failed;
error='Cannot allocate memory' (errno=12)


20201201 10:43:29.495 [ERROR] {qtp2051853139-23369} [c:express s:shard1 r:core_node6 x:express_shard1_replica_n4]
[org.apache.solr.handler.RequestHandlerBase, 148] | org.apache.solr.common.SolrException:
Cannot talk to ZooKeeper - Updates are disabled.
        at org.apache.solr.update.processor.DistributedZkUpdateProcessor.zkCheck(DistributedZkUpdateProcessor.java:1245)
        at org.apache.solr.update.processor.DistributedZkUpdateProcessor.setupRequest(DistributedZkUpdateProcessor.java:582)
        at org.apache.solr.update.processor.DistributedZkUpdateProcessor.processAdd(DistributedZkUpdateProcessor.java:239)

[cid:image004.jpg@01D6CF3F.27574B90]

We have a one collection with one shard, almost 400 million documents (~334GB).

$ sysctl vm.nr_hugepages
vm.nr_hugepages = 32768
$ sysctl vm.max_map_count
vm.max_map_count = 131072

/etc/security/limits.conf

* - core unlimited
* - data unlimited
* - priority unlimited
* - fsize unlimited
* - sigpending 513928
* - memlock unlimited
* - nofile 131072
* - msgqueue 819200
* - rtprio 0
* - stack 8192
* - cpu unlimited
* - rss unlimited #virtual memory unlimited
* - locks unlimited
* soft nproc 65536
* hard nproc 65536
* - nofile 131072



/etc/sysctl.conf

vm.nr_hugepages =  32768
vm.max_map_count = 131072


Could you please provide me some advice to fix this error?

Thanks,

Emmanuel Altamirano

Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message