directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rashid Mahmood (Jira)" <j...@apache.org>
Subject [jira] [Commented] (DIRSERVER-2255) java.lang.OutOfMemoryError: Java heap space
Date Fri, 18 Sep 2020 15:22:00 GMT

    [ https://issues.apache.org/jira/browse/DIRSERVER-2255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17198396#comment-17198396
] 

Rashid Mahmood commented on DIRSERVER-2255:
-------------------------------------------

We are facing same problem in Multi-Master setup in our Reference Env. Next target is our
production environment

 

We have 2 seperate data centers backed by load balancer. Each data center has a Maste-Slave(Machine1
Master and Machine 2 slave) setup with 20k enrtries.

We need a Master-Master setup between Data Centers, i enabled the consumer on each Data Center's
Master instance to poll the other Data Center's Master instance. Those Master instances had
already 20K Entries. Then i noticed Replication was not working, any change made on one Master
is not replicated on other Master. Memory usage on each Master was reaching to 2GB. Took a
heap dump on one instance and found SyncReplSearchListener occupied almost 90% of memory.

 

What could be the reason???

 

Couple of more questions:

1). Are both Masters trying to synch all entries? I think so maybe because both newly enabled
consumers on each Master are not sending any cookie to the target Master, it makes Master
think of sending all entries to consumer.

2). How many consumers one Master can serve, is there any benchmark?

3).  Is there any guideline with best practices in setting up Multi-Master for Production
System?

4). We have a cron job which makes daily backup of our Data Partition, in case of broken DB
on one Master/Slave; should we restore from backup or just create a fresh installation? To
make restore with minimun Repk overhead we need to make backup of the cookie entry so that
Repl-Server send only those modifications which happend after DB corruption, right???   

 

Our Environment:

Linux Centos 7

ApacheDS 2.0.0.AM24

All consumers are with ads-replrefreshnpersist: TRUE configured.

!9-instances-SyncReplSearchListener .png!

> java.lang.OutOfMemoryError: Java heap space
> -------------------------------------------
>
>                 Key: DIRSERVER-2255
>                 URL: https://issues.apache.org/jira/browse/DIRSERVER-2255
>             Project: Directory ApacheDS
>          Issue Type: Bug
>          Components: replication
>    Affects Versions: 2.0.0.AM25
>         Environment: Linux CentOS 7
> Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
>            Reporter: li
>            Priority: Major
>         Attachments: 1.png, 9-instances-SyncReplSearchListener .png, conf.zip, wrapper.conf
>
>
> INFO   | jvm 1    | 2018/10/29 12:01:45 | Exception in thread "Thread-4" java.lang.OutOfMemoryError:
GC overhead limit exceeded
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at java.util.Arrays.copyOf(Arrays.java:3236)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at java.nio.charset.CharsetEncoder.replaceWith(CharsetEncoder.java:301)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at java.nio.charset.CharsetEncoder.<init>(CharsetEncoder.java:207)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at java.nio.charset.CharsetEncoder.<init>(CharsetEncoder.java:233)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at sun.nio.cs.UTF_8$Encoder.<init>(UTF_8.java:558)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at sun.nio.cs.UTF_8$Encoder.<init>(UTF_8.java:554)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at sun.nio.cs.UTF_8.newEncoder(UTF_8.java:72)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at java.lang.StringCoding.encode(StringCoding.java:348)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at java.lang.String.getBytes(String.java:941)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at org.apache.directory.api.util.Strings.getBytesUtf8(Strings.java:1626)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at org.apache.directory.api.ldap.model.entry.Value.<init>(Value.java:261)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at org.apache.directory.api.ldap.model.entry.DefaultAttribute.apply(DefaultAttribute.java:1828)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at org.apache.directory.api.ldap.model.entry.DefaultEntry.readExternal(DefaultEntry.java:2443)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at org.apache.directory.server.ldap.replication.ReplicaEventMessageSerializer.deserialize(ReplicaEventMessageSerializer.java:126)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at jdbm.btree.BPage.deserialize(BPage.java:1188)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at jdbm.btree.BPage.deserialize(BPage.java:81)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at jdbm.recman.BaseRecordManager.fetch(BaseRecordManager.java:329)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at jdbm.btree.BPage.loadBPage(BPage.java:949)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at jdbm.btree.BPage.childBPage(BPage.java:938)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at jdbm.btree.BPage.insert(BPage.java:380)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at jdbm.btree.BPage.insert(BPage.java:381)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at jdbm.btree.BPage.insert(BPage.java:381)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at jdbm.btree.BPage.insert(BPage.java:381)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at jdbm.btree.BPage.insert(BPage.java:381)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at jdbm.btree.BTree.insert(BTree.java:302)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.put(JdbmTable.java:541)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at org.apache.directory.server.ldap.replication.provider.ReplicaEventLog.log(ReplicaEventLog.java:176)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at org.apache.directory.server.ldap.replication.provider.SyncReplSearchListener.entryModified(SyncReplSearchListener.java:314)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at org.apache.directory.server.core.event.EventInterceptor.fire(EventInterceptor.java:169)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at org.apache.directory.server.core.event.EventInterceptor.modify(EventInterceptor.java:333)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at org.apache.directory.server.core.api.interceptor.BaseInterceptor.next(BaseInterceptor.java:508)
> INFO   | jvm 1    | 2018/10/29 12:01:45 | 	at org.apache.directory.server.core.subtree.SubentryInterceptor.modify(SubentryInterceptor.java:1249)
> INFO   | jvm 1    | 2018/10/29 12:01:47 | Exception in thread "pool-13-thread-1" java.lang.OutOfMemoryError:
GC overhead limit exceeded
> INFO   | jvm 1    | 2018/10/29 12:01:47 | 	at org.apache.directory.api.ldap.model.entry.Value.clone(Value.java:425)
> INFO   | jvm 1    | 2018/10/29 12:01:47 | 	at org.apache.directory.api.ldap.model.name.Ava.getValue(Ava.java:728)
> INFO   | jvm 1    | 2018/10/29 12:01:47 | 	at org.apache.directory.api.ldap.model.name.Rdn.<init>(Rdn.java:455)
> INFO   | jvm 1    | 2018/10/29 12:01:47 | 	at org.apache.directory.api.ldap.model.name.Dn.<init>(Dn.java:197)
> INFO   | jvm 1    | 2018/10/29 12:01:47 | 	at org.apache.directory.api.ldap.model.entry.DefaultEntry.normalizeDn(DefaultEntry.java:519)
> INFO   | jvm 1    | 2018/10/29 12:01:47 | 	at org.apache.directory.api.ldap.model.entry.DefaultEntry.<init>(DefaultEntry.java:300)
> INFO   | jvm 1    | 2018/10/29 12:01:47 | 	at org.apache.directory.ldap.client.api.LdapNetworkConnection.messageReceived(LdapNetworkConnection.java:2757)
> INFO   | jvm 1    | 2018/10/29 12:01:47 | 	at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:1019)
> INFO   | jvm 1    | 2018/10/29 12:01:47 | 	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
> INFO   | jvm 1    | 2018/10/29 12:01:47 | 	at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
> INFO   | jvm 1    | 2018/10/29 12:01:47 | 	at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1141)
> INFO   | jvm 1    | 2018/10/29 12:01:47 | 	at org.apache.mina.filter.codec.ProtocolCodecFilter$ProtocolDecoderOutputImpl.flush(ProtocolCodecFilter.java:437)
> INFO   | jvm 1    | 2018/10/29 12:01:47 | 	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:256)
> INFO   | jvm 1    | 2018/10/29 12:01:47 | 	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
> INFO   | jvm 1    | 2018/10/29 12:01:47 | 	at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
> INFO   | jvm 1    | 2018/10/29 12:01:47 | 	at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1141)
> INFO   | jvm 1    | 2018/10/29 12:01:47 | 	at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:122)
> INFO   | jvm 1    | 2018/10/29 12:01:47 | 	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
> INFO   | jvm 1    | 2018/10/29 12:01:47 | 	at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:643)
> INFO   | jvm 1    | 2018/10/29 12:01:47 | 	at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:539)
> INFO   | jvm 1    | 2018/10/29 12:01:47 | 	at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$1200(AbstractPollingIoProcessor.java:68)
> INFO   | jvm 1    | 2018/10/29 12:01:47 | 	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.process(AbstractPollingIoProcessor.java:1242)
> INFO   | jvm 1    | 2018/10/29 12:01:47 | 	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.process(AbstractPollingIoProcessor.java:1231)
> INFO   | jvm 1    | 2018/10/29 12:01:47 | 	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:683)
> INFO   | jvm 1    | 2018/10/29 12:01:47 | 	at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
> INFO   | jvm 1    | 2018/10/29 12:01:47 | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> INFO   | jvm 1    | 2018/10/29 12:01:47 | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> INFO   | jvm 1    | 2018/10/29 12:01:47 | 	at java.lang.Thread.run(Thread.java:748)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | Exception in thread "pool-7-thread-5835" java.lang.OutOfMemoryError:
GC overhead limit exceeded
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at java.io.BufferedReader.<init>(BufferedReader.java:105)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at java.io.BufferedReader.<init>(BufferedReader.java:116)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.directory.api.ldap.model.schema.parsers.OpenLdapSchemaParser.parseAttributeType(OpenLdapSchemaParser.java:2599)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.directory.api.ldap.model.schema.parsers.AttributeTypeDescriptionSchemaParser.parse(AttributeTypeDescriptionSchemaParser.java:81)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.directory.api.ldap.model.schema.syntaxCheckers.AttributeTypeDescriptionSyntaxChecker.isValidSyntax(AttributeTypeDescriptionSyntaxChecker.java:163)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.directory.api.ldap.model.entry.Value.<init>(Value.java:285)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.directory.api.ldap.model.entry.DefaultAttribute.createStringValue(DefaultAttribute.java:449)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.directory.api.ldap.model.entry.DefaultAttribute.add(DefaultAttribute.java:1104)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.directory.server.core.shared.SchemaService.generateAttributeTypes(SchemaService.java:144)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.directory.server.core.shared.SchemaService.generateSchemaSubentry(SchemaService.java:258)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.directory.server.core.shared.SchemaService.getSubschemaEntry(SchemaService.java:432)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.directory.server.core.schema.SchemaInterceptor.search(SchemaInterceptor.java:1519)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.directory.server.core.api.interceptor.BaseInterceptor.next(BaseInterceptor.java:598)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.directory.server.core.authz.DefaultAuthorizationInterceptor.search(DefaultAuthorizationInterceptor.java:354)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.directory.server.core.api.interceptor.BaseInterceptor.next(BaseInterceptor.java:598)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.directory.server.core.authz.AciAuthorizationInterceptor.search(AciAuthorizationInterceptor.java:1247)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.directory.server.core.api.interceptor.BaseInterceptor.next(BaseInterceptor.java:598)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.directory.server.core.authn.AuthenticationInterceptor.search(AuthenticationInterceptor.java:1394)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.directory.server.core.api.interceptor.BaseInterceptor.next(BaseInterceptor.java:598)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.directory.server.core.normalization.NormalizationInterceptor.search(NormalizationInterceptor.java:414)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.directory.server.core.DefaultOperationManager.search(DefaultOperationManager.java:1781)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.directory.server.core.shared.DefaultCoreSession.search(DefaultCoreSession.java:1193)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.directory.server.ldap.handlers.request.SearchRequestHandler.doSimpleSearch(SearchRequestHandler.java:810)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.directory.server.ldap.handlers.request.SearchRequestHandler.handleIgnoringReferrals(SearchRequestHandler.java:1164)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.directory.server.ldap.handlers.request.SearchRequestHandler.handleWithReferrals(SearchRequestHandler.java:1262)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.directory.server.ldap.handlers.request.SearchRequestHandler.handle(SearchRequestHandler.java:212)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.directory.server.ldap.handlers.request.SearchRequestHandler.handle(SearchRequestHandler.java:92)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.directory.server.ldap.handlers.LdapRequestHandler.handleMessage(LdapRequestHandler.java:207)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.directory.server.ldap.handlers.LdapRequestHandler.handleMessage(LdapRequestHandler.java:56)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.mina.handler.demux.DemuxingIoHandler.messageReceived(DemuxingIoHandler.java:243)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.directory.server.ldap.LdapProtocolHandler.messageReceived(LdapProtocolHandler.java:223)
> INFO   | jvm 1    | 2018/10/29 12:50:50 | 	at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:1019)
> STATUS | wrapper  | 2018/10/29 12:56:37 | TERM trapped.  Shutting down.
> INFO   | jvm 1    | 2018/10/29 12:56:46 | Error in WrapperListener.stop callback.  java.lang.OutOfMemoryError:
Java heap space
> INFO   | jvm 1    | 2018/10/29 12:56:46 | java.lang.OutOfMemoryError: Java heap space
> INFO   | jvm 1    | 2018/10/29 12:56:46 | 	at java.util.HashMap.resize(HashMap.java:704)
> INFO   | jvm 1    | 2018/10/29 12:56:46 | 	at java.util.HashMap.putVal(HashMap.java:663)
> INFO   | jvm 1    | 2018/10/29 12:56:46 | 	at java.util.HashMap.put(HashMap.java:612)
> INFO   | jvm 1    | 2018/10/29 12:56:46 | 	at jdbm.recman.RecordFile.commit(RecordFile.java:314)
> INFO   | jvm 1    | 2018/10/29 12:56:46 | 	at jdbm.recman.PageManager.close(PageManager.java:329)
> INFO   | jvm 1    | 2018/10/29 12:56:46 | 	at jdbm.recman.BaseRecordManager.close(BaseRecordManager.java:164)
> INFO   | jvm 1    | 2018/10/29 12:56:46 | 	at org.apache.directory.server.ldap.replication.provider.ReplicaEventLog.stop(ReplicaEventLog.java:226)
> INFO   | jvm 1    | 2018/10/29 12:56:46 | 	at org.apache.directory.server.ldap.replication.provider.SyncReplRequestHandler.stop(SyncReplRequestHandler.java:274)
> INFO   | jvm 1    | 2018/10/29 12:56:46 | 	at org.apache.directory.server.ldap.LdapServer.stop(LdapServer.java:668)
> INFO   | jvm 1    | 2018/10/29 12:56:46 | 	at org.apache.directory.server.ApacheDsService.stop(ApacheDsService.java:610)
> INFO   | jvm 1    | 2018/10/29 12:56:46 | 	at org.apache.directory.server.wrapper.ApacheDsTanukiWrapper.stop(ApacheDsTanukiWrapper.java:205)
> INFO   | jvm 1    | 2018/10/29 12:56:46 | 	at org.tanukisoftware.wrapper.WrapperManager$13.run(WrapperManager.java:3134)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
For additional commands, e-mail: dev-help@directory.apache.org


Mime
View raw message