helix-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HELIX-690) Batch message should not share same NotificationContext object to update CurrentState
Date Thu, 19 Apr 2018 18:10:00 GMT

    [ https://issues.apache.org/jira/browse/HELIX-690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16444513#comment-16444513
] 

ASF GitHub Bot commented on HELIX-690:
--------------------------------------

Github user asfgit closed the pull request at:

    https://github.com/apache/helix/pull/181


> Batch message should not share same NotificationContext object to update CurrentState
> -------------------------------------------------------------------------------------
>
>                 Key: HELIX-690
>                 URL: https://issues.apache.org/jira/browse/HELIX-690
>             Project: Apache Helix
>          Issue Type: Bug
>            Reporter: Hao Zhang
>            Priority: Major
>
> Currently batch message has bugs:
>  1. Batch message is triggering a lot of duplicated state transition messages sent from
controller, result in "state does not match" error on participant side. This will further
create a lot of ERROR znodes in ZK, which adds up both read/write workload in participant
and controller
> 2. We see a lot of concurrent update exceptions as well
> {noformat}
> 9909348:[2018-03-30 18:59:55,025] [ERROR] [pool-1-thread-1917] [org.apache.helix.messaging.handling.HelixTask:113]
- Exception while executing a message. java.util.ConcurrentModificat
> ionException msgId: fbdc37d4-ec95-47cb-950c-f9d3d224bbb3 type: STATE_TRANSITION
> 9909349-java.util.ConcurrentModificationException
> 9909350- at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1115)
> 9909351- at java.util.TreeMap$KeyIterator.next(TreeMap.java:1169)
> 9909352- at org.apache.helix.ZNRecord.merge(ZNRecord.java:497)
> 9909353- at org.apache.helix.GroupCommit.commit(GroupCommit.java:121)
> 9909354- at org.apache.helix.manager.zk.ZKHelixDataAccessor.updateProperty(ZKHelixDataAccessor.java:182)
> 9909355- at org.apache.helix.manager.zk.ZKHelixDataAccessor.updateProperty(ZKHelixDataAccessor.java:170)
> 9909356- at org.apache.helix.messaging.handling.BatchMessageHandler.postHandleMessage(BatchMessageHandler.java:118)
> 9909357- at org.apache.helix.messaging.handling.BatchMessageHandler.handleMessage(BatchMessageHandler.java:203)
> 9909358- at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:96)
> {noformat}
> The above 2 errors are resulted in the fact that in HelixTaskExecutor, all HelixTask
objects from same batch of messages are sharing the same changeContext object. For batch message,
HelixTask will create current state update map to record current state updates, and therefore
result in a racing condition in current state recording - it is very normal that due to such
bug, resource's current state is changed on participant side, current state is not updated
in ZK, and after message is removed, controller still think that state transition is not finished,
and send duplicated state transition message.
>  
> The error situation will only be triggered when the load is high, so not covered by our
unit / e2e tests
> To fix the issue, we should create deep copies of NotificationContext object for each
HelixTask in HelixTaskExecutor. I tried this fix using large data sets, and it worked.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message