ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Mekhanikov <dmekhani...@gmail.com>
Subject High priority TCP discovery messages
Date Thu, 10 Jan 2019 15:11:01 GMT
A bit of background:
When TcpDiscoverySpi is used, TcpDiscoveryMetricsUpdateMessage is sent by a
coordinator once in metricsUpdateFrequency, which is 2 seconds by default.
It serves as a ping message, which ensures, that the ring is connected, and
all nodes are alive. These messages have a high priority, i.e., they are
put into the head of the RightMessageWorker's queue instead of its tail.

Now consider a situation, when a single link between two nodes in the ring
works slowly. It may receive and deliver all messages, but with a certain
delay. This situation is possible, when network is unstable or one of the
nodes experiences a lot of short GC pauses.
It leads to a growing message queue on the node, that stands before the
slow link. The worst part is that if high priority messages are generated
faster than they are processed, then other messages won't even get a chance
to be processed. Thus, no nodes will be kicked out of the cluster, but no
useful progress will happen. Partition map exchange may hang for this
reason, and the reason won't be obvious from the logs.

JIRA ticket: https://issues.apache.org/jira/browse/IGNITE-10808
I made a draft of the fix for this problem:
https://github.com/apache/ignite/pull/5771
The PR also contains a test, that reproduces this situation.

The patch makes sure, that only a single TcpDiscoveryMetricsUpdateMessage
of one kind is stored in the queue, i.e., if a newer message comes to the
node, then the old one is discarded. It also checks, that regular messages
are also processed. If the last processed message had a high priority, then
new high-priority message won't be put to the head of the queue, but to the
tail instead.
The fix addresses the described problem, but increases the utilization of
the discovery threads a bit.

What do you think of this fix? Do you have better ideas, how to improve the
heartbeat mechanism to avoid such situations?

Denis

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message