spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Debasish Das <>
Subject Gossip protocol in Master selection
Date Mon, 29 Jun 2015 01:42:38 GMT

Akka cluster uses gossip protocol for Master election. The approach in
Spark right now is to use Zookeeper for high availability.

Interestingly Cassandra and Redis clusters are both using Gossip protocol.

I am not sure what is the default behavior right now. If the master dies
and zookeeper selects a new master, the whole depedency graph will be
re-executed or only the unfinished stages will be restarted ?

Also why the zookeeper based HA was preferred in Spark ? I was wondering if
there is JIRA to add gossip protocol for Spark Master election ?

In the code I see zookeeper, filesystem, custom and default is
MonarchyLeader. So looks like Spark is designed to add new


View raw message