spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anjana Fernando <>
Subject Re: Custom Cluster Managers / Standalone Recovery Mode in Spark
Date Mon, 02 Feb 2015 02:09:59 GMT
Hi guys,

That's great to hear that this is available in Spark 1.3! .. I will play
around with this feature and let you know the results for integrating
Hazelcast. Also, may I know the tentative release date for Spark 1.3? ..


On Mon, Feb 2, 2015 at 3:07 AM, Aaron Davidson <> wrote:

> For the specific question of supplementing Standalone Mode with a custom
> leader election protocol, this was actually already committed in master and
> will be available in Spark 1.3:
> You can specify spark.deploy.recoveryMode = "CUSTOM"
> and spark.deploy.recoveryMode.factory to a class which
> implements StandaloneRecoveryModeFactory. See the current implementations
> of FileSystemRecoveryModeFactory and ZooKeeperRecoveryModeFactory.
> I will update the JIRA you linked to be more current.
> On Sat, Jan 31, 2015 at 12:55 AM, Anjana Fernando <>
> wrote:
>> Hi everyone,
>> I've been experimenting, and somewhat of a newbie for Spark. I was
>> wondering, if there is any way, that I can use a custom cluster manager
>> implementation with Spark. Basically, as I understood, at the moment, the
>> inbuilt modes supported are with standalone, Mesos and  Yarn. My
>> requirement is basically a simple clustering solution with high
>> availability of the master. I don't want to use a separate Zookeeper
>> cluster, since this would complicate my deployment, but rather, I would
>> like to use something like Hazelcast, which has a peer-to-peer cluster
>> coordination implementation.
>> I found that, there is already this JIRA [1], which requests for a custom
>> persistence engine, I guess for storing state information. So basically,
>> what I would want to do is, use Hazelcast to use for leader election, to
>> make an existing node the master, and to lookup the state information from
>> the distributed memory. Appreciate any help on how to archive this. And if
>> it useful for a wider audience, hopefully I can contribute this back to
>> the
>> project.
>> [1]
>> Cheers,
>> Anjana.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message