spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Davidson <ilike...@gmail.com>
Subject Re: Custom Cluster Managers / Standalone Recovery Mode in Spark
Date Sun, 01 Feb 2015 21:37:57 GMT
For the specific question of supplementing Standalone Mode with a custom
leader election protocol, this was actually already committed in master and
will be available in Spark 1.3:

https://github.com/apache/spark/pull/771/files

You can specify spark.deploy.recoveryMode = "CUSTOM"
and spark.deploy.recoveryMode.factory to a class which
implements StandaloneRecoveryModeFactory. See the current implementations
of FileSystemRecoveryModeFactory and ZooKeeperRecoveryModeFactory.

I will update the JIRA you linked to be more current.

On Sat, Jan 31, 2015 at 12:55 AM, Anjana Fernando <lafernando@gmail.com>
wrote:

> Hi everyone,
>
> I've been experimenting, and somewhat of a newbie for Spark. I was
> wondering, if there is any way, that I can use a custom cluster manager
> implementation with Spark. Basically, as I understood, at the moment, the
> inbuilt modes supported are with standalone, Mesos and  Yarn. My
> requirement is basically a simple clustering solution with high
> availability of the master. I don't want to use a separate Zookeeper
> cluster, since this would complicate my deployment, but rather, I would
> like to use something like Hazelcast, which has a peer-to-peer cluster
> coordination implementation.
>
> I found that, there is already this JIRA [1], which requests for a custom
> persistence engine, I guess for storing state information. So basically,
> what I would want to do is, use Hazelcast to use for leader election, to
> make an existing node the master, and to lookup the state information from
> the distributed memory. Appreciate any help on how to archive this. And if
> it useful for a wider audience, hopefully I can contribute this back to the
> project.
>
> [1] https://issues.apache.org/jira/browse/SPARK-1180
>
> Cheers,
> Anjana.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message