spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anjana Fernando <laferna...@gmail.com>
Subject Re: Custom Cluster Managers / Standalone Recovery Mode in Spark
Date Mon, 02 Feb 2015 02:09:59 GMT
Hi guys,

That's great to hear that this is available in Spark 1.3! .. I will play
around with this feature and let you know the results for integrating
Hazelcast. Also, may I know the tentative release date for Spark 1.3? ..

Cheers,
Anjana.

On Mon, Feb 2, 2015 at 3:07 AM, Aaron Davidson <ilikerps@gmail.com> wrote:

> For the specific question of supplementing Standalone Mode with a custom
> leader election protocol, this was actually already committed in master and
> will be available in Spark 1.3:
>
> https://github.com/apache/spark/pull/771/files
>
> You can specify spark.deploy.recoveryMode = "CUSTOM"
> and spark.deploy.recoveryMode.factory to a class which
> implements StandaloneRecoveryModeFactory. See the current implementations
> of FileSystemRecoveryModeFactory and ZooKeeperRecoveryModeFactory.
>
> I will update the JIRA you linked to be more current.
>
> On Sat, Jan 31, 2015 at 12:55 AM, Anjana Fernando <lafernando@gmail.com>
> wrote:
>
>> Hi everyone,
>>
>> I've been experimenting, and somewhat of a newbie for Spark. I was
>> wondering, if there is any way, that I can use a custom cluster manager
>> implementation with Spark. Basically, as I understood, at the moment, the
>> inbuilt modes supported are with standalone, Mesos and  Yarn. My
>> requirement is basically a simple clustering solution with high
>> availability of the master. I don't want to use a separate Zookeeper
>> cluster, since this would complicate my deployment, but rather, I would
>> like to use something like Hazelcast, which has a peer-to-peer cluster
>> coordination implementation.
>>
>> I found that, there is already this JIRA [1], which requests for a custom
>> persistence engine, I guess for storing state information. So basically,
>> what I would want to do is, use Hazelcast to use for leader election, to
>> make an existing node the master, and to lookup the state information from
>> the distributed memory. Appreciate any help on how to archive this. And if
>> it useful for a wider audience, hopefully I can contribute this back to
>> the
>> project.
>>
>> [1] https://issues.apache.org/jira/browse/SPARK-1180
>>
>> Cheers,
>> Anjana.
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message