spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niranda Perera <niranda.per...@gmail.com>
Subject Re: Error in invoking a custom StandaloneRecoveryModeFactory in java env (Spark v1.3.0)
Date Thu, 25 Jun 2015 06:12:51 GMT
thanks Josh.

this looks very similar to my problem.

On Thu, Jun 25, 2015 at 11:32 AM, Josh Rosen <rosenville@gmail.com> wrote:

> This sounds like https://issues.apache.org/jira/browse/SPARK-7436, which
> has been fixed in Spark 1.4+ and in branch-1.3 (for Spark 1.3.2).
>
> On Wed, Jun 24, 2015 at 10:57 PM, Niranda Perera <niranda.perera@gmail.com
> > wrote:
>
>> Hi all,
>>
>> I'm trying to implement a custom StandaloneRecoveryModeFactory in the
>> Java environment. Pls find the implementation here. [1] . I'm new to Scala,
>> hence I'm trying to use Java environment as much as possible.
>>
>> when I start a master with spark.deploy.recoveryMode.factory property to
>> be "CUSTOM", I encounter a NoSuchMethodException for my custom class's
>> constructor.
>> it has the following constructor.
>>
>>  public AnalyticsStandaloneRecoveryModeFactory(SparkConf conf,
>> Serialization serializer)
>>
>> but from the Master, it looks for a constructor for,
>> org.wso2.carbon.analytics.spark.core.util.master.AnalyticsStandaloneRecoveryModeFactory.<init>(org.apache.spark.SparkConf,
>> akka.serialization.Serialization$)
>>
>> I see in the Spark source code for Master, that it uses reflection to get
>> the custom recovery mode factory class.
>>
>> case "CUSTOM" =>
>>         val clazz =
>> Class.forName(conf.get("spark.deploy.recoveryMode.factory"))
>>         val factory = clazz.getConstructor(conf.getClass,
>> Serialization.getClass)
>>           .newInstance(conf, SerializationExtension(context.system))
>>           .asInstanceOf[StandaloneRecoveryModeFactory]
>>         (factory.createPersistenceEngine(),
>> factory.createLeaderElectionAgent(this))
>>
>> here, Serialization.getClass returns a akka.serialization.Serialization$
>> object, where as my custom class's constructor
>> accepts akka.serialization.Serialization object.
>>
>> so I would like to know,
>> 1. if this happens because I'm using this in the Java environment?
>> 2. what is the workaround to this?
>>
>> thanks
>>
>> Please find the full stack trace of the error below.
>>
>> [2015-06-25 10:59:01,095] ERROR {akka.actor.OneForOneStrategy} -
>>  org.wso2.carbon.analytics.spark.core.util.master.AnalyticsStandaloneRecoveryModeFactory.<init>(org.apache.spark.SparkConf,
>> akka.serialization.Serialization$)
>> akka.actor.ActorInitializationException: exception during creation
>> at akka.actor.ActorInitializationException$.apply(Actor.scala:164)
>> at akka.actor.ActorCell.create(ActorCell.scala:596)
>> at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
>> at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
>> at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
>> at akka.dispatch.Mailbox.run(Mailbox.scala:219)
>> at
>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
>> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> Caused by: java.lang.NoSuchMethodException:
>> org.wso2.carbon.analytics.spark.core.util.master.AnalyticsStandaloneRecoveryModeFactory.<init>(org.apache.spark.SparkConf,
>> akka.serialization.Serialization$)
>> at java.lang.Class.getConstructor0(Class.java:2810)
>> at java.lang.Class.getConstructor(Class.java:1718)
>> at org.apache.spark.deploy.master.Master.preStart(Master.scala:165)
>> at akka.actor.Actor$class.aroundPreStart(Actor.scala:470)
>> at org.apache.spark.deploy.master.Master.aroundPreStart(Master.scala:52)
>> at akka.actor.ActorCell.create(ActorCell.scala:580)
>> ... 9 more
>>
>>
>>
>> [1]
>> https://github.com/nirandaperera/carbon-analytics/blob/spark_master_persistance/components/analytics-processors/org.wso2.carbon.analytics.spark.core/src/main/java/org/wso2/carbon/analytics/spark/core/util/master/AnalyticsStandaloneRecoveryModeFactory.java
>> <https://github.com/nirandaperera/carbon-analytics/blob/spark_master_persistance/components/analytics-processors/org.wso2.carbon.analytics.spark.core/src/main/java/org/wso2/carbon/analytics/spark/core/util/master/AnalyticsStandaloneRecoveryModeFactory.java>
>>
>> --
>> Niranda
>> @n1r44 <https://twitter.com/N1R44>
>> https://pythagoreanscript.wordpress.com/
>>
>
>


-- 
Niranda
@n1r44 <https://twitter.com/N1R44>
https://pythagoreanscript.wordpress.com/

Mime
View raw message