spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <luohui20...@sina.com>
Subject 回复:Re: Spark Configuration of spark.worker.cleanup.appDataTtl
Date Tue, 16 Jun 2015 08:02:57 GMT
thanks saisai,        I should try more times. I thought it will be caculated automatically
as the default.


--------------------------------

 

Thanks&amp;Best regards!
San.Luo

----- 原始邮件 -----
发件人:Saisai Shao <sai.sai.shao@gmail.com>
收件人:罗辉 <luohui20001@sina.com>
抄送人:user <user@spark.apache.org>
主题:Re: Spark Configuration of spark.worker.cleanup.appDataTtl
日期:2015年06月16日 15点00分

I think you have to using "604800" instead of "7 * 24 * 3600",  obviously SparkConf will not
do multiplication for you..
The exception is quite obvious: "Caused by: java.lang.NumberFormatException: For input string:
"3 * 24 * 3600""
2015-06-16 14:52 GMT+08:00  <luohui20001@sina.com>:
Hi guys:       I added a parameter "spark.worker.cleanup.appDataTtl   3 * 24 * 3600" in my
conf/spark-default.conf, then I start my spark cluster. However I got an exception:
15/06/16 14:25:14 INFO util.Utils: Successfully started service 'sparkWorker' on port 43344.
15/06/16 14:25:14 ERROR actor.OneForOneStrategy: exception during creation
akka.actor.ActorInitializationException: exception during creation
        at akka.actor.ActorInitializationException$.apply(Actor.scala:164)
        at akka.actor.ActorCell.create(ActorCell.scala:596)
        at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
        at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
        at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
        at akka.dispatch.Mailbox.run(Mailbox.scala:219)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at akka.util.Reflect$.instantiate(Reflect.scala:66)
        at akka.actor.ArgsReflectConstructor.produce(Props.scala:352)
        at akka.actor.Props.newActor(Props.scala:252)
        at akka.actor.ActorCell.newActor(ActorCell.scala:552)
        at akka.actor.ActorCell.create(ActorCell.scala:578)
        ... 9 more
Caused by: java.lang.NumberFormatException: For input string: "3 * 24 * 3600"
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
        at java.lang.Long.parseLong(Long.java:589)
        at java.lang.Long.parseLong(Long.java:631)
        at scala.collection.immutable.StringLike$class.toLong(StringLike.scala:230)
        at scala.collection.immutable.StringOps.toLong(StringOps.scala:31)
        at org.apache.spark.SparkConf$$anonfun$getLong$2.apply(SparkConf.scala:194)
        at org.apache.spark.SparkConf$$anonfun$getLong$2.apply(SparkConf.scala:194)
        at scala.Option.map(Option.scala:145)
        at org.apache.spark.SparkConf.getLong(SparkConf.scala:194)
        at org.apache.spark.deploy.worker.Worker.<init>(Worker.scala:89)
        ... 18 more

How to set this parameter correctly?       
BTW, I searched this property in http://spark.apache.org/docs/latest/configuration.html and
got no match. This property was found in http://spark.apache.org/docs/latest/spark-standalone.html
with a default value "7 * 24 * 3600 (7 days)", which I also tried but also failed.Thanks
--------------------------------

 

Thanks&amp;Best regards!
San.Luo



Mime
View raw message