spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shixiong Zhu <zsxw...@gmail.com>
Subject Re: Enabling Event Log
Date Fri, 01 May 2015 00:25:13 GMT
"spark.history.fs.logDirectory" is for the history server. For Spark
applications, they should use "spark.eventLog.dir". Since you commented out
"spark.eventLog.dir", it will be "/tmp/spark-events". And this folder does
not exits.

Best Regards,
Shixiong Zhu

2015-04-29 23:22 GMT-07:00 James King <jakwebinbox@gmail.com>:

> I'm unclear why I'm getting this exception.
>
> It seems to have realized that I want to enable  Event Logging but
> ignoring where I want it to log to i.e. file:/opt/cb/tmp/spark-events which
> does exist.
>
> spark-default.conf
>
> # Example:
> spark.master                     spark://master1:7077,master2:7077
> spark.eventLog.enabled           true
> spark.history.fs.logDirectory    file:/opt/cb/tmp/spark-events
> # spark.eventLog.dir               hdfs://namenode:8021/directory
> # spark.serializer
> org.apache.spark.serializer.KryoSerializer
> # spark.driver.memory              5g
> # spark.executor.extraJavaOptions  -XX:+PrintGCDetails -Dkey=value
> -Dnumbers="one two three"
>
> Exception following job submission:
>
> spark.eventLog.enabled=true
> spark.history.fs.logDirectory=file:/opt/cb/tmp/spark-events
>
> spark.jars=file:/opt/cb/scripts/spark-streamer/cb-spark-streamer-1.0-SNAPSHOT.jar
> spark.master=spark://master1:7077,master2:7077
> Exception in thread "main" java.lang.IllegalArgumentException: Log
> directory /tmp/spark-events does not exist.
>         at
> org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:99)
>         at org.apache.spark.SparkContext.<init>(SparkContext.scala:399)
>         at
> org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:642)
>         at
> org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:75)
>         at
> org.apache.spark.streaming.api.java.JavaStreamingContext.<init>(JavaStreamingContext.scala:132)
>
>
> Many Thanks
> jk
>

Mime
View raw message