spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James King <jakwebin...@gmail.com>
Subject Re: Enabling Event Log
Date Fri, 01 May 2015 08:13:39 GMT
Oops! well spotted. Many thanks Shixiong.

On Fri, May 1, 2015 at 1:25 AM, Shixiong Zhu <zsxwing@gmail.com> wrote:

> "spark.history.fs.logDirectory" is for the history server. For Spark
> applications, they should use "spark.eventLog.dir". Since you commented out
> "spark.eventLog.dir", it will be "/tmp/spark-events". And this folder does
> not exits.
>
> Best Regards,
> Shixiong Zhu
>
> 2015-04-29 23:22 GMT-07:00 James King <jakwebinbox@gmail.com>:
>
> I'm unclear why I'm getting this exception.
>>
>> It seems to have realized that I want to enable  Event Logging but
>> ignoring where I want it to log to i.e. file:/opt/cb/tmp/spark-events which
>> does exist.
>>
>> spark-default.conf
>>
>> # Example:
>> spark.master                     spark://master1:7077,master2:7077
>> spark.eventLog.enabled           true
>> spark.history.fs.logDirectory    file:/opt/cb/tmp/spark-events
>> # spark.eventLog.dir               hdfs://namenode:8021/directory
>> # spark.serializer
>> org.apache.spark.serializer.KryoSerializer
>> # spark.driver.memory              5g
>> # spark.executor.extraJavaOptions  -XX:+PrintGCDetails -Dkey=value
>> -Dnumbers="one two three"
>>
>> Exception following job submission:
>>
>> spark.eventLog.enabled=true
>> spark.history.fs.logDirectory=file:/opt/cb/tmp/spark-events
>>
>> spark.jars=file:/opt/cb/scripts/spark-streamer/cb-spark-streamer-1.0-SNAPSHOT.jar
>> spark.master=spark://master1:7077,master2:7077
>> Exception in thread "main" java.lang.IllegalArgumentException: Log
>> directory /tmp/spark-events does not exist.
>>         at
>> org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:99)
>>         at org.apache.spark.SparkContext.<init>(SparkContext.scala:399)
>>         at
>> org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:642)
>>         at
>> org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:75)
>>         at
>> org.apache.spark.streaming.api.java.JavaStreamingContext.<init>(JavaStreamingContext.scala:132)
>>
>>
>> Many Thanks
>> jk
>>
>
>

Mime
View raw message