spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan <>
Subject RE: Problems with Local Checkpoints
Date Mon, 14 Sep 2015 10:58:13 GMT

This looks like the issue. I'll update my path to include the (soon to be added) winutils
& assoc. DLLs.

Thank you,


-----Original Message-----
From: "Akhil Das" <>
Sent: ‎9/‎14/‎2015 6:46 AM
To: "Bryan Jeffrey" <>
Cc: "user" <>
Subject: Re: Problems with Local Checkpoints

You need to set your HADOOP_HOME and make sure the winutils.exe is available in the PATH.

Here's a discussion around the same issue
Also this JIRA

Best Regards

On Wed, Sep 9, 2015 at 11:30 PM, Bryan Jeffrey <> wrote:


I have some basic code that counts numbers using updateStateByKey.  I setup a streaming context
with checkpointing as follows:

def createStreamingContext(masterName : String, checkpointDirectory : String, timeWindow :
Int) : StreamingContext = {  val sparkConf = new SparkConf().setAppName("Program")  val ssc
= new StreamingContext(sparkConf, Seconds(timeWindow))  ssc.checkpoint(checkpointDirectory)

This runs fine on my distributed (Linux) cluster, writing checkpoints to local disk. However,
when I run on my Windows desktop I am seeing a number of checkpoint errors:

15/09/09 13:57:06 INFO CheckpointWriter: Saving checkpoint for time 1441821426000 ms to file
Exception in thread "pool-14-thread-4" java.lang.NullPointerException
 at java.lang.ProcessBuilder.start(
 at org.apache.hadoop.util.Shell.runCommand(
 at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(
 at org.apache.hadoop.util.Shell.execCommand(
 at org.apache.hadoop.util.Shell.execCommand(
 at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(
 at org.apache.hadoop.fs.FilterFileSystem.setPermission(
 at org.apache.hadoop.fs.ChecksumFileSystem.create(
 at org.apache.hadoop.fs.ChecksumFileSystem.create(
 at org.apache.hadoop.fs.FileSystem.create(
 at org.apache.hadoop.fs.FileSystem.create(
 at org.apache.hadoop.fs.FileSystem.create(
 at org.apache.hadoop.fs.FileSystem.create(
 at org.apache.spark.streaming.CheckpointWriter$
 at java.util.concurrent.ThreadPoolExecutor.runWorker(
 at java.util.concurrent.ThreadPoolExecutor$

JAVA_HOME is set correctly, the code runs correctly, it's not a permissions issue (I've run
this as Administrator).  Directories and files are being created in C:\Temp, although all
of the files appear to be empty.

Does anyone have an idea of what is causing these errors?  Has anyone seen something similar?


Bryan Jeffrey
View raw message