spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Lewis <>
Subject How to I get at a SparkContext or better a JavaSparkContext from the middle of a function
Date Tue, 14 Oct 2014 23:47:50 GMT
 I am running a couple of functions on an RDD which require access to data
on the file system known to the context. If I create a class with a context
a a member variable I get a serialization error,

So I am running my function on some slave and I want to read in data from a
Path defined by a string and easy to read from my driver program
Later in the program the same issue arises in writing data except I would
like to access a tast attempt string so different attempts do not write the
same file.

How can a piece of java code running inside a function on some slave get at
the Task?

View raw message