spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Lewis <lordjoe2...@gmail.com>
Subject Stupid Spark question
Date Tue, 07 Oct 2014 18:01:22 GMT
 I am porting a Hadoop job to Spark - One issue is that the workers need to
read files from hdfs reading a different file based on the key or in some
cases reading an object that is expensive to serialize.
This is easy if the worker has  access to the JavaSparkContext (I am
working in Java) but this cannot be serialized -
how can a worker read from a Path - assume hdfs

Mime
View raw message