spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: passing SparkContext as parameter
Date Mon, 21 Sep 2015 11:40:04 GMT
You can use broadcast variable for passing connection information. 

Cheers

> On Sep 21, 2015, at 4:27 AM, Priya Ch <learnings.chitturi@gmail.com> wrote:
> 
> can i use this sparkContext on executors ??
> In my application, i have scenario of reading from db for certain records in rdd. Hence
I need sparkContext to read from DB (cassandra in our case),
> 
> If sparkContext couldn't be sent to executors , what is the workaround for this ??????
> 
>> On Mon, Sep 21, 2015 at 3:06 PM, Petr Novak <oss.mlists@gmail.com> wrote:
>> add @transient?
>> 
>>> On Mon, Sep 21, 2015 at 11:27 AM, Priya Ch <learnings.chitturi@gmail.com>
wrote:
>>> Hello All,
>>> 
>>>     How can i pass sparkContext as a parameter to a method in an object. Because
passing sparkContext is giving me TaskNotSerializable Exception.
>>> 
>>> How can i achieve this ?
>>> 
>>> Thanks,
>>> Padma Ch
> 

Mime
View raw message