spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ian O'Connell" <...@ianoconnell.com>
Subject Re: Some questions about spark-0.7.0
Date Sat, 03 Aug 2013 00:16:14 GMT
There's a closure cleaner that attempts to find variables you've referenced
in a closure it ships around to package them up. those it will send a copy
of, so you'll see serialization errors if you use things that are outside
the scope and aren't serializable(IO things often)


On Fri, Aug 2, 2013 at 3:45 PM, Shangyu Luo <lsyurd@gmail.com> wrote:

> Thank you for your answers!
> Moreover, I am wondering how spark deals with global variables? Will it
> send a copy of global variables to each worker node?
> Thanks.
>
>
> 2013/8/2 Ian O'Connell <ian@ianoconnell.com>
>
>> You can run local mode on a single machine, but really to run master and
>> workers on one machine you'll need to use virtual machines or suffer alot
>> of hassle. Of course there is no benefit to such approach for performance
>> just experiment purposes.
>>
>> It depends on the maps involved, you cannot have nested RDD's or nested
>> RDD operations if thats what you mean
>>
>>
>> On Fri, Aug 2, 2013 at 2:22 PM, Shangyu Luo <lsyurd@gmail.com> wrote:
>>
>>> Hello, I am using spark-0.7.0 python version and I have two questions
>>> about it.
>>> 1. Does spark-0.7.0 support nested use of map function and reduce
>>> function? For example, a function is a parameter for a map and this
>>> function also contains a map.
>>> 2. Does spark-0.7.0 support setting up one master node and multiple
>>> worker nodes on one computer by using different ports? I can set up one
>>> master node and one worker node on one computer, but when I try to set up
>>> another worker node, there will be conflict of port 8081 error.
>>> Any answers or suggestions will be appreciated.
>>> Thanks.
>>>
>>>
>>> --
>>> --
>>>
>>> Shangyu, Luo
>>> Department of Computer Science
>>> Rice University
>>>
>>> --
>>> Not Just Think About It, But Do It!
>>> --
>>> Success is never final.
>>> --
>>> Losers always whine about their best
>>>
>>
>>
>
>
> --
> --
>
> Shangyu, Luo
> Department of Computer Science
> Rice University
>
> --
> Not Just Think About It, But Do It!
> --
> Success is never final.
> --
> Losers always whine about their best
>

Mime
View raw message