spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shangyu Luo <lsy...@gmail.com>
Subject Re: Some questions about spark-0.7.0
Date Fri, 02 Aug 2013 22:45:01 GMT
Thank you for your answers!
Moreover, I am wondering how spark deals with global variables? Will it
send a copy of global variables to each worker node?
Thanks.


2013/8/2 Ian O'Connell <ian@ianoconnell.com>

> You can run local mode on a single machine, but really to run master and
> workers on one machine you'll need to use virtual machines or suffer alot
> of hassle. Of course there is no benefit to such approach for performance
> just experiment purposes.
>
> It depends on the maps involved, you cannot have nested RDD's or nested
> RDD operations if thats what you mean
>
>
> On Fri, Aug 2, 2013 at 2:22 PM, Shangyu Luo <lsyurd@gmail.com> wrote:
>
>> Hello, I am using spark-0.7.0 python version and I have two questions
>> about it.
>> 1. Does spark-0.7.0 support nested use of map function and reduce
>> function? For example, a function is a parameter for a map and this
>> function also contains a map.
>> 2. Does spark-0.7.0 support setting up one master node and multiple
>> worker nodes on one computer by using different ports? I can set up one
>> master node and one worker node on one computer, but when I try to set up
>> another worker node, there will be conflict of port 8081 error.
>> Any answers or suggestions will be appreciated.
>> Thanks.
>>
>>
>> --
>> --
>>
>> Shangyu, Luo
>> Department of Computer Science
>> Rice University
>>
>> --
>> Not Just Think About It, But Do It!
>> --
>> Success is never final.
>> --
>> Losers always whine about their best
>>
>
>


-- 
--

Shangyu, Luo
Department of Computer Science
Rice University

--
Not Just Think About It, But Do It!
--
Success is never final.
--
Losers always whine about their best

Mime
View raw message