spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Deep Pradhan <>
Subject Re: Worker and Nodes
Date Sat, 21 Feb 2015 15:01:43 GMT
In this case, I just wanted to know if a single node cluster with various
workers act like a simulator of a multi-node cluster with various nodes.
Like, if we have a single node cluster with 10 workers, say, then can we
tell that the same behavior will take place with cluster of 10 nodes?
It is like, without having the 10 nodes cluster, I can know the behavior of
the application in 10 nodes cluster by having a single node with 10
workers. The time taken may vary but I am talking about the behavior. Can
we say that?

On Sat, Feb 21, 2015 at 8:21 PM, Deep Pradhan <>

> Yes, I am talking about standalone single node cluster.
> No, I am not increasing parallelism. I just wanted to know if it is
> natural. Does message passing across the workers account for the happenning?
> I am running SparkKMeans, just to validate one prediction model. I am
> using several data sets. I have a standalone mode. I am varying the workers
> from 1 to 16
> On Sat, Feb 21, 2015 at 8:14 PM, Sean Owen <> wrote:
>> I can imagine a few reasons. Adding workers might cause fewer tasks to
>> execute locally (?) So you may be execute more remotely.
>> Are you increasing parallelism? for trivial jobs, chopping them up
>> further may cause you to pay more overhead of managing so many small
>> tasks, for no speed up in execution time.
>> Can you provide any more specifics though? you haven't said what
>> you're running, what mode, how many workers, how long it takes, etc.
>> On Sat, Feb 21, 2015 at 2:37 PM, Deep Pradhan <>
>> wrote:
>> > Hi,
>> > I have been running some jobs in my local single node stand alone
>> cluster. I
>> > am varying the worker instances for the same job, and the time taken
>> for the
>> > job to complete increases with increase in the number of workers. I
>> repeated
>> > some experiments varying the number of nodes in a cluster too and the
>> same
>> > behavior is seen.
>> > Can the idea of worker instances be extrapolated to the nodes in a
>> cluster?
>> >
>> > Thank You

View raw message