spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boromir Widas <vcsub...@gmail.com>
Subject Re: Apache Spark standalone mode: number of cores
Date Fri, 23 Jan 2015 23:12:11 GMT
The local mode still parallelizes calculations and it is useful for
debugging as it goes through the steps of serialization/deserialization as
a cluster would.

On Fri, Jan 23, 2015 at 5:44 PM, olegshirokikh <oleg@solver.com> wrote:

> I'm trying to understand the basics of Spark internals and Spark
> documentation for submitting applications in local mode says for
> spark-submit --master setting:
>
> local[K] Run Spark locally with K worker threads (ideally, set this to the
> number of cores on your machine).
>
> local[*] Run Spark locally with as many worker threads as logical cores on
> your machine.
> Since all the data is stored on a single local machine, it does not benefit
> from distributed operations on RDDs.
>
> How does it benefit and what internally is going on when Spark utilizes
> several logical cores?
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Apache-Spark-standalone-mode-number-of-cores-tp21342.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message