spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <so...@cloudera.com>
Subject Re: How to force parallel processing of RDD using multiple thread
Date Thu, 15 Jan 2015 21:29:03 GMT
What is your cluster manager? For example on YARN you would specify
--executor-cores. Read:
http://spark.apache.org/docs/latest/running-on-yarn.html

On Thu, Jan 15, 2015 at 8:54 PM, Wang, Ningjun (LNG-NPV)
<ningjun.wang@lexisnexis.com> wrote:
> I have a standalone spark cluster with only one node with 4 CPU cores. How
> can I force spark to do parallel processing of my RDD using multiple
> threads? For example I can do the following
>
>
>
> Spark-submit  --master local[4]
>
>
>
> However I really want to use the cluster as follow
>
>
>
> Spark-submit  --master spark://10.125.21.15:7070
>
>
>
> In that case, how can I make sure the RDD is processed with multiple
> threads/cores?
>
>
>
> Thanks
>
> Ningjun
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message