spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: Why does this siimple spark program uses only one core?
Date Sun, 09 Nov 2014 17:27:19 GMT
You can set the following entry inside the conf/spark-defaults.conf file

spark.cores.max 16


If you want to read the default value, then you can use the following api
call

*sc*.defaultParallelism

where ​*sc* is your sparkContext object.​


Thanks
Best Regards

On Sun, Nov 9, 2014 at 6:48 PM, ReticulatedPython <person.of.book@gmail.com>
wrote:

> So, I'm running this simple program on a 16 core multicore system. I run it
> by issuing the following.
>
> spark-submit --master local[*] pi.py
>
> And the code of that program is the following. When I use top to see CPU
> consumption, only 1 core is being utilized. Why is it so? Seconldy, spark
> documentation says that the default parallelism is contained in property
> spark.default.parallelism. How can I read this property from within my
> python program?
>
> #"""pi.py"""
> from pyspark import SparkContext
> import random
>
> NUM_SAMPLES = 12500000
>
> def sample(p):
>     x, y = random.random(), random.random()
>     return 1 if x*x + y*y < 1 else 0
>
> sc = SparkContext("local", "Test App")
> count = sc.parallelize(xrange(0, NUM_SAMPLES)).map(sample).reduce(lambda a,
> b: a + b)
> print "Pi is roughly %f" % (4.0 * count / NUM_SAMPLES)
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Why-does-this-siimple-spark-program-uses-only-one-core-tp18434.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message