spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From htailor <hemant.tai...@live.co.uk>
Subject Pyspark not using all cores
Date Tue, 10 Mar 2015 15:30:46 GMT
Hi All,

I need some help with a problem in pyspark which is causing a major issue. 

Recently I've noticed that the behaviour of the python.deamons on the worker
nodes for compute-intensive tasks have changed from using all the avaliable
cores to using only a single core. On each worker node, 8 python.deamons
exist but they all seem to run on a single core. The remaining 7 cores idle.

Our hardware consists of 9 hosts (1x driver node and 8x worker nodes) each
with 8 cores and 64gb RAM - we are using Spark 1.2.0-SNAPSHOT (Python) in
standalone mode via Cloudera 5.3.2. 

To give a better understanding of the problem I have made a quick script
from one of the given examples which replicates the problem:

When I run the calculating_pies.py script using the command "spark-submit
calculate_pies.py" this is what I typically see on all my worker nodes:

  1  [|||||||||||||||||||||||||||||||||||||||||||||||||||100.0%]      5 
[|||                                                  3.9%]
  2  [                                                                     
0.0%]      6  [|||                                                  2.6%]
  3  [||                                                                   
1.3%]     7  [|||||                                                8.4%]
  4  [||                                                                   
1.3%]     8  [||||||                                               8.7%]

  PID USER      PRI  NI  VIRT   RES    SHR S CPU% MEM%   TIME+  Command
30672 spark    20   0   225M  112M  1156 R 13.0   0.2       0:03.10 python
-m pyspark.daemon
30681 spark    20   0   225M  112M  1152 R 13.0   0.2       0:03.10 python
-m pyspark.daemon
30687 spark    20   0   225M  112M  1152 R 13.0   0.2       0:03.10 python
-m pyspark.daemon
30678 spark    20   0   225M  112M  1152 R 12.0   0.2       0:03.10 python
-m pyspark.daemon
30693 spark    20   0   225M  112M  1152 R 12.0   0.2       0:03.08 python
-m pyspark.daemon
30674 spark    20   0   225M  112M  1152 R 12.0   0.2       0:03.10 python
-m pyspark.daemon
30688 spark    20   0   225M  112M  1152 R 12.0   0.2       0:03.08 python
-m pyspark.daemon
30684 spark    20   0   225M  112M  1152 R 12.0   0.2       0:03.10 python
-m pyspark.daemon

Through the spark UI I do see 8 executor ids with 8 active tasks on each. I
also see the same behaviour if I use the flag --total-executor-cores 64 in
spark-submit.

Strangly, If I run the same script in local mode everything seems to run
fine. This is what I see

  1  [||||||||||||||||||||||||||||||||||||||||||||||100.0%]     5 
[||||||||||||||||||||||||||||||||||||||||||||||100.0%]
  2  [||||||||||||||||||||||||||||||||||||||||||||||100.0%]     6 
[||||||||||||||||||||||||||||||||||||||||||||||100.0%]
  3  [||||||||||||||||||||||||||||||||||||||||||||||100.0%]     7 
[||||||||||||||||||||||||||||||||||||||||||||||100.0%]
  4  [|||||||||||||||||||||||||||||||||||||||||||||||99.3%]     8 
[|||||||||||||||||||||||||||||||||||||||||||||||99.4%]

  PID USER      PRI  NI  VIRT   RES    SHR  S CPU% MEM%   TIME+  Command
22519 data       20   0  225M  106M  1368 R 99.0   0.2        0:10.97 python
-m pyspark.daemon
22508 data       20   0  225M  106M  1368 R 99.0   0.2        0:10.92 python
-m pyspark.daemon
22513 data       20   0  225M  106M  1368 R 99.0   0.2        0:11.02 python
-m pyspark.daemon
22526 data       20   0  225M  106M  1368 R 99.0   0.2        0:10.84 python
-m pyspark.daemon
22522 data       20   0  225M  106M  1368 R 98.0   0.2        0:10.95 python
-m pyspark.daemon
22523 data       20   0  225M  106M  1368 R 97.0   0.2        0:10.92 python
-m pyspark.daemon
22507 data       20   0  225M  106M  1368 R 97.0   0.2        0:10.83 python
-m pyspark.daemon
22516 data       20   0  225M  106M  1368 R 93.0   0.2        0:10.88 python
-m pyspark.daemon


============== calculating_pies.py ==================

#!/usr/bin/pyspark

import random
from pyspark.conf import SparkConf
from pyspark.context import SparkContext
from pyspark.storagelevel import StorageLevel

def pi(NUM_SAMPLE = 1000000):
    count = 0.
    for i in xrange(NUM_SAMPLE):
        x, y = random.random(), random.random()
        if (x * x) + (y * y) < 1:
            count += 1
    return 4.0 * (count / NUM_SAMPLE)

if __name__ == "__main__":

    sconf = (SparkConf()
             .set('spark.default.parallelism','256')
             .set('spark.app.name', 'Calculating PI'))

    # local
    # sc = SparkContext(conf=sconf)

    # standalone
    sc = SparkContext("spark://<driver_host>:7077", conf=sconf)

    # yarn
    # sc = SparkContext("yarn-client", conf=sconf)

    rdd_pies = sc.parallelize(range(10000), 1000)
    rdd_pies.map(lambda x: pi()).collect()
    sc.stop()

=====================================================

Does anyone have any suggestions, or know of any config we should be looking
at that could solve this problem? Does anyone else see the same problem?

Any help is appreciated, Thanks.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Pyspark-not-using-all-cores-tp21989.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message