spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shangyu Luo <>
Subject The functionality of
Date Tue, 08 Oct 2013 04:50:26 GMT
I am using Spark 0.7.3 with python version.  Recently when I run some spark
program on a cluster, I found that some processes invoked
by spark-0.7.3/python/pyspark/ would capturing CPU for a long time
and consume much memory (e.g., 5g for each process). It seemed that the
java process, which was invoked by
java -cp
...  , was 'competing' with the for CPU resources. From my
understanding, the java process should be responsible for the 'real'
computation in spark.
So I am wondering what job the will work on? Is it normal for it
to consume a lot of CPU and memory?

Shangyu Luo

Shangyu, Luo
Department of Computer Science
Rice University

View raw message