spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sven Krasser <kras...@gmail.com>
Subject Re: Large number of pyspark.daemon processes
Date Wed, 28 Jan 2015 01:58:59 GMT
After slimming down the job quite a bit, it looks like a call to coalesce()
on a larger RDD can cause these Python worker spikes (additional details in
Jira:
https://issues.apache.org/jira/browse/SPARK-5395?focusedCommentId=14294570&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14294570
).

Any ideas to what coalesce() is doing that triggers the creation of
additional workers?

On Sat, Jan 24, 2015 at 12:27 AM, Sven Krasser <krasser@gmail.com> wrote:

> Hey Davies,
>
> Sure thing, it's filed here now:
> https://issues.apache.org/jira/browse/SPARK-5395
>
> As far as a repro goes, what is a normal number of workers I should
> expect? Even shortly after kicking the job off, I see workers in the
> double-digits per container. Here's an example using pstree on a worker
> node (the worker node runs 16 containers, i.e. the java--bash branches
> below). The initial Python process per container is forked between 7 and 33
> times.
>
>      ├─java─┬─bash───java─┬─python───22*[python]
>      │      │             └─89*[{java}]
>      │      ├─bash───java─┬─python───13*[python]
>      │      │             └─80*[{java}]
>      │      ├─bash───java─┬─python───9*[python]
>      │      │             └─78*[{java}]
>      │      ├─bash───java─┬─python───24*[python]
>      │      │             └─91*[{java}]
>      │      ├─3*[bash───java─┬─python───19*[python]]
>      │      │                └─86*[{java}]]
>      │      ├─bash───java─┬─python───25*[python]
>      │      │             └─90*[{java}]
>      │      ├─bash───java─┬─python───33*[python]
>      │      │             └─98*[{java}]
>      │      ├─bash───java─┬─python───7*[python]
>      │      │             └─75*[{java}]
>      │      ├─bash───java─┬─python───15*[python]
>      │      │             └─80*[{java}]
>      │      ├─bash───java─┬─python───18*[python]
>      │      │             └─84*[{java}]
>      │      ├─bash───java─┬─python───10*[python]
>      │      │             └─85*[{java}]
>      │      ├─bash───java─┬─python───26*[python]
>      │      │             └─90*[{java}]
>      │      ├─bash───java─┬─python───17*[python]
>      │      │             └─85*[{java}]
>      │      ├─bash───java─┬─python───24*[python]
>      │      │             └─92*[{java}]
>      │      └─265*[{java}]
>
> Each container has 2 cores in case that makes a difference.
>
> Thank you!
> -Sven
>
> On Fri, Jan 23, 2015 at 11:52 PM, Davies Liu <davies@databricks.com>
> wrote:
>
>> It should be a bug, the Python worker did not exit normally, could you
>> file a JIRA for this?
>>
>> Also, could you show how to reproduce this behavior?
>>
>> On Fri, Jan 23, 2015 at 11:45 PM, Sven Krasser <krasser@gmail.com> wrote:
>> > Hey Adam,
>> >
>> > I'm not sure I understand just yet what you have in mind. My takeaway
>> from
>> > the logs is that the container actually was above its allotment of about
>> > 14G. Since 6G of that are for overhead, I assumed there to be plenty of
>> > space for Python workers, but there seem to be more of those than I'd
>> > expect.
>> >
>> > Does anyone know if that is actually the intended behavior, i.e. in this
>> > case over 90 Python processes on a 2 core executor?
>> >
>> > Best,
>> > -Sven
>> >
>> >
>> > On Fri, Jan 23, 2015 at 10:04 PM, Adam Diaz <adam.h.diaz@gmail.com>
>> wrote:
>> >>
>> >> Yarn only has the ability to kill not checkpoint or sig suspend.  If
>> you
>> >> use too much memory it will simply kill tasks based upon the yarn
>> config.
>> >> https://issues.apache.org/jira/browse/YARN-2172
>> >>
>> >>
>> >> On Friday, January 23, 2015, Sandy Ryza <sandy.ryza@cloudera.com>
>> wrote:
>> >>>
>> >>> Hi Sven,
>> >>>
>> >>> What version of Spark are you running?  Recent versions have a change
>> >>> that allows PySpark to share a pool of processes instead of starting
>> a new
>> >>> one for each task.
>> >>>
>> >>> -Sandy
>> >>>
>> >>> On Fri, Jan 23, 2015 at 9:36 AM, Sven Krasser <krasser@gmail.com>
>> wrote:
>> >>>>
>> >>>> Hey all,
>> >>>>
>> >>>> I am running into a problem where YARN kills containers for being
>> over
>> >>>> their memory allocation (which is about 8G for executors plus 6G
for
>> >>>> overhead), and I noticed that in those containers there are tons
of
>> >>>> pyspark.daemon processes hogging memory. Here's a snippet from a
>> container
>> >>>> with 97 pyspark.daemon processes. The total sum of RSS usage across
>> all of
>> >>>> these is 1,764,956 pages (i.e. 6.7GB on the system).
>> >>>>
>> >>>> Any ideas what's happening here and how I can get the number of
>> >>>> pyspark.daemon processes back to a more reasonable count?
>> >>>>
>> >>>> 2015-01-23 15:36:53,654 INFO  [Reporter] yarn.YarnAllocationHandler
>> >>>> (Logging.scala:logInfo(59)) - Container marked as failed:
>> >>>> container_1421692415636_0052_01_000030. Exit status: 143.
>> Diagnostics:
>> >>>> Container
>> [pid=35211,containerID=container_1421692415636_0052_01_000030] is
>> >>>> running beyond physical memory limits. Current usage: 14.9 GB of
>> 14.5 GB
>> >>>> physical memory used; 41.3 GB of 72.5 GB virtual memory used. Killing
>> >>>> container.
>> >>>> Dump of the process-tree for container_1421692415636_0052_01_000030
:
>> >>>>    |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
>> >>>> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES)
>> FULL_CMD_LINE
>> >>>>    |- 54101 36625 36625 35211 (python) 78 1 332730368 16834 python
-m
>> >>>> pyspark.daemon
>> >>>>    |- 52140 36625 36625 35211 (python) 58 1 332730368 16837 python
-m
>> >>>> pyspark.daemon
>> >>>>    |- 36625 35228 36625 35211 (python) 65 604 331685888 17694 python
>> -m
>> >>>> pyspark.daemon
>> >>>>
>> >>>>    [...]
>> >>>>
>> >>>>
>> >>>> Full output here:
>> https://gist.github.com/skrasser/e3e2ee8dede5ef6b082c
>> >>>>
>> >>>> Thank you!
>> >>>> -Sven
>> >>>>
>> >>>> --
>> >>>> krasser
>> >>>
>> >>>
>> >
>> >
>> >
>> > --
>> > http://sites.google.com/site/krasser/?utm_source=sig
>>
>
>
>
> --
> http://sites.google.com/site/krasser/?utm_source=sig
>



-- 
http://sites.google.com/site/krasser/?utm_source=sig

Mime
View raw message