spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koert Kuipers <ko...@tresata.com>
Subject Re: RDD Partitions not distributed evenly to executors
Date Tue, 05 Apr 2016 02:57:01 GMT
can you try:
spark.shuffle.reduceLocality.enabled=false

On Mon, Apr 4, 2016 at 8:17 PM, Mike Hynes <91mbbh@gmail.com> wrote:

> Dear all,
>
> Thank you for your responses.
>
> Michael Slavitch:
> > Just to be sure:  Has spark-env.sh and spark-defaults.conf been
> correctly propagated to all nodes?  Are they identical?
> Yes; these files are stored on a shared memory directory accessible to
> all nodes.
>
> Koert Kuipers:
> > we ran into similar issues and it seems related to the new memory
> > management. can you try:
> > spark.memory.useLegacyMode = true
> I reran the exact same code with a restarted cluster using this
> modification, and did not observe any difference. The partitioning is
> still imbalanced.
>
> Ted Yu:
> > If the changes can be ported over to 1.6.1, do you mind reproducing the
> issue there ?
> Since the spark.memory.useLegacyMode setting did not impact my code
> execution, I will have to change the Spark dependency back to earlier
> versions to see if the issue persists and get back to you.
>
> Meanwhile, if anyone else has any other ideas or experience, please let me
> know.
>
> Mike
>
> On 4/4/16, Koert Kuipers <koert@tresata.com> wrote:
> > we ran into similar issues and it seems related to the new memory
> > management. can you try:
> > spark.memory.useLegacyMode = true
> >
> > On Mon, Apr 4, 2016 at 9:12 AM, Mike Hynes <91mbbh@gmail.com> wrote:
> >
> >> [ CC'ing dev list since nearly identical questions have occurred in
> >> user list recently w/o resolution;
> >> c.f.:
> >>
> >>
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-work-distribution-among-execs-tt26502.html
> >>
> >>
> http://apache-spark-user-list.1001560.n3.nabble.com/Partitions-are-get-placed-on-the-single-node-tt26597.html
> >> ]
> >>
> >> Hello,
> >>
> >> In short, I'm reporting a problem concerning load imbalance of RDD
> >> partitions across a standalone cluster. Though there are 16 cores
> >> available per node, certain nodes will have >16 partitions, and some
> >> will correspondingly have <16 (and even 0).
> >>
> >> In more detail: I am running some scalability/performance tests for
> >> vector-type operations. The RDDs I'm considering are simple block
> >> vectors of type RDD[(Int,Vector)] for a Breeze vector type. The RDDs
> >> are generated with a fixed number of elements given by some multiple
> >> of the available cores, and subsequently hash-partitioned by their
> >> integer block index.
> >>
> >> I have verified that the hash partitioning key distribution, as well
> >> as the keys themselves, are both correct; the problem is truly that
> >> the partitions are *not* evenly distributed across the nodes.
> >>
> >> For instance, here is a representative output for some stages and
> >> tasks in an iterative program. This is a very simple test with 2
> >> nodes, 64 partitions, 32 cores (16 per node), and 2 executors. Two
> >> examples stages from the stderr log are stages 7 and 9:
> >> 7,mapPartitions at DummyVector.scala:113,64,1459771364404,1459771365272
> >> 9,mapPartitions at DummyVector.scala:113,64,1459771364431,1459771365639
> >>
> >> When counting the location of the partitions on the compute nodes from
> >> the stderr logs, however, you can clearly see the imbalance. Examples
> >> lines are:
> >> 13627&INFO&TaskSetManager&Starting task 0.0 in stage 7.0 (TID 196,
> >> himrod-2, partition 0,PROCESS_LOCAL, 3987 bytes)&
> >> 13628&INFO&TaskSetManager&Starting task 1.0 in stage 7.0 (TID 197,
> >> himrod-2, partition 1,PROCESS_LOCAL, 3987 bytes)&
> >> 13629&INFO&TaskSetManager&Starting task 2.0 in stage 7.0 (TID 198,
> >> himrod-2, partition 2,PROCESS_LOCAL, 3987 bytes)&
> >>
> >> Grep'ing the full set of above lines for each hostname, himrod-?,
> >> shows the problem occurs in each stage. Below is the output, where the
> >> number of partitions stored on each node is given alongside its
> >> hostname as in (himrod-?,num_partitions):
> >> Stage 7: (himrod-1,0) (himrod-2,64)
> >> Stage 9: (himrod-1,16) (himrod-2,48)
> >> Stage 12: (himrod-1,0) (himrod-2,64)
> >> Stage 14: (himrod-1,16) (himrod-2,48)
> >> The imbalance is also visible when the executor ID is used to count
> >> the partitions operated on by executors.
> >>
> >> I am working off a fairly recent modification of 2.0.0-SNAPSHOT branch
> >> (but the modifications do not touch the scheduler, and are irrelevant
> >> for these particular tests). Has something changed radically in 1.6+
> >> that would make a previously (<=1.5) correct configuration go haywire?
> >> Have new configuration settings been added of which I'm unaware that
> >> could lead to this problem?
> >>
> >> Please let me know if others in the community have observed this, and
> >> thank you for your time,
> >> Mike
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> >> For additional commands, e-mail: user-help@spark.apache.org
> >>
> >>
> >
>
>
> --
> Thanks,
> Mike
>

Mime
View raw message