spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Olson <aaron.ol...@shopify.com>
Subject Re: Pyspark Memory Woes
Date Tue, 11 Mar 2014 19:52:00 GMT
Hi Sandy,

We're configuring that with the JAVA_OPTS environment variable in
$SPARK_HOME/spark-worker-env.sh like this:

# JAVA OPTS
export SPARK_JAVA_OPTS="-Dspark.ui.port=0 -Dspark.default.parallelism=1024
-Dspark.cores.max=256 -Dspark.executor.memory=1500m
-Dspark.worker.timeout=500 -Dspark.akka.timeout=500 "

Does that value seem low to you?

-Aaron




On Tue, Mar 11, 2014 at 3:08 PM, Sandy Ryza <sandy.ryza@cloudera.com> wrote:

> Hi Aaron,
>
> When you say "Java heap space is 1.5G per worker, 24 or 32 cores across
> 46 nodes. It seems like we should have more than enough to do this
> comfortably.", how are you configuring this?
>
> -Sandy
>
>
> On Tue, Mar 11, 2014 at 10:11 AM, Aaron Olson <aaron.olson@shopify.com>wrote:
>
>> Dear Sparkians,
>>
>> We are working on a system to do relational modeling on top of Spark, all
>> done in pyspark. While we've been learning a lot about Spark internals so
>> far, we're currently running into memory issues and wondering how best to
>> profile to fix them. Here are our symptoms:
>>
>>    - We're operating on data sets up to 80G in size of uncompressed
>>    JSON, 66 million records in the largest one.
>>    - Sometimes we're joining those large data sets, but cardinality
>>    never exceeds 66 million (unless we've got a bug somewhere).
>>    - We're seeing various OOM problems: sometimes python takes all
>>    available mem, sometimes we OOM with no heap space left, and occasionally
>>    OOM with GC overhead limit exceeded.
>>    - Sometimes we also see what looks like a single huge message sent
>>    over the wire that exceeds the wire format limitations.
>>    - Java heap space is 1.5G per worker, 24 or 32 cores across 46 nodes.
>>    It seems like we should have more than enough to do this comfortably.
>>    - We're trying to isolate specific steps now, but every time it
>>    errors, we're partitioning (i.e. partitionByKey is involved somewhere).
>>
>> We've been instrumneting according to the monitoring and tuning docs, but
>> a bit at a loss for where we're going wrong. We suspect poor/wrong
>> partitioning on our part somehow. With that in mind, some questions:
>>
>>    - How exactly is partitioning information propagated? It looks like
>>    within a pipelined RDD the parent partitioning is preserved throughout
>>    unless we either specifically repartition or go through a reduce. We're
>>    splitting as much as we can on maps and letting reduces happen normally. Is
>>    that good practice?
>>    - When doing e.g. partitionByKey, does an entire partition get sent
>>    to one worker process?
>>    - When does Spark stream data? Are there easy ways to sabotage the
>>    streaming? Are there any knobs for us to twiddle here?
>>    - Is there any way to specify the number of shuffles for a given
>>    reduce step?
>>    - How can we get better insight into what our workers are doing,
>>    specifically around moving data in and out of python land?
>>
>> I realise it's hard to troubleshoot in the absence of code but any test
>> case we have would be contrived. We're collecting more metrics and trying
>> to reason about what might be happening, but any guidance at this point
>> would be most helpful.
>>
>> Thanks!
>>
>> --
>> Aaron Olson
>> Data Engineer, Shopify
>>
>
>


-- 
Aaron Olson
Data Engineer, Shopify

Mime
View raw message