spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 李明伟 <>
Subject Re:Re: Will the HiveContext cause memory leak ?
Date Wed, 11 May 2016 04:56:34 GMT
Hi  Ted

Spark version :  spark-1.6.0-bin-hadoop2.6
I tried increase the memory of executor. Still have the same problem.
I can use jmap to capture some thing. But the output is too difficult to understand. 

在 2016-05-11 11:50:14,"Ted Yu" <> 写道:

Which Spark release are you using ?

I assume executor crashed due to OOME.

Did you have a chance to capture jmap on the executor before it crashed ?

Have you tried giving more memory to the executor ?


On Tue, May 10, 2016 at 8:25 PM,<> wrote:
I submit my code to a spark stand alone cluster. Find the memory usage
executor process keeps growing. Which cause the program to crash.

I modified the code and submit several times. Find below 4 line may causing
the issue

    dataframe =
    windowSpec =
    rank = func.dense_rank().over(windowSpec)
    ret =['router'],dataframe['interface'],dataframe['bits'],

It looks a little complicated but it is just some Window function on
dataframe. I use the HiveContext because SQLContext do not support window
function yet. Without the 4 line, my code can run all night. Adding them
will cause the memory leak. Program will crash in a few hours.

I will provided the whole code (50 lines)here.
Please advice me if it is a bug..

Also here is the submit command

    nohup ./bin/spark-submit  \
    --master spark://ES01:7077 \
    --executor-memory 4G \
    --num-executors 1 \
    --total-executor-cores 1 \
    --conf ""  \
    ./ 1>a.log 2>b.log &

View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message