spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andras Barjak <>
Subject Re: Spark runs applications in an inconsistent way
Date Wed, 23 Apr 2014 09:43:42 GMT
>    - Spark UI shows number of succeeded tasks is more than total number
>    of tasks, eg: 3500/3000. There are no failed tasks. At this stage the
>    computation keeps carrying on for a long time without returning an answer.
> No sign of resubmitted tasks in the command line logs either?
You might want to get more information on what is going on in the JVM?
I don't know what others use but jvmtop is easy to install on ec2 and you
can monitor some processes.

>    - The only way to get an answer from an application is to hopelessly
>    keep running that application multiple times, until by some luck it gets
>    converged.
> I was not able to regenerate this by a minimal code, as it seems some
> random factors affect this behavior. I have a suspicion, but I'm not sure,
> that use of one or more groupByKey() calls intensifies this problem.
Is this related to the amount of data you are processing? Is it more likely
to happen on large data?
My experience on ec2 is whenever the the memory/partitioning/timout
settings are reasonable
the output is quite consistent. Even if I stop and restart the cluster the
other day.

View raw message