spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andras Barjak <andras.bar...@lynxanalytics.com>
Subject Re: Spark runs applications in an inconsistent way
Date Wed, 23 Apr 2014 09:43:42 GMT
>
>    - Spark UI shows number of succeeded tasks is more than total number
>    of tasks, eg: 3500/3000. There are no failed tasks. At this stage the
>    computation keeps carrying on for a long time without returning an answer.
>
> No sign of resubmitted tasks in the command line logs either?
You might want to get more information on what is going on in the JVM?
I don't know what others use but jvmtop is easy to install on ec2 and you
can monitor some processes.

>
>    - The only way to get an answer from an application is to hopelessly
>    keep running that application multiple times, until by some luck it gets
>    converged.
>
> I was not able to regenerate this by a minimal code, as it seems some
> random factors affect this behavior. I have a suspicion, but I'm not sure,
> that use of one or more groupByKey() calls intensifies this problem.
>
Is this related to the amount of data you are processing? Is it more likely
to happen on large data?
My experience on ec2 is whenever the the memory/partitioning/timout
settings are reasonable
the output is quite consistent. Even if I stop and restart the cluster the
other day.

Mime
View raw message