spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gerard Maas <gerard.m...@gmail.com>
Subject Re: [Spark Streaming] One Batch hangs always and others process correctly
Date Mon, 28 Aug 2017 08:19:26 GMT
What is your streaming job doing? There's an important difference in #of
tasks (partitions) between the 'huge' task and the others. Looks like the
work to be done on that one is rather heavy.

-kr, Gerard

On Mon, Aug 28, 2017 at 9:27 AM, 李书明 <alemmontree@126.com> wrote:

>
>     Hi, everybody. I met a problem:  one batch in spark streaming hangs
> allover, and other batchs run normally.
>
>     *Environment:*
>
>     spark-version: 2.0.0
>     mode: yarn-cluster
>     spark.executor.cores: 8
>     spark.executor.memory: 12g
>     spark.driver.memory: 3g
>
>     I also check the driver/executors logs, don't find any usefull
> exceptions, and also do not find any useful solutions in website, can
> somebody expain to me why this batch hangs , or why not fails?
>
>
>
>
>
>
>
>

Mime
View raw message