spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Waleed Fateem <waleed.fat...@gmail.com>
Subject Re: Spark stuck at removing broadcast variable
Date Sat, 18 Apr 2020 21:24:12 GMT
This might be obvious but just checking anyways, did you confirm whether or
not all of the messages have already been consumed by Spark? If that's the
case then I wouldn't expect much to happen unless new data comes into your
Kafka topic.

If you're a hundred percent sure that there's still plenty more data to be
consumed by Spark and that didn't happen, then I would suggest generating
Java thread dumps (use Java's jstack command) from your driver's process.

On Sat, Apr 18, 2020 at 2:43 PM Sean Owen <srowen@gmail.com> wrote:

> I don't think that means it's stuck on removing something; it was
> removed. Not sure what it is waiting on - more data perhaps?
>
> On Sat, Apr 18, 2020 at 2:22 PM Alchemist <alchemistsrivastava@gmail.com>
> wrote:
> >
> > I am running a simple Spark structured streaming application that is
> pulling data from a Kafka Topic. I have a Kafka Topic with nearly 1000
> partitions. I am running this app on 6 node EMR cluster with 4 cores and
> 16GB RAM. I observed that Spark is trying to pull data from all 1024 Kafka
> partition and after running successful for few iteration it is stuck with
> following exception:
> >
> > 20/04/18 00:51:41 INFO ContextCleaner: Cleaned accumulator 101
> > 20/04/18 00:51:41 INFO ContextCleaner: Cleaned accumulator 66
> > 20/04/18 00:51:41 INFO ContextCleaner: Cleaned accumulator 77
> > 20/04/18 00:51:41 INFO ContextCleaner: Cleaned accumulator 78
> >
> > 20/04/18 00:51:41 INFO BlockManagerInfo: Removed broadcast_2_piece0 on
> in memory (size: 4.5 KB, free: 2.7 GB)
> > 20/04/18 00:51:41 INFO BlockManagerInfo: Removed broadcast_2_piece0 on
> ip- in memory (size: 4.5 KB, free: 2.7 GB)
> > 20/04/18 00:51:41 INFO BlockManagerInfo: Removed broadcast_2_piece0 on
> ip- in memory (size: 4.5 KB, free: 2.7 GB)
> > Then Sparks show RUNNING but it is NOT Processing any data.
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>

Mime
View raw message