spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun Mahadevan <ar...@apache.org>
Subject Re: Error - Dropping SparkListenerEvent because no remaining room in event queue
Date Thu, 25 Oct 2018 01:08:59 GMT
Maybe you have spark listeners that are not processing the events fast
enough?
Do you have spark event logging enabled?
You might have to profile the built in and your custom listeners to see
whats going on.

- Arun

On Wed, 24 Oct 2018 at 16:08, karan alang <karan.alang@gmail.com> wrote:

>
> Pls note - Spark version is 2.2.0
>
> On Wed, Oct 24, 2018 at 3:57 PM karan alang <karan.alang@gmail.com> wrote:
>
>> Hello -
>> we are running a Spark job, and getting the following error -
>>
>> "LiveListenerBus: Dropping SparkListenerEvent because no remaining room
>> in event queue"
>>
>> As per the recommendation in the Spark Docs -
>>
>> I've increased the value of property
>> spark.scheduler.listenerbus.eventqueue.capacity to 90000 (from the
>> default 10000)
>> and also increased the Diver memory
>>
>> That seems to have mitigated the issue.
>>
>> The question is - is there is any Code optimization (or any other) that
>> can be done to resolve this problem ?
>> Pls note - we are primarily using functions like - reduce(),
>> collectAsList() and persist() as part of the job.
>>
>

Mime
View raw message