spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcelo Vanzin <>
Subject Re: Rolling policy in Spark event logs for long living streaming applications
Date Fri, 01 Dec 2017 17:25:41 GMT
There's really no current solution to this. There's a brief discussion
about it on SPARK-12140.

Here we recommend people disable event logs for streaming apps, as
sub-optimal as that might be...

On Fri, Dec 1, 2017 at 3:27 AM, kankalapti omkar naidu
<> wrote:
> Dear spark developers,
> I did not find rolling policy for spark event logs,for long living streaming
> applications the event log is becoming huge with in few hours.
> The only way I found is fixing the existing event logging spark
> listener(EventLoggingListener),
> but I believe there should be already a solution tackling this problem
> because it is a frequently occurring problem when ever somebody runs
> streaming applications for long time.
> What would be the right way to fix this,whether any config settings are
> available for this or writing our own scripts to cleanup the log directory
> regularly based on time/size or we need to implement rolling policy feature?
> Regards
> Omkar


To unsubscribe e-mail:

View raw message