spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kankalapti omkar naidu <omkarnaid...@gmail.com>
Subject Re: Rolling policy in Spark event logs for long living streaming applications
Date Wed, 20 Dec 2017 09:45:07 GMT
I reported the issue as  SPARK-22783.

Hope this will be resolved soon.

Regards
Omkar

On Fri, Dec 1, 2017 at 10:55 PM, Marcelo Vanzin <vanzin@cloudera.com> wrote:

> There's really no current solution to this. There's a brief discussion
> about it on SPARK-12140.
>
> Here we recommend people disable event logs for streaming apps, as
> sub-optimal as that might be...
>
> On Fri, Dec 1, 2017 at 3:27 AM, kankalapti omkar naidu
> <omkarnaidu.k@gmail.com> wrote:
> > Dear spark developers,
> >
> > I did not find rolling policy for spark event logs,for long living
> streaming
> > applications the event log is becoming huge with in few hours.
> > The only way I found is fixing the existing event logging spark
> > listener(EventLoggingListener),
> > but I believe there should be already a solution tackling this problem
> > because it is a frequently occurring problem when ever somebody runs
> > streaming applications for long time.
> >
> > What would be the right way to fix this,whether any config settings are
> > available for this or writing our own scripts to cleanup the log
> directory
> > regularly based on time/size or we need to implement rolling policy
> feature?
> >
> > Regards
> > Omkar
>
>
>
> --
> Marcelo
>

Mime
View raw message