spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shahid K. I." <shahidk...@gmail.com>
Subject Re: event log directory(spark-history) filled by large .inprogress files for spark streaming applications
Date Wed, 17 Jul 2019 16:28:08 GMT
Hi,
With the current design, eventlogs are not ideal for long running streaming
applications. So, it is better then to disable the eventlogs. There was  a
proposal for splitting the eventlogs based on size/Job/query for long
running applications,  not sure about the followup for that.

Regards,
Shahid

On Tue, 16 Jul 2019, 3:38 pm raman gugnani, <ramangugnani.007@gmail.com>
wrote:

> HI ,
>
> I have long running spark streaming jobs.
> Event log directories are getting filled with .inprogress files.
> Is there fix or work around for spark streaming.
>
> There is also one jira raised for the same by one reporter.
>
> https://issues.apache.org/jira/browse/SPARK-22783
>
> --
> Raman Gugnani
>
> 8588892293
> Principal Engineer
> *ixigo.com <http://ixigo.com>*
>

Mime
View raw message