spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen Song <chen.song...@gmail.com>
Subject Re: application logs for long lived job on YARN
Date Thu, 27 Aug 2015 18:32:33 GMT
Anyone has similar problem or thoughts on this?

On Wed, Aug 26, 2015 at 10:37 AM, Chen Song <chen.song.82@gmail.com> wrote:

> When running long-lived job on YARN like Spark Streaming, I found that
> container logs gone after days on executor nodes, although the job itself
> is still running.
>
>
> I am using cdh5.4.0 and have aggregated logs enabled. Because the local
> logs are gone on executor nodes, I don't see any aggregated logs on hdfs
> after the job is killed or failed.
>
> Is there a YARN config to keep the logs from being deleted for long-lived
> streaming job?
>
> --
> Chen Song
>
>


-- 
Chen Song

Mime
View raw message