hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gera Shegalov (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-5672) Provide optional RollingFileAppender for container log4j (syslog)
Date Thu, 05 Dec 2013 23:16:35 GMT
Gera Shegalov created MAPREDUCE-5672:

             Summary: Provide optional RollingFileAppender for container log4j (syslog)
                 Key: MAPREDUCE-5672
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5672
             Project: Hadoop Map/Reduce
          Issue Type: New Feature
          Components: mr-am, mrv2
    Affects Versions: 2.2.0
            Reporter: Gera Shegalov
            Assignee: Gera Shegalov

This JIRA is an alternative take on YARN-1130

We propose providing an option of using a RollingFileAppender(RFA)-based implementation of
container log appender as means of log size control via mapreduce.task.userlog.limit.kb. 

The idea is to use mapreduce.task.userlog.limit.kb as maximumFileSize of RFA. In addition
yarn.app.mapreduce.container.log.backups (task attempt containers) and yarn.app.mapreduce.am.log.backups
(MR-AM) are passed as maxBackupIndex.

Both current ContainerLogAppender (CLA) and new ContainerRollingLogAppender (CRLA) co-exist.
CLA is the default. CRLA is chosen when  mapreduce.task.userlog.limit.kb > 0 &&
*.backups > 0.

1) CRLA output is visible in UI right away. CLA output with mapreduce.task.userlog.limit.kb
> 0 is not visible until the task attempt finishes that prevents timely diagnostics. 
2) Even with excessive logging and a large mapreduce.task.userlog.limit.kb, no space is taken
from the JVM heap.
3) No UI impact, since YARN is already designed to deal with any log name beyond stderr/out,
syslog, debug.out, profile.out

1) if the logging is excessive there will be more local filesystem metadata I/O due to roll.
That should be negligible in the grand scheme.

Furthermore, to improve log consistency and completeness in the case of JVM crashes and SIGTERMing
by NM, we propose to restore the MRv1 behavior of periodic log syncing (every 5s) and having
log sync as part of a shutdown hook.

This message was sent by Atlassian JIRA

View raw message