hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Travis Crawford (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-6746) hadoop-daemon.sh does not append to log files
Date Tue, 15 Jun 2010 03:05:24 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-6746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Travis Crawford updated HADOOP-6746:

          Status: Patch Available  (was: Open)
    Release Note: Open log files with append (rather than creating new files) so file truncation
works. This allows log rotation tools to operate correctly without restarting Hadoop processes.

This patch is useful from an operations perspective because Hadoop processes are typically
long-lived and generate large log files, risking full disks which cause node failure or pages
to operations staff to resolve the issue. By opening files in append mode they can be truncated,
thus freeing up disk space consumed by logs. Using a log rotation tool that (a) copies a log
file aside, (b) truncates "live" file, and (c) compresses rotates file for later analysis
if needed, is a very common task.

Currently log files are not opened in append mode, so truncation simply fills a file with
nothing, and new bytes are written at the existing offset; disk space is not reclaimed until
restarting the process.

> hadoop-daemon.sh does not append to log files
> ---------------------------------------------
>                 Key: HADOOP-6746
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6746
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Travis Crawford
> Looking in hadoop-daemon.sh we see the start command redirects logs to the log file by
creating a new file. It does not append to the log.
> nohup nice -n $HADOOP_NICENESS "$HADOOP_HOME"/bin/hadoop --config $HADOOP_CONF_DIR $command
"$@" > "$log" 2>&1 < /dev/null &
> I believe output should be appended to the log file so logrotate copytruncate works.
Currently a large log file cannot be truncated without restarting the running service, which
causes issues in production environments.
> The improved line would be exactly the same, except use >> to open the log file
in append mode.
> nohup nice -n $HADOOP_NICENESS "$HADOOP_HOME"/bin/hadoop --config $HADOOP_CONF_DIR $command
"$@" >> "$log" 2>&1 < /dev/null &

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message