hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anupam Seth (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7993) Hadoop ignores old-style config options for enabling compressed output
Date Thu, 26 Jan 2012 17:58:38 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194026#comment-13194026

Anupam Seth commented on HADOOP-7993:

Just to clarify the reason for proposing the code change the way it is in the patch is that
upon investigation, our understanding was that the old-style config was being written to file
by the JobSubmitter before the replacement with the keys listed in MR deprecated keys are
invoked, e.g. by a get. Thus, when the AppMaster tried to read in the config coupled together
with mapred-default which currently has this set to false, the old-style config was being
overriden to be whatever was specified in the default file. As a solution, therefore, we force
handling deprecation on all keys before the job.xml gets written out.
> Hadoop ignores old-style config options for enabling compressed output
> ----------------------------------------------------------------------
>                 Key: HADOOP-7993
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7993
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.23.1
>            Reporter: Anupam Seth
>            Assignee: Anupam Seth
>         Attachments: HADOOP-7993_branch_0_23.patch, HADOOP-7993_branch_0_23.patch
> Hadoop seems to ignore the config options even though they are printed as deprecation
warnings in the log: mapred.output.compress and
> mapred.output.compression.codec
> - settings that work on 0.20 but not on 0.23
> mapred.output.compress=true
> mapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
> - settings that work on 0.23
> mapreduce.output.fileoutputformat.compress=true
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
> This breaks backwards compatibility and causes existing jobs to fail.
> This was found to happen due to the JobSubmitter writing out the job.xml file with the
old-style configs and can be fixed by handdling deprecation before the file is written out.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message