From mapreduce-dev-return-10721-apmail-hadoop-mapreduce-dev-archive=hadoop.apache.org@hadoop.apache.org Mon Dec 2 14:32:33 2013 Return-Path: X-Original-To: apmail-hadoop-mapreduce-dev-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0C36110432 for ; Mon, 2 Dec 2013 14:32:33 +0000 (UTC) Received: (qmail 86910 invoked by uid 500); 2 Dec 2013 14:31:52 -0000 Delivered-To: apmail-hadoop-mapreduce-dev-archive@hadoop.apache.org Received: (qmail 86797 invoked by uid 500); 2 Dec 2013 14:31:42 -0000 Mailing-List: contact mapreduce-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-dev@hadoop.apache.org Delivered-To: mailing list mapreduce-dev@hadoop.apache.org Received: (qmail 86740 invoked by uid 99); 2 Dec 2013 14:31:38 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Dec 2013 14:31:38 +0000 Date: Mon, 2 Dec 2013 14:31:38 +0000 (UTC) From: "Eric Sirianni (JIRA)" To: mapreduce-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (MAPREDUCE-5661) ShuffleHandler using yarn.nodemanager.local-dirs instead of mapreduce.cluster.local.dir MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 Eric Sirianni created MAPREDUCE-5661: ---------------------------------------- Summary: ShuffleHandler using yarn.nodemanager.local-dirs instead of mapreduce.cluster.local.dir Key: MAPREDUCE-5661 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5661 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.2.0 Reporter: Eric Sirianni Priority: Trivial While debugging an issue where a MapReduce job is failing due to running out of disk space, I noticed that the {{ShuffleHandler}} uses {{yarn.nodemanager.local-dirs}} for its {{LocalDirAllocator}} whereas all of the other MapReduce classes use {{mapreduce.cluster.local.dir}}: {noformat} $ find hadoop-mapreduce-project/hadoop-mapreduce-client/*/src/main/java/ -name "*.java" | xargs grep "new LocalDirAllocator(" hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java: LocalDirAllocator lDirAlloc = new LocalDirAllocator(MRConfig.LOCAL_DIR); hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnOutputFiles.java: new LocalDirAllocator(MRConfig.LOCAL_DIR); hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java: new LocalDirAllocator(MRConfig.LOCAL_DIR); hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/BackupStore.java: this.lDirAlloc = new LocalDirAllocator(MRConfig.LOCAL_DIR); hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MROutputFiles.java: new LocalDirAllocator(MRConfig.LOCAL_DIR); hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java: new LocalDirAllocator(MRConfig.LOCAL_DIR); hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java: this.lDirAlloc = new LocalDirAllocator(MRConfig.LOCAL_DIR); *****hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java: new LocalDirAllocator(YarnConfiguration.NM_LOCAL_DIRS); {noformat} This inconsistency feels like something that is likely to confuse admins. -- This message was sent by Atlassian JIRA (v6.1#6144)