hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nishkam Ravi (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-2041) Hard to co-locate MR2 and Spark jobs on the same cluster in YARN
Date Sat, 10 May 2014 22:00:39 GMT
Nishkam Ravi created YARN-2041:
----------------------------------

             Summary: Hard to co-locate MR2 and Spark jobs on the same cluster in YARN
                 Key: YARN-2041
                 URL: https://issues.apache.org/jira/browse/YARN-2041
             Project: Hadoop YARN
          Issue Type: Improvement
          Components: nodemanager
    Affects Versions: 2.3.0
            Reporter: Nishkam Ravi
             Fix For: 2.4.0, 2.3.0


Performance of MR2 jobs falls drastically as YARN config parameter yarn.nodemanager.resource.memory-mb
 is increased beyond a certain value. 

Performance of Spark falls drastically as the value of yarn.nodemanager.resource.memory-mb
is decreased beyond a certain value for a large data set.

This makes it hard to co-locate MR2 and Spark jobs in YARN.

The experiments are being conducted on a 6-node cluster. The following workloads are being
run: TeraGen, TeraSort, TeraValidate, WordCount, ShuffleText and PageRank.

Will add more details to this JIRA over time.




--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message