hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (YARN-6449) Enable YARN to accept jobs with < 1 core allocations
Date Fri, 07 Apr 2017 05:59:41 GMT

     [ https://issues.apache.org/jira/browse/YARN-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Allen Wittenauer resolved YARN-6449.
    Resolution: Duplicate

> Enable YARN to accept jobs with < 1 core allocations
> ----------------------------------------------------
>                 Key: YARN-6449
>                 URL: https://issues.apache.org/jira/browse/YARN-6449
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: yarn
>            Reporter: Daniel Tomes
>              Labels: features, performance
> Product Enhancement Request
> In Spark/HIVE/etc. I often need to complete work for which an entire core is overkill
such as managing a JDBC connection or doing a simple map/transform; however, when I do this
on large datasets, 1 core X 500 partitions/mappers winds up with quite the cluster level footprint
even though most of those processor cycles are idle.
> I propose that we enable YARN to allow a user to submit jobs that "allocate < 1 core".
Under the covers, the JVM will still receive one core but YARN/ZK could keep track of the
fractions of cores being used and allow other jobs to consume the same core twice provided
that both jobs were submitted with <= .5 cores. Now, YARN can more effectively utilize
multi-threading and decrease CPU idle for the power users.
> Obviously this can ultimately result in very bad outcomes, but if we also enable security
controls then customers can configure such that only admins/gates can submit with < 1 full
core and ultimately resulting in a cluster that can do more.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

View raw message