hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Roni Burd (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-6426) Compress ZK YARN keys to scale up (especially AppStateData
Date Sat, 01 Apr 2017 00:11:41 GMT
Roni Burd created YARN-6426:
-------------------------------

             Summary: Compress ZK YARN keys to scale up (especially AppStateData
                 Key: YARN-6426
                 URL: https://issues.apache.org/jira/browse/YARN-6426
             Project: Hadoop YARN
          Issue Type: Improvement
          Components: yarn
            Reporter: Roni Burd
            Assignee: Roni Burd
             Fix For: 2.8.0, 3.0.0-alpha2


ZK today stores the protobuf files uncompressed. This is not an issue except that if a customer
job has thousands of files, AppStateData will store the user context as a string with multiple
URLs and it is easy to get to 1MB or more. 

This can put unnecessary strain on ZK and make the process slow. 

The proposal is to simply compress protobufs before sending them to ZK



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org


Mime
View raw message