hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Eagles (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-16119) KMS on Hadoop RPC Engine
Date Mon, 18 Feb 2019 21:31:00 GMT
Jonathan Eagles created HADOOP-16119:

             Summary: KMS on Hadoop RPC Engine
                 Key: HADOOP-16119
                 URL: https://issues.apache.org/jira/browse/HADOOP-16119
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Jonathan Eagles

Per discussion on common-dev and text copied here for ease of reference.


Thanks all for the inputs,

To offer additional information (while Daryn is working on his stuff),
optimizing RPC encryption opens up another possibility: migrating KMS
service to use Hadoop RPC.

Today's KMS uses HTTPS + REST API, much like webhdfs. It has very
undesirable performance (a few thousand ops per second) compared to
NameNode. Unfortunately for each NameNode namespace operation you also need
to access KMS too.

Migrating KMS to Hadoop RPC greatly improves its performance (if
implemented correctly), and RPC encryption would be a prerequisite. So
please keep that in mind when discussing the Hadoop RPC encryption
improvements. Cloudera is very interested to help with the Hadoop RPC
encryption project because a lot of our customers are using at-rest
encryption, and some of them are starting to hit KMS performance limit.

This whole "migrating KMS to Hadoop RPC" was Daryn's idea. I heard this
idea in the meetup and I am very thrilled to see this happening because it
is a real issue bothering some of our customers, and I suspect it is the
right solution to address this tech debt.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message