hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wei-Chiu Chuang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10768) Optimize Hadoop RPC encryption performance
Date Wed, 23 May 2018 17:19:01 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16487689#comment-16487689
] 

Wei-Chiu Chuang commented on HADOOP-10768:
------------------------------------------

Sorry for the delay on my part -- uploaded v010 that added the thread local Mac instance,
without addressing the code review for rev009:

(here's the diff comparing v010 and v009)
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SaslCryptoCodec.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SaslCryptoCodec.java
index 12dd436..e49f9c5 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SaslCryptoCodec.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SaslCryptoCodec.java
@@ -48,6 +48,26 @@

   private final Integrity integrity;

+  /**
+   * The name of the hashing algorithm.
+   */
+  private static final String DEFAULT_HMAC_ALGORITHM = "HmacMD5";
+  /**
+   * A thread local store for the Macs.
+   */
+  private static final ThreadLocal<Mac> threadLocalMac =
+      new ThreadLocal<Mac>(){
+        @Override
+        protected Mac initialValue() {
+          try {
+            return Mac.getInstance(DEFAULT_HMAC_ALGORITHM);
+          } catch (NoSuchAlgorithmException nsa) {
+            throw new RuntimeException(
+                "Error creating instance of MD5 MAC algorithm", nsa);
+          }
+        }
+      };
+
   public SaslCryptoCodec(Configuration conf, CipherOption cipherOption,
                          boolean isServer) throws IOException {
     CryptoCodec codec = CryptoCodec.getInstance(conf,
@@ -174,9 +194,9 @@ void incPeerSeqNum() {
       System.arraycopy(seqNum, 0, seqAndMsg, 0, SEQ_NUM_LENGTH);
       System.arraycopy(msg, start, seqAndMsg, SEQ_NUM_LENGTH, len);

+      Mac m = threadLocalMac.get();
       try {
-        SecretKey keyKi = new SecretKeySpec(key, "HmacMD5");
-        Mac m = Mac.getInstance("HmacMD5");
+        SecretKey keyKi = new SecretKeySpec(key, DEFAULT_HMAC_ALGORITHM);
         m.init(keyKi);
         m.update(seqAndMsg);
         byte[] hMacMd5 = m.doFinal();
@@ -189,9 +209,6 @@ void incPeerSeqNum() {
       } catch (InvalidKeyException e) {
         throw new SaslException("Invalid bytes used for key of HMAC-MD5 hash.",
             e);
-      } catch (NoSuchAlgorithmException e) {
-        throw new SaslException("Error creating instance of MD5 MAC algorithm",
-            e);
       }
     }

diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPCCallBenchmark.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPCCallBenchmark.java
index 2393f49..93329fd 100644
--- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPCCallBenchmark.java
+++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPCCallBenchmark.java
@@ -35,7 +35,10 @@ public void testBenchmarkWithProto() throws Exception {
           "--time", "5",
           "--serverReaderThreads", "4",
           "--messageSize", "1024",
-          "--engine", "protobuf"});
+          "--engine", "protobuf",
+          "--sasl",
+          "--qop", "PRIVACY",
+          "--cipher", "AES/CTR/NoPadding"});
     assertEquals(0, rc);
   }
 }

> Optimize Hadoop RPC encryption performance
> ------------------------------------------
>
>                 Key: HADOOP-10768
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10768
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: performance, security
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Yi Liu
>            Assignee: Dapeng Sun
>            Priority: Major
>         Attachments: HADOOP-10768.001.patch, HADOOP-10768.002.patch, HADOOP-10768.003.patch,
HADOOP-10768.004.patch, HADOOP-10768.005.patch, HADOOP-10768.006.patch, HADOOP-10768.007.patch,
HADOOP-10768.008.patch, HADOOP-10768.009.patch, HADOOP-10768.010.patch, Optimize Hadoop RPC
encryption performance.pdf, cpu_profile_RPC_encryption_AES.png, cpu_profile_rpc_encryption_optimize_calculateHMAC.png
>
>
> Hadoop RPC encryption is enabled by setting {{hadoop.rpc.protection}} to "privacy". It
utilized SASL {{GSSAPI}} and {{DIGEST-MD5}} mechanisms for secure authentication and data
protection. Even {{GSSAPI}} supports using AES, but without AES-NI support by default, so
the encryption is slow and will become bottleneck.
> After discuss with [~atm], [~tucu00] and [~umamaheswararao], we can do the same optimization
as in HDFS-6606. Use AES-NI with more than *20x* speedup.
> On the other hand, RPC message is small, but RPC is frequent and there may be lots of
RPC calls in one connection, we needs to setup benchmark to see real improvement and then
make a trade-off. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message