hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siyao Meng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-16011) OsSecureRandom very slow compared to other SecureRandom implementations
Date Thu, 28 Mar 2019 22:17:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-16011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804354#comment-16804354

Siyao Meng commented on HADOOP-16011:

[~jojochuang] I see no reference of OsSecureRandom in core-default.xml, only OpensslAesCtrCryptoCodec
  <value>org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec, org.apache.hadoop.crypto.JceAesCtrCryptoCodec</value>
    Comma-separated list of crypto codec implementations for AES/CTR/NoPadding.
    The first implementation will be used if available, others are fallbacks.
What change do we need here?

> OsSecureRandom very slow compared to other SecureRandom implementations
> -----------------------------------------------------------------------
>                 Key: HADOOP-16011
>                 URL: https://issues.apache.org/jira/browse/HADOOP-16011
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Todd Lipcon
>            Assignee: Siyao Meng
>            Priority: Major
>         Attachments: HADOOP-16011.001.patch, MyBenchmark.java
> In looking at performance of a workload which creates a lot of short-lived remote connections
to a secured DN, [~philip] and I found very high system CPU usage. We tracked it down to reads
from /dev/random, which are incurred by the DN using CryptoCodec.generateSecureRandom to generate
a transient session key and IV for AES encryption.
> In the case that the OpenSSL codec is not enabled, the above code falls through to the
JDK SecureRandom implementation, which performs reasonably. However, OpenSSLCodec defaults
to using OsSecureRandom, which reads all random data from /dev/random rather than doing something
more efficient like initializing a CSPRNG from a small seed.
> I wrote a simple JMH benchmark to compare various approaches when running with concurrency
>  testHadoop - using CryptoCodec
>  testNewSecureRandom - using 'new SecureRandom()' each iteration
>  testSha1PrngNew - using the SHA1PRNG explicitly, new instance each iteration
>  testSha1PrngShared - using a single shared instance of SHA1PRNG
>  testSha1PrngThread - using a thread-specific instance of SHA1PRNG
> {code:java}
> Benchmark                         Mode  Cnt        Score   Error  Units
> MyBenchmark.testHadoop           thrpt          1293.000          ops/s  [with libhadoop.so]
> MyBenchmark.testHadoop           thrpt        461515.697          ops/s [without libhadoop.so]
> MyBenchmark.testNewSecureRandom  thrpt         43413.640          ops/s
> MyBenchmark.testSha1PrngNew      thrpt        395515.000          ops/s
> MyBenchmark.testSha1PrngShared   thrpt        164488.713          ops/s
> MyBenchmark.testSha1PrngThread   thrpt       4295123.210          ops/s
> {code}
> In other words, the presence of the OpenSSL acceleration slows down this code path by
356x. And, compared to the optimal (thread-local Sha1Prng) it's 3321x slower.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message