kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Wilkinson <aa...@modopayments.com>
Subject Re: Massive SSL performance degredation
Date Sat, 19 Nov 2016 01:06:56 GMT
So the kafka performance tools seem to indicate that the problem is not in
the broker, but rather somewhere in librdkafka/OpenSSL.  I'm not completely
sure I got the configs right to try and eliminate any batching
considerations in the latency calculation (it seems like encrypting /
decrypting a batch of 1000 messages 1 time would be more efficient than
encrypting / decrypting 1 message 1000 times and I am interested in the
latter).  BUT doing a relative test of plaintext vs ssl seemed to show the
promised 20% - 50% overhead rather than the thousands of percent I am
seeing with librdkafka + OpenSSL.

Plaintext connection
/usr/local/kafka/kafka_2.11-0.10.1.0$ bin/kafka-run-class.sh
kafka.tools.EndToEndLatency <broker hostname>:9092 latency.test 10000 1 128
/usr/local/kafka/kafka_2.11-0.10.1.0/config/client.properties
...
Avg latency: 1.8739 ms

SSL Connection
bin/kafka-run-class.sh kafka.tools.EndToEndLatency <broker hostname>:9093
latency.test 10000 1 128
/usr/local/kafka/kafka_2.11-0.10.1.0/config/client-ssl.properties
...
Avg latency: 2.4234 ms

Also bin/kafka-producer-perf-test.sh --topic producer.latency.test
--num-records 20 --record-size 128 --throughput 1 --producer.config
/usr/local/kafka/kafka_2.11-0.10.1.0/config/client-ssl.properties
7 records sent, 1.3 records/sec (0.00 MB/sec), 94.7 ms avg latency, 591.0
max latency.
6 records sent, 1.0 records/sec (0.00 MB/sec), 1.8 ms avg latency, 2.0 max
latency.
5 records sent, 1.0 records/sec (0.00 MB/sec), 2.8 ms avg latency, 3.0 max
latency.
20 records sent, 1.022809 records/sec (0.00 MB/sec), 34.75 ms avg latency,
591.00 ms max latency, 3 ms 50th, 591 ms 95th, 591 ms 99th, 591 ms 99.9th.

Seems to show decent latency after the initial SSL handshake as well.

So I will try to look harder at how librdkafka + OpenSSL are doing SSL.  If
I figure anything out, I'll do one last follow up email to save someone
else with this stack a similar headache.

Thanks for teaching me about the command line tools, guys!
- Aaron



On Fri, Nov 18, 2016 at 2:59 PM, Aaron Wilkinson <aaron@modopayments.com>
wrote:

> Thank you both, Hans and Rajini.
>
> I will try out all the methods you suggested and report back.
>
> As an aside my investigation into the known, slow software implementation
> of the GCM class of cipher algorithms in java 8 was a bust.  I tried all of
> the default cipher suites common to OpenSSL (on the client) and java (on
> the broker) and they all gave consistent (slow) results of about 40 ms per
> hop.
>
> For posterity at the time of this writing those were (OpenSSL format):
> DHE-DSS-AES256-GCM-SHA384
> DHE-DSS-AES256-SHA256
> DHE-DSS-AES256-SHA
> DHE-DSS-AES128-GCM-SHA256
> DHE-DSS-AES128-SHA256
> DHE-DSS-AES128-SHA
> EDH-DSS-DES-CBC3-SHA
>
> I can't guarantee that I'm not looking at a problem where the java crypto
> module is not using hardware acceleration.  (I've verified that OpenSSL has
> access to the aesni hardware instructions, but I have no idea how to tell
> if the java crypto module is making use of them.)  However, it would appear
> that it is at least not a problem specific to the GCM algorithm.
>
> - Aaron
>
>
> On Fri, Nov 18, 2016 at 2:37 AM, Rajini Sivaram <
> rajinisivaram@googlemail.com> wrote:
>
>> You can use the tools shipped with Kafka to measure latency.
>>
>> For latency at low load, run:
>>
>>
>>    - bin/kafka-run-class.sh kafka.tools.EndToEndLatency
>>
>>
>> You may also find it useful to run producer performance test at different
>> throughputs. The tool prints out latency as well:
>>
>>
>>    - bin/kafka-producer-perf-test.sh
>>
>>
>> On Fri, Nov 18, 2016 at 1:25 AM, Hans Jespersen <hans@confluent.io>
>> wrote:
>>
>> > Publish lots of messages and measure in seconds or minutes. Otherwise
>> you
>> > are just benchmarking the initial SSL handshake setup time which should
>> > normally be a one time overhead, not a per message overhead. If you just
>> > send one message then of course SSL is much slower.
>> >
>> > -hans
>> >
>> > > On Nov 18, 2016, at 1:07 AM, Aaron Wilkinson <aaron@modopayments.com>
>> > wrote:
>> > >
>> > > Hi, Hans.  I was able to get the command line producer / consumer
>> working
>> > > with SSL but I'm not sure how to measure millisecond resolution
>> latency
>> > > with them.  I thought maybe the '--property print.timestamp=true'
>> > argument
>> > > would help, but only has second resolution.  Do you know of any way to
>> > get
>> > > the consumer to print out a receipt time-stamp with millisecond
>> > > resolution?  Or of any extended documentation on the command line
>> tools
>> > in
>> > > general?
>> > >
>> > > Oh also, a couple other tidbits that may help:
>> > > Ubuntu 16.04
>> > > Kafka 10.1.0
>> > > openjdk version "1.8.0_111"
>> > > TLS 1.2
>> > >
>> > > I was wondering if maybe this could be my problem:
>> > > http://stackoverflow.com/questions/25992131/slow-aes-
>> > gcm-encryption-and-decryption-with-java-8u20
>> > >
>> > > I didn't specify any cipher suites in either the broker or the client
>> > > config which I gather leaves it up to the broker/client to decide
>> during
>> > > TLS handshaking.  I'm not sure if there is an easy way to figure out
>> > which
>> > > one they ended up with...  I'll work on specifying which cipher suite
>> I
>> > > want and try to pick something with which java is simpatico.
>> > >
>> > >
>> > >> On Thu, Nov 17, 2016 at 4:04 PM, Hans Jespersen <hans@confluent.io>
>> > wrote:
>> > >>
>> > >> What is the difference using the bin/kafka-console-producer and
>> > >> kafka-console-consumer as pub/sub clients?
>> > >>
>> > >> see http://docs.confluent.io/3.1.0/kafka/ssl.html
>> > >>
>> > >> -hans
>> > >>
>> > >> /**
>> > >> * Hans Jespersen, Principal Systems Engineer, Confluent Inc.
>> > >> * hans@confluent.io (650)924-2670
>> > >> */
>> > >>
>> > >> On Thu, Nov 17, 2016 at 11:56 PM, Aaron Wilkinson <
>> > aaron@modopayments.com>
>> > >> wrote:
>> > >>
>> > >>> Pardon if this is a oft repeated issue, but all the information
I
>> could
>> > >>> find said I should expect a 20-50% performance hit when using SSL
>> with
>> > >>> kafka, and I am seeing closer to 2000-3000%
>> > >>>
>> > >>> I'm trying to get kafka to behave like a fast, secured message
bus.
>> > So I
>> > >>> am sending small messages, one at a time.  I have set up a simple,
2
>> > >>> machine experiment in AWS with 1 client machine and 1
>> zookeeper/broker
>> > >>> machine and I'm an running a very linear test.
>> > >>>
>> > >>> There are 2 topics: "request" and "response" and 2 threads on the
>> > client
>> > >>> machine each of which connects to those 2 topics.  Thread 1
>> produces a
>> > >>> "request", thread 2 consumes it and then produces a "response"
which
>> > >> thread
>> > >>> 1 then consumes.  At that point thread 1 proceeds to send the next
>> > >>> "request" and the process repeats.
>> > >>>
>> > >>> So there are a total of 4 connections to the broker.
>> > >>>
>> > >>> I can run a sustained test without SSL and see 1 to 1.5 ms per
>> message
>> > >> hop
>> > >>> (where a "hop" means the message has traveled across 1 of the 4
>> > >>> connections- either a production or a consumption of either the
>> request
>> > >> or
>> > >>> the response).
>> > >>>
>> > >>> Each connection for which I turn on SSL increases the hop time
35
>> to 45
>> > >> ms.
>> > >>>
>> > >>> Now, the problem could be with the stack I'm using (PHP 7 talking
to
>> > the
>> > >>> broker via the librdkafka C library).  But before I go about trying
>> to
>> > >>> reproduce this with a java client (which is not my forte) I was
>> > wondering
>> > >>> if anyone else has run into a similar issue either with PHP or
any
>> > other
>> > >>> language / library.  Or does anyone know a direct way to figure
out
>> > >> whether
>> > >>> this slow down is at the broker or at the client?
>> > >>>
>> > >>> Thanks in advance for your help!
>> > >>> Aaron
>> > >>>
>> > >>
>> >
>>
>>
>>
>> --
>> Regards,
>>
>> Rajini
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message