kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Calvin Chen <pingc...@hotmail.com>
Subject Kafka SSL
Date Fri, 30 Apr 2021 16:26:57 GMT
Hi all

I'm working on Kafka(kafka_2.13-2.7.0)cluster with SSL enabled, and I need help on Kafka broker
config(I got error of connection failed) and client SSL config(I got error of SSL handshake
failed).


I setup Kafka and client SSL config by taking reference of
Apache Kafka<https://kafka.apache.org/documentation/#security_ssl>
Apache Kafka TLS encryption & authentication - Azure HDInsight | Microsoft Docs<https://docs.microsoft.com/en-us/azure/hdinsight/kafka/apache-kafka-ssl-encryption-authentication>

And I can verify my Kafka cluster SSL with below command:

openssl s_client -debug -connect sc2-kafka-dev-001_node-1:9093 -tls1_2

some output is:

Server certificate
-----BEGIN CERTIFICATE-----
MIID1TCCAb0CFGy5db0MHYKTnZZAQpnHsR3ywrsqMA0GCSqGSIb3DQEBCwUAMBwx
GjAYBgNVBAMMEUthZmthLVNlY3VyaXR5LUNBMB4XDTIxMDQzMDE0NDEzMVoXDTIy
MDQzMDE0NDEzMVowMjEwMC4GA1UEAwwnc2MyLWthZmthLWRldi0wMDFfbm9kZS0x
LmVuZy52bXdhcmUuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA
wuL14qBmI++Ii/lxLU32TlGd0VlDX29JXjyqEUoaXDjYBroY5+FDhawladB3YU3/
IY2fQ9PHoPLVntBnMMf29m8buVFKXsRT0mOjkyVuUUZcp0L9mLMKnKE1Rn+EJM93
Ys0A8/YJgp3LYu0cbLbqw9TUdFkyesaV5zqAXse14npi0eqXk5pk5ss2ePfqa6bN
m2zM1eZrJjjp1vFx0oL8N6z2z6+AS67unyj9x2SjyXQgigbnz36VM99EUeMeQLuz
weuZN97sKKW4ub+ya0R6lbS5pum+iQ4ukA9TeiXllqwoFZTEZistsbec5OvgVgC0
41I6rtlGdqkAPEyU8xtfnwIDAQABMA0GCSqGSIb3DQEBCwUAA4ICAQAdTBndO51t
IK40oYHf2dWHE4WPvZfDoQpAVwhLptsbQD4RVdpPUxagbh4F4zAFwIZgCpwU0YBz
sq71p45x/3NjX40eIWsC0WgQoCQsCWimXQSMOltopNEhrSICd7mD1H/C1uftNXU1
uAGRGUC8wgX1ULdHLg0Szvz519ia+uZqOKyzsMBDZnmtesli3lTmXjjO5E5aPLaU
ztLeZrhHzR7ib9ZtIidl4hviPKbdLBPkeBqk7b821RbCK1Ny8eSOBYY3wePqTGU3
LbLEEeFgNBr9wEsmEcr237QW4UrYX5TjxeoykQj72u9tAb8mTrAY8QXUo9f826hQ
kTcSe504t6hMmX6oP9R3wUHqpIAZ3woqOV/I2KwCt2L3thUXyJK7F9XTSZQq89DT
E4SQlEthR+Mq/eIqyunq403MnQuxRGpfkiOLzBO1vUYDbnWjaC3oouTW9Y1rhF0L
t+DqaMXSTLyhcLZ8xUMcpgfROMArjufTsQ5KWqUYCTUffsrRVFzlyg02OjzgYJ5a
XR/lp64V3Ul1/8EM7QujDgdq9KTRu4FxuOk+8AFMOz4UJ1iqFONBKz6UTYmKjECw
aEp8k8WjuyHeuO5+d9qav+xYSQbHhZ5QSILKlyDSDkLWTjgNyvCMKzabtTW1HfQJ
p4DsCTjGse76yHJNAnH0jdGBVvi8ONdhuA==
-----END CERTIFICATE-----
subject=CN = sc2-kafka-dev-001_node-1.eng.vmware.com

issuer=CN = Kafka-Security-CA


So when I see above output, does it means my SSL setup for Kafka broker is ok?


However, I didn't get below keyword in server.log, as mentioned from Kafka webpage, I should
see below in server.log.


with addresses: PLAINTEXT -> EndPoint({{fqdn}},9092,PLAINTEXT),SSL -> EndPoint({{fqdn}},9093,SSL)

My two server.log output are:

[2021-04-30 09:05:08,954] INFO [KafkaServer id=1] started (kafka.server.KafkaServer)

While another one is:

[2021-04-30 09:05:30,183] WARN [Controller id=2, targetBrokerId=1] Connection to node 1 (sc2-kafka-dev-001_node-1.eng.vmware.com/10.185.50.10:9093)
could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2021-04-30 09:05:30,311] WARN [Controller id=2, targetBrokerId=3] Connection to node 3 (sc2-kafka-dev-001_node-3.eng.vmware.com/10.185.50.12:9093)
could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

It looks like the Kafka cluster with SSL enabled has some problem on setup connection across
brokers. BTW, I haven't apply for the DNS record for my brokers, I setup domain name in /etc/hosts,
and it shall be ok for the test?


Also, when I test Kafka command line with SSL config, I see auth error, but I didn't config
auth, I just config ssl encryption:

[worker@sc2-kafka-dev-001_node-1 client]$ /opt/kafka/kafka_2.13-2.7.0/bin/kafka-console-producer.sh
--broker-list sc2-kafka-dev-001_node-1:9093 --topic topic1 --producer.config ./client-ssl.properties
>[2021-04-30 09:11:19,574] ERROR [Producer clientId=console-producer] Connection to node
-1 (sc2-kafka-dev-001_node-1/10.185.50.10:9093) failed authentication due to: SSL handshake
failed (org.apache.kafka.clients.NetworkClient)
[2021-04-30 09:11:19,575] WARN [Producer clientId=console-producer] Bootstrap broker sc2-kafka-dev-001_node-1:9093
(id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)


Here is my part of Kafka broker config:

listeners=PLAINTEXT://sc2-kafka-dev-001_node-2.eng.vmware.com:9092, SSL://sc2-kafka-dev-001_node-2.eng.vmware.com:9093
advertised.listeners=PLAINTEXT://sc2-kafka-dev-001_node-2.eng.vmware.com:9092, SSL://sc2-kafka-dev-001_node-2.eng.vmware.com:9093

ssl.endpoint.identification.algorithm=
security.inter.broker.protocol=SSL

ssl.keystore.location=/data/ssl1/kafka.server.keystore.jks
ssl.keystore.password=MyServerPassword123
ssl.key.password=MyServerPassword123
ssl.truststore.location=/data/ssl1/kafka.server.truststore.jks
ssl.truststore.password=MyServerPassword123
ssl.enabled.protocols=TLSv1.2
ssl.truststore.type=JKS
ssl.keystore.type=JKS
ssl.secure.random.implementation=SHA1PRNG


Here is my client config:

security.protocol=SSL
ssl.truststore.location=/data/client/kafka.client.truststore.jks
ssl.truststore.password=MyClientPassword123
ssl.enabled.protocols=TLSv1.2



THANKS
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message