kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ivan Janes <ivan.ja...@oryxgaming.com>
Subject Re: SASL + SSL : authentication error in broker-to-broker communication
Date Wed, 05 Jun 2019 16:10:45 GMT
Hi,

maybe the problem is zookeeper.connect setting:

zookeeper.connect=zoo:2181/kafka

Which means that you have to use /kafka path with kafka-configs.sh:

bin/kafka-configs.sh --zookeeper localhost:2181/kafka --alter --add-config 'SCRAM-SHA-256=[iterations=4096,password=alice-secret],SCRAM-SHA-512=[password=alice-secret]'
--entity-type users --entity-name alice> 

Change  --zookeeper localhost:2181 to  --zookeeper localhost:2181/kafka

Without /kafka prefix users are created in wrong path and kafka broker does not find users
...

I have needed about 2 weeks to figure it out :)

Regards,
Ivan


On 2019/05/15 12:28:03, Martin Gainty <m...@hotmail.com> wrote: 
> assuming ScramSaslProvider/ScramSaslServer your credentials are stored in ZK /config/users/<encoded-user>>

> but you cannot see plain-text attributes in ZK so use kafka tool to view> 
> kafka-configs.sh -describe /config/users/<encoded-user>> 
> 
> /*2019 update for kafka-configs.sh */> 
> 
> For ease of use, kafka-configs.sh will take a password and an optional iteration count
and generate a random salt, ServerKey and StoredKey as specified in in RFC 5802<https://tools.ietf.org/html/rfc5802>.
For example:> 
> 
> bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config 'SCRAM-SHA-256=[iterations=4096,password=alice-secret],SCRAM-SHA-512=[password=alice-secret]'
--entity-type users --entity-name alice> 
> 
> /*once you have verified username,password from  ZK credentials */> 
> you can now export your cert from /opt/kafka/ssl/kafka.server.keystore.jks> 
> keytool -exportcert -alias admin -keystore /opt/kafka/ssl/kafka.server.keystore.jks -keypass
xxxx -storepass xxxx -file admin.cert> 
> 
> (note the storepass is for truststore!)> 
> 
> if you can view the admin.cert with cert-viewer and validate username(subject) are consistent
with ZK creds> 
> if you dont have cert-viewer you can convert to pem> 
>  openssl pkcs12 -export -in "admin.p12" -out "admin.pem"> 
> check UID in either cert or pem is consistent with ZK> 
> 
> finally check zk credentials are propagated to jaas.conf> 
> 
> #used by interbroker connections> 
> KafkaServer {> 
>     org.apache.kafka.common.security.scram.ScramLoginModule required> 
>     username="admin"> 
>     password="xxxx";> 
> }> 
> 
> if there is consistency for all entities in> 
> username> 
> password> 
> then your kafka-broker(s) *should* authenticate (assuming they all reference the same
ZK server!)> 
> 
> bon chance> 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-84%3A+Support+SASL+SCRAM+mechanisms#KIP-84:SupportSASLSCRAMmechanisms-JAASconfiguration>

> KIP-84: Support SASL SCRAM mechanisms - Apache Kafka - Apache Software Foundation - Dashboard<https://cwiki.apache.org/confluence/display/KAFKA/KIP-84%3A+Support+SASL+SCRAM+mechanisms#KIP-84:SupportSASLSCRAMmechanisms-JAASconfiguration>>

> This Confluence has been LDAP enabled, if you are an ASF Committer, please use your LDAP
Credentials to login. Any problems file an INFRA jira ticket please.> 
> cwiki.apache.org> 
> 
> 
> 
> 
> ________________________________> 
> From: Kieran JOYEUX <kj...@splio.com>> 
> Sent: Wednesday, May 15, 2019 4:42 AM> 
> To: users@kafka.apache.org> 
> Subject: SASL + SSL : authentication error in broker-to-broker communication> 
> 
> Hello,> 
> 
> I'm facing trouble activating SASL on my currrent  working SSL only cluster. I have read
the doc many times and my configuration seems to be good. However, It's like Kafka cannot
authenticate and broker to broker communication is not working at all.> 
> 
> Do you have any ideas ? (Descriptions below)> 
> 
> Thanks a lot.> 
> 
> Kieran> 
> 
> --------------------------------------------> 
> 
> # Versions> 
> Kafka: 2.2.0> 
> Zookeeper: 3.4.9-3+deb9u1> 
> 
> # Error message> 
> [2019-05-15 10:14:00,811] DEBUG Set SASL server state to HANDSHAKE_OR_VERSIONS_REQUEST
during authentication (org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)>

> [2019-05-15 10:14:00,811] DEBUG Handling Kafka request API_VERSIONS during authentication
(org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)> 
> [2019-05-15 10:14:00,811] DEBUG Set SASL server state to HANDSHAKE_REQUEST during authentication
(org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)> 
> [2019-05-15 10:14:00,812] DEBUG Handling Kafka request SASL_HANDSHAKE during authentication
(org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)> 
> [2019-05-15 10:14:00,812] DEBUG Using SASL mechanism 'SCRAM-SHA-512' provided by client
(org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)> 
> [2019-05-15 10:14:00,813] DEBUG Setting SASL/SCRAM_SHA_512 server state to RECEIVE_CLIENT_FIRST_MESSAGE
(org.apache.kafka.common.security.scram.internals.ScramSaslServer)> 
> [2019-05-15 10:14:00,813] DEBUG Set SASL server state to AUTHENTICATE during authentication
(org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)> 
> [2019-05-15 10:14:00,814] DEBUG Setting SASL/SCRAM_SHA_512 server state to FAILED (org.apache.kafka.common.security.scram.internals.ScramSaslServer)>

> [2019-05-15 10:14:00,814] DEBUG Set SASL server state to FAILED during authentication
(org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)> 
> [2019-05-15 10:14:00,814] INFO [SocketServer brokerId=2] Failed authentication with 10.101.60.15
(Authentication failed during authentication due to invalid credentials with SASL mechanism
SCRAM-SHA-512) (org.apache.kafka.common.network.Selector)> 
> [2019-05-15 10:14:00,815] DEBUG [SocketServer brokerId=2] Connection with 10.101.60.15
disconnected (org.apache.kafka.common.network.Selector)> 
> java.io.EOFException> 
> at org.apache.kafka.common.network.SslTransportLayer.read(SslTransportLayer.java:573)>

> at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:94)>

> at org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.authenticate(SaslServerAuthenticator.java:267)>

> at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:173)> 
> at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:536)>

> at org.apache.kafka.common.network.Selector.poll(Selector.java:472)> 
> at kafka.network.Processor.poll(SocketServer.scala:830)> 
> at kafka.network.Processor.run(SocketServer.scala:730)> 
> at java.lang.Thread.run(Thread.java:748)> 
> 
> 
> # User creation in ZK & output> 
> /opt/kafka/bin/kafka-configs.sh --zookeeper xxxx:2181 --alter --add-config 'SCRAM-SHA-512=[password=xxxx]'
--entity-type users --entity-name admin> 
> entity-name admin> 
> Configs for user-principal 'admin' are SCRAM-SHA-512=salt=bnBicjI4NWd5dDBweGJoMmJ1bnlzdzFxYQ==,stored_key=xxxxx,server_key=xxxxxx==,iterations=4096>

> 
> 
> # ps fauxww> 
> kafka     2523  7.1 15.9 5838668 972848 ?      Ssl  mai14  52:46 java -Xmx1G -Xms1G -server
-XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent
-Djava.awt.headless=true -Xloggc:/var/log/kafka/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10
-XX:GCLogFileSize=100M -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9990 -Djava.security.auth.login.config=/opt/kafka/config/kafka_server_jaas.conf
-Djava.rmi.server.hostname=xxxxx -Dkafka.logs.dir=/var/log/kafka -Dlog4j.configuration=file:/opt/kafka/config/log4j.properties
-cp /opt/kafka/bin/../libs/activation-1.1.1.jar:/opt/kafka/bin/../libs/aopalliance-repackaged-2.5.0-b42.jar:/opt/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/bin/../libs/audience-annotations-0.5.0.jar:/opt/kafka/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka/bin/../libs/connect-api-2.2.0.jar:/opt/kafka/bin/../libs/connect-basic-auth-extension-2.2.0.jar:/opt/kafka/bin/../libs/connect-file-2.2.0.jar:/opt/kafka/bin/../libs/connect-json-2.2.0.jar:/opt/kafka/bin/../libs/connect-runtime-2.2.0.jar:/opt/kafka/bin/../libs/connect-transforms-2.2.0.jar:/opt/kafka/bin/../libs/guava-20.0.jar:/opt/kafka/bin/../libs/hk2-api-2.5.0-b42.jar:/opt/kafka/bin/../libs/hk2-locator-2.5.0-b42.jar:/opt/kafka/bin/../libs/hk2-utils-2.5.0-b42.jar:/opt/kafka/bin/../libs/jackson-annotations-2.9.8.jar:/opt/kafka/bin/../libs/jackson-core-2.9.8.jar:/opt/kafka/bin/../libs/jackson-databind-2.9.8.jar:/opt/kafka/bin/../libs/jackson-datatype-jdk8-2.9.8.jar:/opt/kafka/bin/../libs/jackson-jaxrs-base-2.9.8.jar:/opt/kafka/bin/../libs/jackson-jaxrs-json-provider-2.9.8.jar:/opt/kafka/bin/../libs/jackson-module-jaxb-annotations-2.9.8.jar:/opt/kafka/bin/../libs/javassist-3.22.0-CR2.jar:/opt/kafka/bin/../libs/javax.annotation-api-1.2.jar:/opt/kafka/bin/../libs/javax.inject-1.jar:/opt/kafka/bin/../libs/javax.inject-2.5.0-b42.jar:/opt/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/bin/../libs/javax.ws.rs-api-2.1.jar:/opt/kafka/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/bin/../libs/jersey-client-2.27.jar:/opt/kafka/bin/../libs/jersey-common-2.27.jar:/opt/kafka/bin/../libs/jersey-container-servlet-2.27.jar:/opt/kafka/bin/../libs/jersey-container-servlet-core-2.27.jar:/opt/kafka/bin/../libs/jersey-hk2-2.27.jar:/opt/kafka/bin/../libs/jersey-media-jaxb-2.27.jar:/opt/kafka/bin/../libs/jersey-server-2.27.jar:/opt/kafka/bin/../libs/jetty-client-9.4.14.v20181114.jar:/opt/kafka/bin/../libs/jetty-continuation-9.4.14.v20181114.jar:/opt/kafka/bin/../libs/jetty-http-9.4.14.v20181114.jar:/opt/kafka/bin/../libs/jetty-io-9.4.14.v20181114.jar:/opt/kafka/bin/../libs/jetty-security-9.4.14.v20181114.jar:/opt/kafka/bin/../libs/jetty-server-9.4.14.v20181114.jar:/opt/kafka/bin/../libs/jetty-servlet-9.4.14.v20181114.jar:/opt/kafka/bin/../libs/jetty-servlets-9.4.14.v20181114.jar:/opt/kafka/bin/../libs/jetty-util-9.4.14.v20181114.jar:/opt/kafka/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/bin/../libs/kafka_2.11-2.2.0.jar:/opt/kafka/bin/../libs/kafka_2.11-2.2.0-sources.jar:/opt/kafka/bin/../libs/kafka-clients-2.2.0.jar:/opt/kafka/bin/../libs/kafka-log4j-appender-2.2.0.jar:/opt/kafka/bin/../libs/kafka-streams-2.2.0.jar:/opt/kafka/bin/../libs/kafka-streams-examples-2.2.0.jar:/opt/kafka/bin/../libs/kafka-streams-scala_2.11-2.2.0.jar:/opt/kafka/bin/../libs/kafka-streams-test-utils-2.2.0.jar:/opt/kafka/bin/../libs/kafka-tools-2.2.0.jar:/opt/kafka/bin/../libs/log4j-1.2.17.jar:/opt/kafka/bin/../libs/lz4-java-1.5.0.jar:/opt/kafka/bin/../libs/maven-artifact-3.6.0.jar:/opt/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/kafka/bin/../libs/plexus-utils-3.1.0.jar:/opt/kafka/bin/../libs/reflections-0.9.11.jar:/opt/kafka/bin/../libs/rocksdbjni-5.15.10.jar:/opt/kafka/bin/../libs/scala-library-2.11.12.jar:/opt/kafka/bin/../libs/scala-logging_2.11-3.9.0.jar:/opt/kafka/bin/../libs/scala-reflect-2.11.12.jar:/opt/kafka/bin/../libs/slf4j-api-1.7.25.jar:/opt/kafka/bin/../libs/slf4j-log4j12-1.7.25.jar:/opt/kafka/bin/../libs/snappy-java-1.1.7.2.jar:/opt/kafka/bin/../libs/validation-api-1.1.0.Final.jar:/opt/kafka/bin/../libs/zkclient-0.11.jar:/opt/kafka/bin/../libs/zookeeper-3.4.13.jar:/opt/kafka/bin/../libs/zstd-jni-1.3.8-1.jar
kafka.Kafka /opt/kafka/config/server.properties> 
> 
> 
> # Broker conf> 
> auto.create.topics.enable=false> 
> broker.id=1> 
> compression.type=snappy> 
> delete.topic.enable=true> 
> listeners=SASL_SSL://:9093> 
> log.dir=/var/lib/kafka> 
> min.insync.replicas=2> 
> sasl.enabled.mechanisms=SCRAM-SHA-512,PLAIN> 
> sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512> 
> security.inter.broker.protocol=SASL_SSL> 
> ssl.client.auth=required> 
> ssl.enabled.protocols=TLSv1.2> 
> ssl.endpoint.identification.algorithm=> 
> ssl.key.password=xxx> 
> ssl.keystore.location=/opt/kafka/ssl/kafka.server.keystore.jks> 
> ssl.keystore.password=xxx> 
> ssl.keystore.type=JKS> 
> ssl.secure.random.implementation=SHA1PRNG> 
> ssl.truststore.location=/opt/kafka/ssl/kafka.server.truststore.jks> 
> ssl.truststore.password=xxx> 
> ssl.truststore.type=JKS> 
> 
> # /opt/kafka/config/kafka_server_jaas.conf> 
> KafkaServer {> 
>  org.apache.kafka.common.security.scram.ScramLoginModule required> 
>  username="admin"> 
>  password="adminpass";> 
> 
>  org.apache.kafka.common.security.plain.PlainLoginModule required> 
>  username="admin"> 
>  password="adminpass"> 
>  user_admin="adminpass"> 
>  user_app="blabla";> 
> };> 
> 
>  

Lep pozdrav / Kind regards,
Ivan JaneŇ°
System Engineer

Email: ivan.janes@oryxgaming.com
Skype: ivan.oryx
Mobile: +386 31 862 833




The information transmitted is intended only for the person or entity to which it is addressed
and may contain confidential and/or privileged material. Any review, re-transmission, dissemination
or other use of, or taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received this in error, please
contact the sender and delete the material from any computer.


Mime
View raw message