spot-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Shlyakhtin <Vladimir.Shlyakh...@sstech.us>
Subject RE: Kerberos support
Date Thu, 07 Sep 2017 20:17:24 GMT
Yes, I tried, I added new arguments to spot-ingest/common/kafka_client.py to start method:
consumer =  KC(bootstrap_servers=[kafka_brokers],group_id=self._topic,security_protocol='SASL_PLAINTEXT',sasl_mechanism='GSSAPI')

Used kafka-python provided by pip (1.3.4)

Regards,
- Vladimir

________________________________________
From: Nathanael Smith [nathanael@apache.org]
Sent: Thursday, September 07, 2017 4:08 PM
To: dev@spot.incubator.apache.org
Subject: Re: Kerberos support

assuming you are using the latest kaka-python version,
Have you tried using the below with the kafka producer/consumer?

security_protocol=“SASL_PLAINTEXT”,sasl_mechanism=“GSSAPI”

- Nathanael



> On Sep 7, 2017, at 12:53 PM, Nathanael Smith <nathanael@apache.org> wrote:
>
> Hi Vladimir,
>
> 1. Apache Spot does not support kerberos as of the current release (1.0).
>
> 2. My first test with python-kafka  using “SASL_PLAINTEXT” and “GSSAPI” is resulting
in no brokers found.
> I’m not able to say at this moment that this was completely a code issue not something
with infrastructure.
>
> That being said,
> ____
> Pro’s for librdkafka:
>
> Kerberos has been supported for quite some time
> Used by several large projects including Confluent's own library
>
> Cons for librdkafka:
>
> requires more setup, This is not a show stopper by any means but should be considered
as it’s not a simple pip install.
> will require more development time
> ____
>
> I think if librdkafka can be implemented quickly and the usage of the KafkaTopic and
KafkaConsumer in spot-ingest/common/kafka_client.py remains the same it will add some stability.
>
> - Nathanael
>
>
>
>> On Sep 7, 2017, at 7:17 AM, Vladimir Shlyakhtin <vladimir.shlyakhtin@sstech.us>
wrote:
>>
>> Hello,
>>
>> I'm just wondering if Apache Spot fully supports Kerberos. As far as I understand
not yet. (According to jira cases in "Spot Kerberos enablement" Epic).
>> Currently I'm working on installing Apache Spot on Kerberized environment. So was
created spot principal, granted access to hive, kafka and hdfs. Now I'm trying to run ingest
component.
>> But in ingest's code I see only "kinit" call. For python-kafka it is not enough.
As it is required security_protocol and sasl_mechanism for creation KafkaConsumer.
>> I did such changes but still get exception "kafka.errors.NoBrokersAvailable: NoBrokersAvailable"
>> It is the same as was reported in: https://github.com/dpkp/kafka-python/issues/1189
>> Nathanael mentioned in SPOT-77 about librdkafka.
>> What do you think? Is it worth to switch to librdkafka or continue use python-kafka.
>>
>> Thanks
>>
>> Regards,
>> - Vladimir
>


Mime
View raw message