nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: Nifi 0.50 and GetKafka Issues
Date Sun, 21 Feb 2016 22:23:46 GMT
Yeah the intent is to support 0.8 and 0.9.  Will figure something out.

Thanks
Joe
On Feb 21, 2016 4:47 PM, "West, Joshua" <Josh_West@bose.com> wrote:

> Hi Oleg,
>
> Hmm -- from what I can tell, this isn't a Zookeeper communication issue.
> Nifi is able to connect into the Kafka brokers' Zookeeper cluster and
> retrieve the list of the kafka brokers to connect to.  Seems, from the
> logs, to be a problem when attempting to consume from Kafka itself.
>
> I'm guessing that the Kafka 0.9.0 client libraries just are not compatible
> with Kafka 0.8.2.1 so in order to use Nifi 0.5.0 with Kafka, the Kafka
> version must be >= 0.9.0.
>
> Any change Nifi could add backwards compatible support for Kafka 0.8.2.1
> too?  Let you choose which client library version, when setting up the
> GetKafka processor?
>
> --
> Josh West <josh_west@bose.com>
> Bose Corporation
>
>
>
> On Sun, 2016-02-21 at 15:02 +0000, Oleg Zhurakousky wrote:
>
> Josh
>
> Also, keep in mind that there are incompatible property names in Kafka
> between the 0.7 and 0.8 releases. One of the change that went it was
> replacing “zk.connectiontimeout.ms” with “zookeeper.connection.timeout.ms”.
> Not sure if it’s related though, but realizing that 0.4.1 was relying on
> this property it’s value was completely ignored with 0.8 client libraries
> (you could actually see the WARN message to that effect) and now it is not
> ignored, so take a look and see if tinkering with its value changes
> something.
>
> Cheers
> Oleg
>
> On Feb 20, 2016, at 6:47 PM, Oleg Zhurakousky <
> ozhurakousky@hortonworks.com> wrote:
>
> Josh
>
> The only change that ’s went and relevant to your issue is the fact that
> we’ve upgraded client libraries to Kafka 0.9 and between 0.8 and 0.9 Kafka
> introduced wire protocol changes that break compatibility.
> I am still digging so stay tuned.
>
> Oleg
>
> On Feb 20, 2016, at 4:10 PM, West, Joshua <Josh_West@bose.com> wrote:
>
> Hi Oleg and Joe,
>
> Kafka 0.8.2.1
>
> Attached is the app log with hostnames scrubbed.
>
> Thanks for your help.  Much appreciated.
>
> --
> Josh West <josh_west@bose.com>
> Bose Corporation
>
>
>
> On Sat, 2016-02-20 at 15:46 -0500, Joe Witt wrote:
>
> And also what version of Kafka are you using?
> On Feb 20, 2016 3:37 PM, "Oleg Zhurakousky" <ozhurakousky@hortonworks.com>
> wrote:
>
> Josh
>
> Any chance to attache the app-log or relevant stack trace?
>
> Thanks
> Oleg
>
> On Feb 20, 2016, at 3:30 PM, West, Joshua <Josh_West@bose.com> wrote:
>
> Hi folks,
>
> I've upgraded from Nifi 0.4.1 to 0.5.0 and I am no longer able to use the
> GetKafka processor.  I'm seeing errors like so:
>
> 2016-02-20 20:10:14,953 WARN
> [ConsumerFetcherThread-NiFi-sldjflkdsjflksjf_**SCRUBBED**-1455999008728-5b8c7108-0-0]
> kafka.consumer.ConsumerFetcherThread
> [ConsumerFetcherThread-NiFi-sldjflkdsjflksjf_**SCRUBBED**-1455999008728-5b8c7108-0-0],
> Error in fetchkafka.consumer.ConsumerFetcherThread$FetchRequest@7b49a642.
> Possible cause: java.lang.IllegalArgumentException
>
> ^ Note  the hostname of the server has been scrubbed.
>
> My configuration is pretty generic, except that with Zookeeper we use a
> different root path, so our Zookeeper connect string looks like so:
>
> zookeeper-node1:2181,zookeeper-node2:2181,zookeeper-node3:2181/kafka
>
> Is anybody else experiencing issues?
>
> Thanks.
>
> --
> Josh West <josh_west@bose.com>
>
> Cloud Architect
> Bose Corporation
>
>
>
>
>
> <nifi-app.log.kafkaissues.bz2>
>
>
>
>

Mime
View raw message