nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Payne <marka...@hotmail.com>
Subject RE: GetKafka Processor and Hardcoded Kafka Consumer Configs
Date Tue, 28 Jul 2015 00:14:26 GMT
Jeremiah,

We can certainly enable the "auto.offset.reset" to be configurable. Not sure how making the
"auto.commit.enable" configurable would work.
Are you thinking that another property would be added to indicate how often to commit? Or
would it work completely differently? Just need that
fleshed out a bit more.

I do like the suggestion of exposing the config properties as user-defined properties. 

I have created a ticket to track this information: https://issues.apache.org/jira/browse/NIFI-791

Please feel free to update the ticket with any relevant information as you think of it.

Thanks!
-Mark

----------------------------------------
> Date: Mon, 27 Jul 2015 15:42:37 -0600
> Subject: GetKafka Processor and Hardcoded Kafka Consumer Configs
> From: jeremiah.adams@pearson.com
> To: dev@nifi.apache.org
>
> The GetKafka processor has a couple of Kafka Consumer Config values that
> are hard-coded.
>
> props.setProperty("auto.commit.enable", "true"); // just be explicit
> props.setProperty("auto.offset.reset", "smallest");
>
> These should be configurable property values in the Processor. Most
> notable for me is the "auto.offset.reset". Smallest vs. Largest has some
> implications concerning fault tolerance strategies.
>
> It would be best to expose all of the available Kafka Consumer Config
> properties. If these change though between kafka versions it would create
> maintenance work for the Processors.
>
> Another option would be to allow ad-hoc property values and end-user just
> supply the kafka config values they want to override.
>
>
> *Jeremiah Adams*
>
> Senior Software Developer
> Pearson
>
> 2154 East Commons Ave.
> Suite 400
> Centennial, CO 80122
>
>
> Always Learning
> Learn more at www.pearson.com
 		 	   		  
Mime
View raw message