nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From p pathiyil <pathi...@gmail.com>
Subject Re: Using MiNiFi as a library from with a Java program
Date Wed, 27 Sep 2017 03:04:54 GMT
As an alternative approach, is it conceivable to write a Custom Processor
and run that in the MiNiFi 'pipeline' without connecting back to NiFi
(Custom Processor -> Some Connector with socket / buffer input ->
PublishToKafka) ? This wouldn't hide the MiNiFi installation / runtime from
the user, but will keep the 'agent' independent of the NiFi backend.





On Wed, Sep 27, 2017 at 6:50 AM, p pathiyil <pathiyil@gmail.com> wrote:

> Thanks for the pointer Joe. Let me look at Kafka Connect and see if that
> can be used to address this use case.
>
> On Wed, Sep 27, 2017 at 6:35 AM, Joe Witt <joe.witt@gmail.com> wrote:
>
>> Praveen
>>
>> I think the direction of MiNiFi and where it can head will support
>> this case nicely.  Today though I dont think we offer the simple
>> library model you're looking for.  Fortunately, for Apache Kafka
>> specifically their community has developed Kafka Connect [1] which
>> sounds like it could be just what you need.
>>
>> Take a look at that and if that gets you where you want to be then
>> great.  If not, or I've misunderstood the ask please let us know.
>>
>> [1] https://kafka.apache.org/documentation/#connect
>>
>> Thanks
>> Joe
>>
>> On Tue, Sep 26, 2017 at 8:50 PM, p pathiyil <pathiyil@gmail.com> wrote:
>> > Hi Aldrin,
>> >
>> > I am looking at a couple of different flows, but to take the most simple
>> > scenario, I would like to leverage the PublishToKafka_0_10 processor of
>> > MiNiFi alone. The application will obtain some data, send it to a
>> Processor
>> > that has a simple upstream connection interface like a socket or in
>> memory
>> > buffer (so that conversion to FlowFiles can be taken care of in that
>> > Processor) and then use PublishToKafka_0_10 to directly publish to a
>> Kafka
>> > cluster. Will that be easy enough to do ?
>> >
>> > Thanks,
>> > Praveen.
>> >
>> >
>> >
>> >
>> > On Wed, Sep 27, 2017 at 12:24 AM, Aldrin Piri <aldrinpiri@gmail.com>
>> wrote:
>> >>
>> >> Hello,
>> >>
>> >> Neither NiFi nor MiNiFi (Java) is currently packaged as such.  Would
>> you
>> >> be able to expand upon your use case of publishing to Kafka and/or what
>> >> facets of MiNiFi you are looking to utilize?  There may be other
>> avenues
>> >> that get you toward your solution.
>> >>
>> >> Thanks,
>> >> Aldrin
>> >>
>> >> On Tue, Sep 26, 2017 at 1:03 PM, p pathiyil <pathiyil@gmail.com>
>> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> I am starting to look at MiNiFi for a use case that involves
>> publishing
>> >>> to Kafka. Is it feasible to use MiNiFi as a library (or set of
>> libraries)
>> >>> from within another Java program ? The primary reason for looking at
>> such an
>> >>> approach is the desire to hide the MiNiFi installation / process
>> management
>> >>> from the users of my application.
>> >>>
>> >>> Thanks.
>> >>>
>> >>
>> >
>>
>
>

Mime
View raw message