nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kant kodali <kanth...@gmail.com>
Subject Re: How to integrate a custom protocol with Apache Nifi
Date Wed, 07 Dec 2016 19:48:22 GMT
Sorry I should have been more clear. My question is even more simpler and
naive. Say I am writing a HTTP based microservice (Now I assume Nifi has
integration with HTTP since it is a well known protocol ). Now, how would I
integrate Nifi with my HTTP based server?

On Wed, Dec 7, 2016 at 4:55 AM, Oleg Zhurakousky <
ozhurakousky@hortonworks.com> wrote:

> Kant
>
> There are couple of questions here so, let me try one at the time.
> 1. “why Nifi would act as a server?”. Well, NiFi is a runtime environment
> where things are *happening*. To be more exact; NiFi is a runtime
> environment where things are *triggered*. The big distinction here is
> trigger vs happening. In other words you may choose (as most processors do)
> to have NiFi act as an execution container and run your code, or you may
> chose for the NiFi to be a triggering container that triggers to run your
> code elsewhere. Example of the later one is the SpringContextProcessor
> which while still runs in the same JVM delegates execution to Spring
> application that runs in its own container (which could as well be a
> separate JVM or nah other container - ala micro services). So in this case
> NiFi still manages the orchestration, mediation, security etc.
>
> 2. “...do I need to worry about how my service would talk to Nifi or do I
> have the freedom of just focusing on my application and using whatever
> protocol I want and In the end just plugin to Nifi which would take care?”
> That is a loaded question and I must admit not fully understood, but i’ll
> give it a try. When integrating with NiFi you use one of the integration
> points such as Processor and/or ControllerService. Those are NiFi known
> strategies (interfaces) and so your custom Processor would need to
> implement such strategy and obviously be compliant with its contract.
> However, the other part of your question is about implementing something
> “independent” and just plug-in and if t’s possible. My answer is still YES
> as long as you design it that way. As an example of such design you may
> want to look at AMQP support where there are a pair of processors
> (PublishAMQP/ConsumeAMQP), but when you look at the code of these
> processors you’ll see that neither has any AMQP dependencies. Instead they
> depend on another case (let’s call it Worker) and that class is completely
> independent of NiFi. So in summary your protocol-specific Processor is
> independent of the protocol and your protocol-specific Worker is
> independent of NiFi and the two delegate between one another through common
> to both object (in this case bye[]). The Spring processor mentioned above
> implements the same pattern.
>
> And one more point about Microservices. I am willing to go as far as
> saying that NiFi is a variation of Microservices container. I am basing it
> n the fact that NiFi components (i.e., processors) implement a fundamental
> micro services pattern  - certain independence from it’s runtime and
> persistence of its results - which allows each and every component to be
> managed (start/stopped/reconfigured/changed) independently of
> upstream/downstream components.
>
> Ok, that is a load to process, so I’ll stop ;)
>
> Look forward to more questions
>
> Cheers
> Oleg
>
>
> On Dec 7, 2016, at 4:52 AM, kant kodali <kanth909@gmail.com> wrote:
>
> I am also confused a little bit since I am new to Nifi. I wonder why Nifi
> would act as a server? isn't Nifi a routing layer between systems? because
> this brings in another question about Nifi in general.
>
> When I write my applications/microservices do I need to worry about how my
> service would talk to Nifi or do I have the freedom of just focusing on my
> application and using whatever protocol I want and In the end just plugin
> to Nifi which would take care? other words is Nifi a tight integration with
> applications such that I always have to import a Nifi Library within my
> application/microservice ? other words do I need to worry about Nifi at
> programming/development time of an Application/Microservice or at
> deployment time?
>
> Sorry if these are naive questions. But answers to those will help greatly
> and prevent me from asking more questions!
>
> Thanks much!
> kant
>
>
>
>
> On Wed, Dec 7, 2016 at 1:23 AM, kant kodali <kanth909@gmail.com> wrote:
>
>> Hi Koji,
>>
>> That is an awesome explanation! I expected processors for HTTP2 at very
>> least since it is widely used ( the entire GRPC stack runs on that). I am
>> not sure how easy or hard it is to build one?
>>
>> Thanks!
>>
>> On Wed, Dec 7, 2016 at 1:08 AM, Koji Kawamura <ijokarumawak@gmail.com>
>> wrote:
>>
>>> Hi Kant,
>>>
>>> Although I'm not aware of existing processor for HTTP2 or NSQ, NiFi
>>> has a set of processors for WebSocket since 1.1.0.
>>> It enables NiFi to act as a WebSocket client to communicate with a
>>> remote WebSocket server, or makes NiFi a WebSocket server so that
>>> remote clients access to it via WebSocket protocol.
>>>
>>> I've written a blog post about how to use it, I hope it will be useful
>>> for your use case:
>>> http://ijokarumawak.github.io/nifi/2016/11/04/nifi-websocket/
>>>
>>> Thanks,
>>> Koji
>>>
>>>
>>>
>>> On Wed, Dec 7, 2016 at 3:23 PM, kant kodali <kanth909@gmail.com> wrote:
>>> > Thanks a ton guys! Didn't expect Nifi community to be so good! (Another
>>> > convincing reason!)
>>> >
>>> > Coming back to the problem, We use NSQ a lot (although not my
>>> favorite) and
>>> > want to be able integrate Nifi with NSQ to other systems such as Kafka,
>>> > Spark, Cassandra, ElasticSearch, some micro services that uses HTTP2
>>> and
>>> > some micro service that uses Websockets
>>> >
>>> > (which brings in other question if Nifi has HTTP2 and Websocket
>>> processors?)
>>> >
>>> > Also below is NSQ protocol spec. They have Java client library as well.
>>> >
>>> > http://nsq.io/clients/tcp_protocol_spec.html
>>> >
>>> >
>>> > On Tue, Dec 6, 2016 at 5:14 PM, Oleg Zhurakousky
>>> > <ozhurakousky@hortonworks.com> wrote:
>>> >>
>>> >> Hi Kant
>>> >>
>>> >> What you’re trying to accomplish is definitely possible, however more
>>> >> information may be needed from you.
>>> >> For example, the way I understand your statement about “integration
>>> with
>>> >> many systems” is something like JMS, Kafka, TCP, FTP etc…. If that
is
>>> the
>>> >> case such integration is definitely possible with your “custom
>>> system” by
>>> >> developing custom Processor and/or ControllerService.
>>> >> Processors and ControllerServices are the two main integration points
>>> >> within NiFi
>>> >> You can definitely find may examples by looking at some of the
>>> processors
>>> >> (i.e., PublishKafka or ConsumeKafka, PublishJMS or ConsumeJMS etc.)
>>> >>
>>> >> Let us know if you need more help to guide you through the process.
>>> >>
>>> >> Cheers
>>> >> Oleg
>>> >>
>>> >> > On Dec 6, 2016, at 7:46 PM, kant kodali <kanth909@gmail.com>
wrote:
>>> >> >
>>> >> > HI All,
>>> >> >
>>> >> > I understand that Apache Nifi has integration with many systems
but
>>> what
>>> >> > If I have an application that talks a custom protocol ? How do
I
>>> integrate
>>> >> > Apache Nifi with the custom protocol?
>>> >> >
>>> >> > Thanks,
>>> >> > kant
>>> >>
>>> >
>>>
>>
>>
>
>

Mime
View raw message