nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kant kodali <kanth...@gmail.com>
Subject Re: How to integrate a custom protocol with Apache Nifi
Date Wed, 07 Dec 2016 20:08:46 GMT
so do I need to import PostHTTP processor, GetHTTP processor Libraries in
my application code? or can I just build a HTTP Server and tell Nifi JVM
that "hey here is my Application (it runs on specific IP and port) and it
uses HTTP".

On Wed, Dec 7, 2016 at 12:00 PM, Oleg Zhurakousky <
ozhurakousky@hortonworks.com> wrote:

> Kant
>
> So, yes NiFi already provides basic integration with HTTP by virtue of
> PostHTTP processor (post a content of the FlowFile to HTTP endpoint) and
> the same in reverse with GetHTTP. So for you case your integration will be
> as simple as posting or getting from your micro service
>
> And what makes it even more interesting is that NIFi in the microservices
> architecture can play an even more significant role as coordinator,
> orchestrator and mediator of heterogeneous environment of many micro
> services.
>
> Hope that answers it.
> Cheers
> Oleg
>
>
> On Dec 7, 2016, at 2:48 PM, kant kodali <kanth909@gmail.com> wrote:
>
> Sorry I should have been more clear. My question is even more simpler and
> naive. Say I am writing a HTTP based microservice (Now I assume Nifi has
> integration with HTTP since it is a well known protocol ). Now, how would I
> integrate Nifi with my HTTP based server?
>
> On Wed, Dec 7, 2016 at 4:55 AM, Oleg Zhurakousky <
> ozhurakousky@hortonworks.com> wrote:
>
>> Kant
>>
>> There are couple of questions here so, let me try one at the time.
>> 1. “why Nifi would act as a server?”. Well, NiFi is a runtime environment
>> where things are *happening*. To be more exact; NiFi is a runtime
>> environment where things are *triggered*. The big distinction here is
>> trigger vs happening. In other words you may choose (as most processors do)
>> to have NiFi act as an execution container and run your code, or you may
>> chose for the NiFi to be a triggering container that triggers to run your
>> code elsewhere. Example of the later one is the SpringContextProcessor
>> which while still runs in the same JVM delegates execution to Spring
>> application that runs in its own container (which could as well be a
>> separate JVM or nah other container - ala micro services). So in this case
>> NiFi still manages the orchestration, mediation, security etc.
>>
>> 2. “...do I need to worry about how my service would talk to Nifi or do I
>> have the freedom of just focusing on my application and using whatever
>> protocol I want and In the end just plugin to Nifi which would take care?”
>> That is a loaded question and I must admit not fully understood, but i’ll
>> give it a try. When integrating with NiFi you use one of the integration
>> points such as Processor and/or ControllerService. Those are NiFi known
>> strategies (interfaces) and so your custom Processor would need to
>> implement such strategy and obviously be compliant with its contract.
>> However, the other part of your question is about implementing something
>> “independent” and just plug-in and if t’s possible. My answer is still YES
>> as long as you design it that way. As an example of such design you may
>> want to look at AMQP support where there are a pair of processors
>> (PublishAMQP/ConsumeAMQP), but when you look at the code of these
>> processors you’ll see that neither has any AMQP dependencies. Instead they
>> depend on another case (let’s call it Worker) and that class is completely
>> independent of NiFi. So in summary your protocol-specific Processor is
>> independent of the protocol and your protocol-specific Worker is
>> independent of NiFi and the two delegate between one another through common
>> to both object (in this case bye[]). The Spring processor mentioned above
>> implements the same pattern.
>>
>> And one more point about Microservices. I am willing to go as far as
>> saying that NiFi is a variation of Microservices container. I am basing it
>> n the fact that NiFi components (i.e., processors) implement a fundamental
>> micro services pattern  - certain independence from it’s runtime and
>> persistence of its results - which allows each and every component to be
>> managed (start/stopped/reconfigured/changed) independently of
>> upstream/downstream components.
>>
>> Ok, that is a load to process, so I’ll stop ;)
>>
>> Look forward to more questions
>>
>> Cheers
>> Oleg
>>
>>
>> On Dec 7, 2016, at 4:52 AM, kant kodali <kanth909@gmail.com> wrote:
>>
>> I am also confused a little bit since I am new to Nifi. I wonder why Nifi
>> would act as a server? isn't Nifi a routing layer between systems? because
>> this brings in another question about Nifi in general.
>>
>> When I write my applications/microservices do I need to worry about how
>> my service would talk to Nifi or do I have the freedom of just focusing on
>> my application and using whatever protocol I want and In the end just
>> plugin to Nifi which would take care? other words is Nifi a tight
>> integration with applications such that I always have to import a Nifi
>> Library within my application/microservice ? other words do I need to worry
>> about Nifi at programming/development time of an Application/Microservice
>> or at deployment time?
>>
>> Sorry if these are naive questions. But answers to those will help
>> greatly and prevent me from asking more questions!
>>
>> Thanks much!
>> kant
>>
>>
>>
>>
>> On Wed, Dec 7, 2016 at 1:23 AM, kant kodali <kanth909@gmail.com> wrote:
>>
>>> Hi Koji,
>>>
>>> That is an awesome explanation! I expected processors for HTTP2 at very
>>> least since it is widely used ( the entire GRPC stack runs on that). I am
>>> not sure how easy or hard it is to build one?
>>>
>>> Thanks!
>>>
>>> On Wed, Dec 7, 2016 at 1:08 AM, Koji Kawamura <ijokarumawak@gmail.com>
>>> wrote:
>>>
>>>> Hi Kant,
>>>>
>>>> Although I'm not aware of existing processor for HTTP2 or NSQ, NiFi
>>>> has a set of processors for WebSocket since 1.1.0.
>>>> It enables NiFi to act as a WebSocket client to communicate with a
>>>> remote WebSocket server, or makes NiFi a WebSocket server so that
>>>> remote clients access to it via WebSocket protocol.
>>>>
>>>> I've written a blog post about how to use it, I hope it will be useful
>>>> for your use case:
>>>> http://ijokarumawak.github.io/nifi/2016/11/04/nifi-websocket/
>>>>
>>>> Thanks,
>>>> Koji
>>>>
>>>>
>>>>
>>>> On Wed, Dec 7, 2016 at 3:23 PM, kant kodali <kanth909@gmail.com> wrote:
>>>> > Thanks a ton guys! Didn't expect Nifi community to be so good!
>>>> (Another
>>>> > convincing reason!)
>>>> >
>>>> > Coming back to the problem, We use NSQ a lot (although not my
>>>> favorite) and
>>>> > want to be able integrate Nifi with NSQ to other systems such as
>>>> Kafka,
>>>> > Spark, Cassandra, ElasticSearch, some micro services that uses HTTP2
>>>> and
>>>> > some micro service that uses Websockets
>>>> >
>>>> > (which brings in other question if Nifi has HTTP2 and Websocket
>>>> processors?)
>>>> >
>>>> > Also below is NSQ protocol spec. They have Java client library as
>>>> well.
>>>> >
>>>> > http://nsq.io/clients/tcp_protocol_spec.html
>>>> >
>>>> >
>>>> > On Tue, Dec 6, 2016 at 5:14 PM, Oleg Zhurakousky
>>>> > <ozhurakousky@hortonworks.com> wrote:
>>>> >>
>>>> >> Hi Kant
>>>> >>
>>>> >> What you’re trying to accomplish is definitely possible, however
more
>>>> >> information may be needed from you.
>>>> >> For example, the way I understand your statement about “integration
>>>> with
>>>> >> many systems” is something like JMS, Kafka, TCP, FTP etc…. If
that
>>>> is the
>>>> >> case such integration is definitely possible with your “custom
>>>> system” by
>>>> >> developing custom Processor and/or ControllerService.
>>>> >> Processors and ControllerServices are the two main integration points
>>>> >> within NiFi
>>>> >> You can definitely find may examples by looking at some of the
>>>> processors
>>>> >> (i.e., PublishKafka or ConsumeKafka, PublishJMS or ConsumeJMS etc.)
>>>> >>
>>>> >> Let us know if you need more help to guide you through the process.
>>>> >>
>>>> >> Cheers
>>>> >> Oleg
>>>> >>
>>>> >> > On Dec 6, 2016, at 7:46 PM, kant kodali <kanth909@gmail.com>
>>>> wrote:
>>>> >> >
>>>> >> > HI All,
>>>> >> >
>>>> >> > I understand that Apache Nifi has integration with many systems
>>>> but what
>>>> >> > If I have an application that talks a custom protocol ? How
do I
>>>> integrate
>>>> >> > Apache Nifi with the custom protocol?
>>>> >> >
>>>> >> > Thanks,
>>>> >> > kant
>>>> >>
>>>> >
>>>>
>>>
>>>
>>
>>
>
>

Mime
View raw message