nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Davy De Waele <ddewa...@gmail.com>
Subject Re: Processors on the fly for many sensor devices
Date Mon, 03 Oct 2016 14:56:22 GMT
Hi Andrew,

We'd be happy to test and provide feedback.
We have a use-case now with Nifi that we're in the process of implementing.

If you have sources or binaries feel free to let me know.....

Regards,
Davy

On Fri, Sep 30, 2016 at 9:24 PM, Andrew Psaltis <psaltis.andrew@gmail.com>
wrote:

> Hi Davy,
> Sorry for the slow response, I am traveling today. However, I should be
> able to get a build of it for 1.0.0 out later today or tomorrow.
>
> Thanks,
> Andrew
>
> On Fri, Sep 30, 2016 at 6:54 PM, Davy De Waele <ddewaele@gmail.com> wrote:
>
>> Hi Andrew,
>>
>> We tested your 0.0.6 binary and it seemed to do what we want.
>> Unfortunately it is not compatible with the current Nifi version.
>> Do you have a binary available that would work with nifi 1.0.0 ?
>>
>> Thx.
>>
>> On Wed, Sep 28, 2016 at 11:39 PM, Andrew Psaltis <
>> psaltis.andrew@gmail.com> wrote:
>>
>>> Davy,
>>> The processor I have been working on may meet your needs. You are
>>> correct, at this time I have not pushed the source for it, still working
>>> through some hurdles. The one thing to work out is how you would
>>> dynamically add the processors -- suppose you may be able to use the REST
>>> API for NiFi. I would imagine there are quite a number of these devices
>>> that you would need to have processors for. In the use case I have been
>>> working on, there may be 600 or so endpoints that I need to connect to and
>>> I'm trying to figure out does it make sense to do it this way.
>>>
>>> I'll hopefully be in a place soon that I can push the code I have for
>>> the GetTCP processor.
>>>
>>>
>>>
>>>
>>> On Thu, Sep 29, 2016 at 5:22 AM, Bryan Bende <bbende@gmail.com> wrote:
>>>
>>>> Just wanted to clarify something about ListenTCP... it does support
>>>> multiple incoming connections, however if you using the batch output
>>>> capability, one flow file will contain data across all the connections.
>>>>
>>>> I do agree with Andrew that based on the description it sounds like
>>>> NiFi is expected to be the client that initiates a connection and keeps
>>>> reading for some amount of time/threshold, like a GetTCP processor.
>>>>
>>>> Currently we have ListenTCP which is waiting for incoming connections,
>>>> and PutTCP which makes a connection, but only writes data.
>>>>
>>>> On Wed, Sep 28, 2016 at 5:02 PM, Andrew Psaltis <
>>>> psaltis.andrew@gmail.com> wrote:
>>>>
>>>>> Davy,
>>>>> It sounds like you need a GetTCP type of processor that connects from
>>>>> NiFi to the TCP endpoint on the sensor, is that correct?
>>>>>
>>>>> Thanks,
>>>>> Andrew
>>>>>
>>>>> On Thu, Sep 29, 2016 at 4:46 AM, Davy De Waele <ddewaele@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Thanks for the response ... it's an existing network of sensors.
The
>>>>>> sensors spit out data over a serial interface that is exposed over
a tcp
>>>>>> connection. (rs232 -> ethernet converter in the sensor).
>>>>>> The current sensor architecture involves clients making direct
>>>>>> connections to the individual sensors. (establishing a tcp connection
to
>>>>>> the specific ip of the sensor).
>>>>>>
>>>>>> If I understand correctly, ListenTCP would not work in this case
for
>>>>>> multiple sensors
>>>>>>
>>>>>> Are you talking about a setup where the sensors would be in a
>>>>>> "client" mode where each sensor would each establish a tcp connections
to a
>>>>>> single ListTCP processor  ?
>>>>>>
>>>>>> Thx
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Sep 28, 2016 at 10:03 PM, Joe Witt <joe.witt@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello
>>>>>>>
>>>>>>> Can you talk a bit about why you'd want ListenTCP processors
tied to
>>>>>>> a
>>>>>>> given sensor?  You should be able to have many sensors to a single
>>>>>>> ListenTCP.  Each stream will be between a source/sensor and nifi
so
>>>>>>> data won't be getting intermingled there.  If we're not providing
>>>>>>> enough session/stream metadata on the flow files to make demux
of the
>>>>>>> streams easy using something like RouteOnAttribute or whatnot
we
>>>>>>> definitely should.
>>>>>>>
>>>>>>> Now, that said, you could certainly programmatically deploy (via
the
>>>>>>> REST API) instances of these processors along the lines of what
your
>>>>>>> endpoint registry tells you.  It just seems on the surface like
doing
>>>>>>> so would be avoidable at least for the listening of data.  Typically
>>>>>>> such a registry would be useful to do additional tagging/enrichment
>>>>>>> of
>>>>>>> the data and would occur once it is in the flow.
>>>>>>>
>>>>>>> Thanks
>>>>>>> Joe
>>>>>>>
>>>>>>> On Wed, Sep 28, 2016 at 3:39 PM, Davy De Waele <ddewaele@gmail.com>
>>>>>>> wrote:
>>>>>>> > We have a large number of sensors that send out data via
TCP. The
>>>>>>> idea is to
>>>>>>> > use a ListenTCP processor in Nifi to capture the data, do
some
>>>>>>> filtering /
>>>>>>> > basic transformation before sending it upstream into our
stack.
>>>>>>> >
>>>>>>> > We can configure individual ListenTCP processors for each
sensor,
>>>>>>> and that
>>>>>>> > works fine when the number of sensors is small, but once
you hit a
>>>>>>> larger
>>>>>>> > number if becomes cumbersome and difficult to manage.
>>>>>>> >
>>>>>>> > We have an inventory of those sensors (exposed via a REST
service
>>>>>>> endpoint),
>>>>>>> > containing  the sensor tcp information like ip and port)
>>>>>>> >
>>>>>>> > Is there an easy way to create these ListenTCP processors
on the
>>>>>>> fly based
>>>>>>> > on a REST endpoint or some other external configuration
? How
>>>>>>> would that
>>>>>>> > work ?
>>>>>>> >
>>>>>>> > Thx.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks,
>>>>> Andrew
>>>>>
>>>>> Subscribe to my book: Streaming Data <http://manning.com/psaltis>
>>>>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
>>>>> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Andrew
>>>
>>> Subscribe to my book: Streaming Data <http://manning.com/psaltis>
>>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
>>> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
>>>
>>
>>
>
>
> --
> Thanks,
> Andrew
>
> Subscribe to my book: Streaming Data <http://manning.com/psaltis>
> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
>

Mime
View raw message