nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Julian Feinauer <j.feina...@pragmaticminds.de>
Subject Re: NiFi for IoT Analytics
Date Mon, 01 Oct 2018 19:21:58 GMT
Hi Marc,

thanks to you also for your reply and your thoughts.
I cannot comment to the C++ development as we are at home in the java world and for our use
cases the footprint of MiNiFis Java Runtime is fine.
But Indeed, your exemplary question is pretty close to what we hear day to day, and all sorts
of related questions like “for long has it been down”, “how often in the last x days”,
… .

Regarding your Sensor Example, this is pretty close to where I come from, and last week Chris,
one of the plc4x commiters created a NiFi integration for plc4x which allows to query PLCs
and ingest the data to NiFi or MiNiFi (java) Dataflows, which is kind of an extension / complement
to your Code, as I understand it.

What you write seems to be in good accordance to what we reason about a lot, building blocks
for analytics with powerful and adequate semantics for analyzing this kind of data.
In fact, we have some of this already achieved in our “home made” framework which we want
to join with an eco system to leverage its abilities.

I think you answered some of the questions I didn’t ask which is extremely helpful for me
and as I see it you are working on problems related to ours which gives me the impression
that MiNiFi / NiFi can’t be that wrong of a tool.

To your last question, you are absolutely right, keeping and managing state is the core of
what we are doing (we used to refer to it as stateful stream processing, but nowadays this
term seem to be overused a bit). We used Apache Flink for that but for Edge applications this
is overkill.

Best
Julian

PS.: I hope that I find the time to look at the video posted by Joe tomorrow to deepen my
understanding of NiFi

Von: Marc Parisi <phrocker@apache.org>
Antworten an: "users@nifi.apache.org" <users@nifi.apache.org>
Datum: Montag, 1. Oktober 2018 um 19:25
An: "users@nifi.apache.org" <users@nifi.apache.org>
Betreff: Re: NiFi for IoT Analytics

Julian,

Very good question. One of the issues I've been tackling has been 'how to ease development'
within MiNiFi C++. Users and developers will likely have similar questions to yours, a common
one I've heard on forums is "[h]ow can I monitor for pull-down failure in this circuit?" There
are a variety of ways we could answer that and a variety of ways we could get that information,
but it's perfectly reasonable that consumers want this answer to be sent to NiFi.

  One thing I've wrestled with is the fact that any given library you have to support sensor
or device management is going to fit every use case [1]  -- so how can we ease the analysis
of that type of information without creating individual stovepipes of flows. I've been thinking
a lot about enabling those who wish to use MiNIFi as a library, supporting a variety of sensor/control
libraries in a variety of a languages using MiNIFi's C library as an interconnect. This might
help you capture that bit and monitor how long it's been there, facilitating the endpoints
around it to store that information. This piggybacks on what Joe said, but I think enabling
the python scripts, c programs, etc that already exist might help improve the development
process while also giving the capabilities that people need.

This brings me to some other ideas we've been working on -- create useful building blocks
to facilitate this type of development. This may mean providing facilities for the languages
to keep and compute state -- as consumables for these specific purposes. A great example might
be to monitor a sensor in python, capture and store that information with a few library functions
to send back through NiFi where you may perform further evaluation/analysis on all sensor
devices.

I know I haven't answered your question, but I'm trying to selfishly gauge interest with this
type of work. Some of it already exists, but knowing the value of these capabilities is supremely
helpful.

In the end is the state management of most importance to your evaluation or am I completely
off base?

  [1]https://github.com/apache/nifi-minifi-cpp/tree/master/extensions/sensors -- specific
to certain sensors and servos and likely not wildly helpful to others.

  Thanks,
  Marc

On Sun, Sep 30, 2018 at 7:58 AM Julian Feinauer <j.feinauer@pragmaticminds.de<mailto:j.feinauer@pragmaticminds.de>>
wrote:
Hi Nifi-team,

I’m from the incubating plc4x project [1] and I am looking for a framework which is suitable
for the management of IoT Datastreams and do some edge computing.
As nifi is often times mentioned in relation with IoT I tried to find out what nifi realy
does and how it would fit with our ideas (and also the MiNiFi Project seems to fit into this).

From what I understood from the Docs and some Videos NiFi looks for me a bit like Apache Camel
[2] as it is able to (dynamically) integrate different systems and manage the dataflow between
them. So what I did not get exactly I how the payloads are managed between these Endpoints
and how much of processing Nifi does itself and how much it delegates to other components
(like e.g. Service Activater in EIP).

What I am looking for is a framework which does some analysis of data streams coming from
controllers that, e.g., control machines or robots. chrisdutz already prepared the first version
of an NiFi Endpoint in th Plc4x Repo so we are already able to stream these datasets to NiFi.
Whats unclear to me is how we could tackle some of the questions like “how long was this
bit set” or “notify me when this signal is below a certain threshold for more than 30s”
or so.
Is this in the scope of NiFi or is NiFi more of an integration / data-flow layer which is
absolutely agnostic of these processing blocks?

I hope my questions are not too dumb or I’m not missing NiFis core too much with my current
knowledge.
I would be happy for some answers or some ideas about how to approach the questions stated
above by some experienced users.

Best
Julian

[1] http://plc4x.incubator.apache.org/
[2] https://camel.apache.org/
[3] https://github.com/apache/incubator-plc4x/tree/master/integrations/apache-nifi
Mime
View raw message