nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andre <andre-li...@fucs.org>
Subject Re: Loss tolerance in NiFi
Date Fri, 03 Mar 2017 22:44:23 GMT
Andy, Bryan ,

Thanks, ok, so I wasn't off mark (I call it flow expiration in my original
email).

Now another question:

I was aware of expiration due to time, but not aware of discarding based on
queue levels.

Is there a way to do so and if not, should we allow users to discard
flowfiles that are stuck in a queue when it reaches a certain level?

Kind regards


On 4 Mar 2017 05:29, "Andy LoPresto" <alopresto@apache.org> wrote:

I agree with Bryan’s interpretation here. My understanding of that feature
reference is in relation to high volume streams of data coming in and the
DFM making proactive and/or realtime decisions about how to prioritize data
when they don’t have the capability to capture/operate on all of it.
Flowfile expiration and routing to failure (the provenance event is called
“drop” for a reason) are explicit design decisions that address the
scenario in legacy systems when "bad/less valuable/not currently able to be
operated on" data could prevent the "better/more valuable" data from
flowing.

Andy LoPresto
alopresto@apache.org
*alopresto.apache@gmail.com <alopresto.apache@gmail.com>*
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Mar 3, 2017, at 6:15 AM, Bryan Bende <bbende@gmail.com> wrote:

Andre,

My understanding of that statement is that NiFi lets the user decide
how to build their dataflow, and allows them to build it in a way that
favors guaranteed delivery, or favors loss tolerance, depending on
their needs.

I would think loss tolerance is referencing the idea of expiring data
from queues after a certain threshold, and maybe terminating failure
relationships.

The overall point being that the user makes the decision and not the
framework, which is the same motto for a lot of other parts of NiFi.

-Bryan


On Thu, Mar 2, 2017 at 11:54 PM, Andre <andre-lists@fucs.org> wrote:

All,

A good number of NiFi related material makes the following statement about
NiFi:

Loss tolerant vs guaranteed delivery

I have not historically used NiFi in a loss tolerant environment but after
investigating the documentation it isn't clear to me how NiFi implements
loss tolerance?

Are we referring to Clustering? Flow expiration? To the ability to discard
flowfiles that get piped to the REL_FAILURE relationships? etc?

Would anyone mind shedding some light about what are we referring to when
presenting the "Loss tolerant vs guaranteed delivery"

I thank you in advance

Mime
View raw message