nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pierre Villard <pierre.villard...@gmail.com>
Subject Improve S2S to ease "star deployments"
Date Sat, 15 Sep 2018 13:05:27 GMT
Hi all,

Here is my use case: I've multiple NiFi standalone instances deployed over
multiple sites (that could be MiNiFi instances) and a central NiFi
cluster.  The standalone instances generate data, the data is sent to the
central cluster to be parsed and enriched before being sent back to the
standalone instances. The data needs to go back where it's been generated.

At the moment, since RPG cannot be configured using EL and FFs attributes,
you need to have one port (or one RPG if the RPG is on central NiFi's side)
per standalone instance. And I don't think that changing the RPG to handle
FFs attributes scope would be a good idea in terms of implementation.

Instead I'd like to change the S2S protocol to allow RPG pulling based on
FFs attributes.

On the standalone instances, we would have:
Workflow generating data => RPG => workflow receiving enriched data from
central cluster

On the NiFi cluster, we would have:
input port => workflow parsing and enriching data => output port

The idea would be that, when configuring an output port in the RPG, it'd be
possible to enable "host based pulling" so that only flow files having the
attribute 's2s.host' matching the host of the instance hosting the RPG
would be pulled. (the s2s.port attribute is already set when data is sent
through S2S).

I already started working on that approach and even though I don't have
something fully functional yet, I wanted to discuss it here to be sure this
would be interesting for the wider community and, also, if I'm not missing
something obvious that would prevent it.

Happy to file a JIRA if that sounds interesting.

Thanks,
Pierre

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message