nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Weeks <swe...@weeksconsulting.us>
Subject Re: Open / Close Gate examples?
Date Sat, 18 May 2019 12:45:39 GMT
See this example as I had a lot of questions about wait and notify earlier and this helped
a lot.


https://gist.github.com/ijokarumawak/9e1a4855934f2bb9661f88ca625bd244

Thanks
Shawn
Sent from my iPhone

On May 17, 2019, at 1:59 PM, David Gallagher <dgallagher@cleverdevices.com<mailto:dgallagher@cleverdevices.com>>
wrote:

Hi Joe – thanks for answering. I can actually live with the message ack at the moment. But,
I can’t have two messages at once because the underlying process will deadlock if two flows
try to use it at the same time. This is a problem because my nifi node might be brought up
after having been shut off for a few days, at which point it will read multiple messages off
the queue and attempt to process them. The problem I’m having is getting flow files to wait
for each other. If I use a Notify processor to set a release signal as indicated in the linked
gist, it does release the Wait as designed. But, the next flowfile then enters the queue and
it will get released when the Notify is triggered again by the GenerateFlowFile. What I really
want to have happen is for each flowfile to block the next flowfile from entering until it
has exited the process. I’m not really sure how to set that up successfully…

Thanks,

Dave

From: Joe Witt <joe.witt@gmail.com<mailto:joe.witt@gmail.com>>
Sent: Friday, May 17, 2019 12:33 PM
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: Re: Open / Close Gate examples?

Dave

Using Wait/Notify would ensure you only have one message in flight at a time (or it can/should).
 But the message will be ack'd before processed.

For Kafka and some of these message queue mechanisms if we want to offer a 'do not ack until
the whole flow is done' behavior we should update the processors to allow for that and actually
this is a great use case for the nifi-fn/stateless work that SamH in the community has been
talking about.

Thanks

On Fri, May 17, 2019 at 12:30 PM David Gallagher <dgallagher@cleverdevices.com<mailto:dgallagher@cleverdevices.com>>
wrote:
Hi – I have a requirement to read one message off RabbitMQ and fully process that message
before I take another message off the queue. I found some great guidance to setting up a ‘traffic
light’ pattern here (https://gist.github.com/ijokarumawak/9e1a4855934f2bb9661f88ca625bd244<https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgist.github.com%2Fijokarumawak%2F9e1a4855934f2bb9661f88ca625bd244&data=02%7C01%7Cdgallagher%40cleverdevices.com%7Ca958a30dc3b241d7cac608d6dae57529%7Cc24ec497266d4560bcda1d93096d9d18%7C0%7C0%7C636937076386146849&sdata=uSQN5yGmW0b8h%2FxYvYtEIJryz2JkHLjla3kS8GrRtr4%3D&reserved=0>),
but it won’t work for my process because the Notify will get triggered by the GenerateFlowFile
before the long-running process triggers, which will put two or more messages into the process
at once. The documentation for Notify makes reference to using a 0 value in Signal Counter
Delta to simulate ‘open-close-gate’ flow control, which sounds like what I’m looking
for. Can someone point me in the right direction?

Thanks,

Dave


Mime
View raw message