nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Gallagher <dgallag...@CleverDevices.com>
Subject RE: Open / Close Gate examples?
Date Fri, 17 May 2019 18:59:12 GMT
Hi Joe – thanks for answering. I can actually live with the message ack at the moment. But,
I can’t have two messages at once because the underlying process will deadlock if two flows
try to use it at the same time. This is a problem because my nifi node might be brought up
after having been shut off for a few days, at which point it will read multiple messages off
the queue and attempt to process them. The problem I’m having is getting flow files to wait
for each other. If I use a Notify processor to set a release signal as indicated in the linked
gist, it does release the Wait as designed. But, the next flowfile then enters the queue and
it will get released when the Notify is triggered again by the GenerateFlowFile. What I really
want to have happen is for each flowfile to block the next flowfile from entering until it
has exited the process. I’m not really sure how to set that up successfully…

Thanks,

Dave

From: Joe Witt <joe.witt@gmail.com>
Sent: Friday, May 17, 2019 12:33 PM
To: users@nifi.apache.org
Subject: Re: Open / Close Gate examples?

Dave

Using Wait/Notify would ensure you only have one message in flight at a time (or it can/should).
 But the message will be ack'd before processed.

For Kafka and some of these message queue mechanisms if we want to offer a 'do not ack until
the whole flow is done' behavior we should update the processors to allow for that and actually
this is a great use case for the nifi-fn/stateless work that SamH in the community has been
talking about.

Thanks

On Fri, May 17, 2019 at 12:30 PM David Gallagher <dgallagher@cleverdevices.com<mailto:dgallagher@cleverdevices.com>>
wrote:
Hi – I have a requirement to read one message off RabbitMQ and fully process that message
before I take another message off the queue. I found some great guidance to setting up a ‘traffic
light’ pattern here (https://gist.github.com/ijokarumawak/9e1a4855934f2bb9661f88ca625bd244<https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgist.github.com%2Fijokarumawak%2F9e1a4855934f2bb9661f88ca625bd244&data=02%7C01%7Cdgallagher%40cleverdevices.com%7Ca958a30dc3b241d7cac608d6dae57529%7Cc24ec497266d4560bcda1d93096d9d18%7C0%7C0%7C636937076386146849&sdata=uSQN5yGmW0b8h%2FxYvYtEIJryz2JkHLjla3kS8GrRtr4%3D&reserved=0>),
but it won’t work for my process because the Notify will get triggered by the GenerateFlowFile
before the long-running process triggers, which will put two or more messages into the process
at once. The documentation for Notify makes reference to using a 0 value in Signal Counter
Delta to simulate ‘open-close-gate’ flow control, which sounds like what I’m looking
for. Can someone point me in the right direction?

Thanks,

Dave


Mime
View raw message