nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boris Tyukin <bo...@boristyukin.com>
Subject stop processing related flowfiles
Date Wed, 28 Nov 2018 13:50:04 GMT
Hi guys,

I am trying to come up with a good design for the following challenge:

1. ConsumeKafka processor consumes messages from 200 topics.

2. The next processor is a custom groovy processor that does some data
transformation and also puts transformed data into target system.* It is
crucial to process messages in topics in order. *

3. This is the tricky part - if any error is raised during step 2, I want
to stop processing any new flowfiles for *that topic *while continue
processing messages for other topics.

In other words, I want to enforce order and process only one message per
topic at the time and stop processing messages for a given topic, if
currently processed message failed. It is kind like FIFO queue, that stops
pushing items out of the queue if current item errors out.

Is it possible to do it? I am open to use an external queue or cache like
Redis.

I've played a bit with EnforceOrder and Notify/Wait processors but still
cannot wrap my head about it.

Appreciate your help,
Boris

Mime
View raw message