nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joey Frazee <>
Subject Re: Nifi and Kafka Inserts and Updates
Date Sun, 15 Oct 2017 17:50:45 GMT
Uwe, on the issue on a LookupService for a database, you could use the ScriptedLookupService
[1] and the usual DBCP to do this or maybe check out the following, somewhat limited, DatabaseLookupService
[2]. It builds on Apache Commons Config so it forces you to treat the table a bit like a key-value
store and can’t do multi-column where’s, but it would probably be sufficient to identify
an existing row if there’s a natural key in the data somewhere.


On Oct 13, 2017, 2:20 PM -0500, Uwe Geercken <>, wrote:
> Hello,
> I am looking for some advice: I have Nifi sending flowfiles to Kafka. As we know for
Kafka everything is an "insert". Messages are inserted into the Kafka log.
> Now I wonder what is the best way to insert OR update a relational database table from
Kafka messages using Nifi. What is the best way to determine if the record (data) I get from
Kafka needs to be updated or inserted in the relational db table? Ok. MySQL e.g. has an "upsert"
mode, but I don't see we have a processor for that. So I would need to generate the SQL statement
and execute that. But is there a more elegant (straight ahead) way?
> I could use MongoDB as well where we have a processor that allows for an upsert, but
I wanted to know how to solve this with a relational db.
> My idea was to use a db lookup service controller - to lookup if a certain key exists
- but there is none from what I have seen. I don't think it is an unusual use case, so I hope
that somebody can help or share thought with me.
> Thanks,
> Uwe

View raw message