nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Provenzano Nicolas <>
Subject RE: Nifi vs Sqoop
Date Thu, 10 Nov 2016 16:08:38 GMT
Thanks Bryan.

De : Bryan Bende []
Envoyé : jeudi 10 novembre 2016 15:26
À :
Objet : Re: Nifi vs Sqoop


I can't speak to a direct comparison between NiFi and sqoop, but I can say that sqoop is a
specific tool that was built just for database extraction, so it can probably do some things
NiFi can't, since NiFi is a general purpose data flow tool.

That being said, NiFi does have the ability to extraction from relation databases...

The GenerateTableFetch processor [1] would likely be what you want for more of a bulk-extraction,
and QueryDatabaseTable [2] for incremental fetching

I believe the "Maximum Value Columns" property on QueryDatabaseTable is how you achieve finding
new rows since last execution.




On Wed, Nov 9, 2016 at 4:37 AM, Provenzano Nicolas <<>>
Hi all,

I have the following requirements :

•         I need to load at day 1 a full SQL table,

•         And then need to incrementally load new data (using capture data change mechanism).

Initially, I was thinking using Sqoop to do it.

Looking at Nifi and especially the QueryDatabaseTable processor, I’m wondering if I could
use Nifi instead.

Has someone already compared both to do it and what were the outcomes ?

I can’t see however how to configure the QueryDatabaseTable to handle the new lines (for
example, looking at a “lastmodificationdate” field and taking only the lines for which
lastModificationDate > lastRequestDate) ?

Thanks in advance



View raw message