nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Provenzano Nicolas <nicolas.provenz...@gfi.fr>
Subject RE: Nifi vs Sqoop
Date Thu, 10 Nov 2016 16:08:38 GMT
Thanks Bryan.

De : Bryan Bende [mailto:bbende@gmail.com]
Envoyé : jeudi 10 novembre 2016 15:26
À : users@nifi.apache.org
Objet : Re: Nifi vs Sqoop

Hello,

I can't speak to a direct comparison between NiFi and sqoop, but I can say that sqoop is a
specific tool that was built just for database extraction, so it can probably do some things
NiFi can't, since NiFi is a general purpose data flow tool.

That being said, NiFi does have the ability to extraction from relation databases...

The GenerateTableFetch processor [1] would likely be what you want for more of a bulk-extraction,
and QueryDatabaseTable [2] for incremental fetching

I believe the "Maximum Value Columns" property on QueryDatabaseTable is how you achieve finding
new rows since last execution.

Thanks,

Bryan

[1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.GenerateTableFetch/index.html
[2] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.QueryDatabaseTable/index.html


On Wed, Nov 9, 2016 at 4:37 AM, Provenzano Nicolas <nicolas.provenzano@gfi.fr<mailto:nicolas.provenzano@gfi.fr>>
wrote:
Hi all,

I have the following requirements :


•         I need to load at day 1 a full SQL table,

•         And then need to incrementally load new data (using capture data change mechanism).

Initially, I was thinking using Sqoop to do it.

Looking at Nifi and especially the QueryDatabaseTable processor, I’m wondering if I could
use Nifi instead.

Has someone already compared both to do it and what were the outcomes ?

I can’t see however how to configure the QueryDatabaseTable to handle the new lines (for
example, looking at a “lastmodificationdate” field and taking only the lines for which
lastModificationDate > lastRequestDate) ?

Thanks in advance

BR

Nicolas

Mime
View raw message