nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Bende <bbe...@gmail.com>
Subject Re: Nifi vs Sqoop
Date Thu, 10 Nov 2016 14:25:52 GMT
Hello,

I can't speak to a direct comparison between NiFi and sqoop, but I can say
that sqoop is a specific tool that was built just for database extraction,
so it can probably do some things NiFi can't, since NiFi is a general
purpose data flow tool.

That being said, NiFi does have the ability to extraction from relation
databases...

The GenerateTableFetch processor [1] would likely be what you want for more
of a bulk-extraction, and QueryDatabaseTable [2] for incremental fetching

I believe the "Maximum Value Columns" property on QueryDatabaseTable is how
you achieve finding new rows since last execution.

Thanks,

Bryan

[1]
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.GenerateTableFetch/index.html
[2]
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.QueryDatabaseTable/index.html


On Wed, Nov 9, 2016 at 4:37 AM, Provenzano Nicolas <
nicolas.provenzano@gfi.fr> wrote:

> Hi all,
>
>
>
> I have the following requirements :
>
>
>
> ·         I need to load at day 1 a full SQL table,
>
> ·         And then need to incrementally load new data (using capture
> data change mechanism).
>
>
>
> Initially, I was thinking using Sqoop to do it.
>
>
>
> Looking at Nifi and especially the QueryDatabaseTable processor, I’m
> wondering if I could use Nifi instead.
>
>
>
> Has someone already compared both to do it and what were the outcomes ?
>
>
>
> I can’t see however how to configure the QueryDatabaseTable to handle the
> new lines (for example, looking at a “lastmodificationdate” field and
> taking only the lines for which lastModificationDate > lastRequestDate) ?
>
>
>
> Thanks in advance
>
>
>
> BR
>
>
>
> Nicolas
>

Mime
View raw message