nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan H <ryan.howell.developm...@gmail.com>
Subject NiFi Stream Database Records
Date Fri, 25 Jan 2019 16:27:18 GMT
Hi All,

A colleague asked me the following question and would like to throw it out
here as I am not sure about this (we are currently on NiFi 1.5.0, but
moving to 1.8.0 as we speak):

Is there a Nifi process to stream 20 million records from a database into
Nifi ? Currently we are using GenerateTableFetch with ExecuteSQL and get
batches of these 20 million records but ExecuteSQL processor is slow and so
is this entire process. We are looking for a solution that streams records
instead of getting all 20 of them in one go (which results in gigantic
flowfiles which result in OutOfMemory) or even in these batches which make
it manageable but still very slow.

Thanks in advance!


Cheers,

Ryan H

Mime
View raw message