nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raf Huys <raf.h...@gmail.com>
Subject stream one large file, only once
Date Mon, 14 Nov 2016 12:23:03 GMT
I would like to read in a large (several gigs) of logdata, and route every
line to a (potentially different) Kafka topic.

- I don't want this file to be in memory
- I want it to be read once, not more

using `GetFile` takes the whole file in memory. Same with `FetchFile` as
far as I can see.

I also used a `ExecuteProcess` processor in which the file is `cat` and
which splits off a flowfile every millisecond. This looked to be a somewhat
streaming approach to the problem, but this processor runs continuously (or
cron based) and by consequence the logfile is re-injected all the time.

What's the typical Nifi for this? Tx

Raf Huys

Mime
View raw message