nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <>
Subject Re: NiFi for backup solution
Date Thu, 13 Oct 2016 15:14:42 GMT

NiFi can certainly be used for some data replication scenarios and
quite often is.  If you can treat the source like a continuous data
source then there is some way to keep state about what has been pulled
already, what has changed or needs yet to be pulled, and it can just
keep running then generally speaking it will work out well.  Depending
on how the flow is setup, error conditions that can occur in remote
delivery, and cluster topology NiFi won't be able to ensure the order
that data is received is the order in which data is delivered.  So, if
you need to ensure data is copied in precisely the same order (like
log replication) and each object/message/event is on the order of KBs
in size then I'd recommend looking at Apache Kafka and Kafka Connect's
support for keeping things ordered within the same partition of the
same topic.


On Thu, Oct 13, 2016 at 11:05 AM, Gop Krr <> wrote:
> Hi All,
> I am learning NiFi as well as trying to deploy it in production for few  use
> cases. One of the use case is ETL and another use case is, using NiFi as a
> backup solution, where it takes the data from one source and moves to
> another database|file. Is anyone using NiFi for this purpose? Does NiFi
> support incremental data move?
> It would be awesome if someone can point me to right documentation.
> Thanks
> Rai

View raw message