nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Grande <agra...@hortonworks.com>
Subject Re: Nifi processor last execution time
Date Fri, 13 Nov 2015 19:56:56 GMT
There are 2 approached NiFi processors follow today.

E.g. a new TailFile saves it in a local state file (relative to NiFi root, in the ./conf/
directory).

Others leverage a DistributedCache service (it's actually more like a RemoteCache, not a distributed/replicated),
which they declare as a requirement in the properties. This also has a benefit of being able
to share the state between nodes and, optionally, persist the cache to the disk to be able
to re-hydrate it on restart.

Andrew



On 11/13/15, 10:09 AM, "Sean Busbey" <busbey@cloudera.com> wrote:

>Sam,
>
>For now I would store this information in HDFS. Alternatively, if you have
>it ZooKeeper would also work.
>
>On Thu, Nov 12, 2015 at 10:45 PM, Sam Kamau <skamau@gmail.com> wrote:
>
>> I am working on a processor that loads data to HDFS for an RDBMS using
>> ExecuteSQL processor. The data has a last update time that I would like to
>> use to load records that have changed since the last load.
>>
>> How best can I go about saving the last execution time so I can use it as a
>> filter in the query.
>>
>> Thanks
>>
>
>
>
>-- 
>Sean
Mime
View raw message