nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koji Kawamura <ijokaruma...@gmail.com>
Subject Re: Incremental Fetch for RestAPI
Date Tue, 13 Nov 2018 01:06:16 GMT
Hi Manee,

It depends on the Client API on how to tell what the next response
data set should be.
That may be an additional query parameter such as last fetch
timestamp, or something like HTTP etag header in most APIs.
You can pass FlowFiles to InvokeHTTP to tell such parameters.

Also, I recommend using Record processors over splitting dataset to process.
Because Record processors work more efficiently and the data set unit
will be more meaningful in your case.

I think your flow would be something like:

InvokeHTTP (Assuming the API result contains value to make next
incremental request)
 -> Do some JSON transformation (QueryRecord, UpdateRecord or
JoltTransformRecord)
 -> PutDatabaseRecord
 -> Then connect success back to InvokeHTTP to fetch next dataset

Hope this helps.

Thanks,
Koji

On Mon, Nov 12, 2018 at 9:06 PM Manee <mani.nathan08@gmail.com> wrote:
>
> Hi Team,
>
> I am new to NiFi .I have a task like .We need to fetch a data from  Client
> API and stored into Postgresql
> My Flow ;
>
> InvokeHTTP -->>--SplitJson-->>--EvaluateJsonPath
> -->>ConvertJSONTosql--->>PutSql
>
> This is my flow is working fine but i need to make this as a incremental
> fetch from API ..whenever client API made changes it should reflect in our
> databases .How i can process for incremental fetch in API call ?.Please
> guide me to fix this problem .
>
>
> Thanks in Advance,
> Manikandan K
>
>
>
>
>
>
> -----
> Thanks,
> Manee
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Mime
View raw message