nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: Thread Control of Processors
Date Fri, 12 Feb 2016 05:34:18 GMT
Jeff,

This is definitely a strong use case for nifi.

It might be that InvokeHTTP is the better choice here.

If what you'd like to do is effectively throttle the rate at which you
hit the web service with the InvokeHttp calls you can schedule that
processor to run as often as you like (for example every 100 ms).
Then use backpressure settings on the queues feeding that InvokeHTTP
process.  Effectively you can control where data will back up in the
flow while it is being throttled.

If the lookup data is a good candidate for caching then there may be
other great options to make this more efficient.

Perhaps you can share a flow template of what you have so far and we
can make recommendations on next steps?

Thanks
Joe

On Fri, Feb 12, 2016 at 12:28 AM, Jeff - Data Bean Australia
<databean.au@gmail.com> wrote:
> Hi
>
> I got a use case like this:
>
> There is a file that contains thousands of items, each on one line. For each
> item, it will trigger one GetHTTP processor to fetch some data.
>
> Here is what I am trying to do:
>
> 1. Fetch this file
> 2. For each line, I generate one file using SplitText
> 3. Drive GetHTTP downstream.
>
> However, given there are more than 2000 lines, more than 2000 HTTP Get
> processes will be created and flooded into one web site, which doesn't sound
> like a good idea. So I would like to control the processors, so that only a
> couple of them will be running at the same time, and maybe delay for a
> couple of seconds after finish.
>
> How can I do that in NiFi?
>
> Thanks,
> Jeff
>
>
>
>
> --
> Data Bean - A Big Data Solution Provider in Australia.

Mime
View raw message