For each of the input files could it be that you would pull data from
Have you had a chance to learn about the NiFi Expression language?
That will come in quite handy for constructing the URL used in
The general pattern I think makes sense here is:
- Gather Data
- Extract Features from data to construct URL
- Fetch document/response from URL
During 'Gather Data' you acquire the files.
During 'Extract features' you pull out elements of the content of the
file into flow file attributes. You can use RouteOnAttribute to send
to an UpdateAttribute processor which constructs a new attribute of
URL pattern A or URL pattern B respectively. You can also collapse
that into a single UpdateAttribute possibly using the advanced UI and
set specific URLs based on patterns of attributes. Lots of ways to
During Fetch document you should be able to just have a single
InvokeHTTP potentially which looks at some attribute you've defined
say 'the-url' and specify in InvokeHTTP the remote URL value to be
We should publish a template for this pattern/approach if we've not
already but let's see how you progress and decide what would be most
useful for others.
On Tue, Feb 16, 2016 at 9:36 PM, Jeff - Data Bean Australia
> I got a use case like this:
> There are two files, say fileA and fileB, both of them contains multiple
> lines of items and used for generate URLs. However, the algorithm for
> generating URLs are different. If items come from fileA, the URL template
> looks like this:
> If items come from fileB, the template looks like this:
> I am going to create a NiFi template to for the Data Flow from reading the
> list file up to downloading data using InvokeHTTP, and place a
> UpdateAttribute processor in front of the template to feed in different file
> names (I have only two files).
> The problem I have so far is how to generate the URLs based on different
> input, so that I can make a general NiFi template for reusability.
> Data Bean - A Big Data Solution Provider in Australia.