nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brett Ryan <brett.r...@gmail.com>
Subject Re: Enrichment plugin for adding attributes from SQL
Date Sat, 06 Jan 2018 03:48:54 GMT
Looking at using a lookupservice, this doesn't seem to support sending multiple keys to the
LookupService at the same time.

What I was thinking of doing was implement a LookupService that took an attribute "sql.query"
which would use this to evaluate the query but then pass in a map of key/value pairs for attribute-name/column-name
to set the attributes.

I could implement this as I imagined it to work, however it will evaluate the SQL expression
multiple times for the same query on the same flow.

I am also wondering why LookupService#getRequiredKeys must return a Set<String>, yet;
this set must only contain one value.


> On 5 Jan 2018, at 08:32, Andy LoPresto <alopresto@apache.org> wrote:
> 
> UpdateAttribute doesn’t pull from a database, it uses static or dynamic attribute values
and supports NiFi Expression Language. In your original message, you didn’t mention any
database interaction, so I thought you were just trying to accomplish "I wanted to add some
attributes to a FlowFile while not altering the contents of that FlowFile”, which is indeed
what UpdateAttribute does.
> 
> If you need to retrieve those values from a database, as Joey mentions, the LookupService
is the right tool.
> 
> With your prior setup, the distributed map cache is as secure as the NiFi configuration
— if using secured NiFi, the communication between that node and any other is over TLS,
and within the node it’s a memory access.
> 
> A big part of the NiFi philosophy is the same as the Unix philosophy — each tool should
do one job very well, and to perform complicated tasks, chain those tools together. This helps
with provenance reporting, usage reporting, debugging, flow development lifecycle, maintenance,
etc. A processor which retrieves attributes from a database and updates the incoming flowfile
with them is certainly useful in the use case you describe, but is not a generic pattern.
There’s no intent to discourage custom development, and whatever makes your flow work is
fine. Just explaining why you likely won’t see a solution like that in the NiFi bundled
components.
> 
> 
> Andy LoPresto
> alopresto@apache.org <mailto:alopresto@apache.org>
> alopresto.apache@gmail.com <mailto:alopresto.apache@gmail.com>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 
>> On Jan 4, 2018, at 4:07 PM, Brett Ryan <brett.ryan@gmail.com <mailto:brett.ryan@gmail.com>>
wrote:
>> 
>> Thanks Andy, how would update attribute be able to get the value from sql?
>> 
>> Consider a flow where a piece of information needs to be obtained from a DB but i
do not want the contents of the current FF to be altered, using ExecuteSQL anywhere prior
would not be possible due to replacing the FF contents.
>> 
>> What I had was two seperate flows, one that updates an oauth key in a db keeping
it fresh, the main flow would then read the db just before an invokehttp.
>> 
>> I was originally using a distributed map cache but had concerns that it might not
be secure and was also advised the cache server has been known to go down.
>> 
>>> On 5 Jan 2018, at 04:01, Joey Frazee <joey.frazee@icloud.com <mailto:joey.frazee@icloud.com>>
wrote:
>>> 
>>> Andy, Brett,
>>> 
>>> Taking a quick glance at the code it looks like it's enriching attributes from
a database according to a query.
>>> 
>>> If that's correct, there's a LookupAttribute processor that delegates lookups
to a "LookupService" and adds attributes without altering content. There are a variety of
these LookupServices included. I think what you implemented would make sense as a LookupService
and then you could just configure the processor to use that. It could also be used with LookupRecord
then too so there'd be a double payoff.
>>> 
>>> -joey
>>> 
>>>> On Jan 4, 2018, 8:44 AM -0800, Andy LoPresto <alopresto@apache.org <mailto:alopresto@apache.org>>,
wrote:
>>>> Hi Brett,
>>>> 
>>>> It’s great that you found it easy to write a new processor for Apache NiFi.
It is probably an indicator that we need to improve education/evangelism/documentation, however,
that you did not find UpdateAttribute [1], which should do exactly what you were looking for.
>>>> 
>>>> [1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-update-attribute-nar/1.4.0/org.apache.nifi.processors.attributes.UpdateAttribute/index.html
<https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-update-attribute-nar/1.4.0/org.apache.nifi.processors.attributes.UpdateAttribute/index.html>
>>>> 
>>>> Andy LoPresto
>>>> alopresto@apache.org <mailto:alopresto@apache.org>
>>>> alopresto.apache@gmail.com
>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>> 
>>>>> On Jan 4, 2018, at 7:03 AM, Brett Ryan <brett.ryan@gmail.com> wrote:
>>>>> 
>>>>> Hi all, having used NiFi for a couple days I wanted to add some attributes
to a FlowFile while not altering the contents of that FlowFile.
>>>>> 
>>>>> I had suggestions to use a script processor but that just sounded like
a hack which could become a nuisance to replicate.
>>>>> 
>>>>> Anyway, I figured I'd write a processor to do this, anyone interested
you can find it here.
>>>>> 
>>>>> https://github.com/brettryan/nifi-drunken-bundle
>>>>> 
>>>>> Feel free to do with it what you please.
>>>>> 
>>>>> I've published to maven central but it will take a day to appear in the
search.
>>>>> 
>>>>> <dependency>
>>>>>  <groupId>com.drunkendev</groupId>
>>>>>  <artifactId>nifi-drunken-nar</artifactId>
>>>>>  <version>1.0.0</version>
>>>>>  <type>nar</type>
>>>>> </dependency>
>>>>> <dependency>
>>>>>  <groupId>com.drunkendev</groupId>
>>>>>  <artifactId>nifi-drunken-processors</artifactId>
>>>>>  <version>1.0.0</version>
>>>>> </dependency>
>>>>> <dependency>
>>>>>  <groupId>com.drunkendev</groupId>
>>>>>  <artifactId>nifi-drunken-bundle</artifactId>
>>>>>  <version>1.0.0</version>
>>>>>  <type>pom</type>
>>>>> </dependency>
>>>>> 
>>>> 
> 


Mime
View raw message