nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brett Ryan <brett.r...@gmail.com>
Subject Re: Enrichment plugin for adding attributes from SQL
Date Tue, 09 Jan 2018 13:48:15 GMT
Have now implemented SQLLookupService which is a LookupService<Record> implementation.

This works as advertised to provide a service based method of assigning attributes to a flow
file based on a Lookup Service.
https://github.com/brettryan/nifi-drunken-bundle/blob/dev/nifi-drunken-services/src/main/java/com/drunkendev/nifi/services/SQLLookupService.java
<https://github.com/brettryan/nifi-drunken-bundle/blob/dev/nifi-drunken-services/src/main/java/com/drunkendev/nifi/services/SQLLookupService.java>

The matching LookupAttributeFromRecord
https://github.com/brettryan/nifi-drunken-bundle/blob/dev/nifi-drunken-processors/src/main/java/com/drunkendev/nifi/processors/LookupAttributeFromRecord.java
<https://github.com/brettryan/nifi-drunken-bundle/blob/dev/nifi-drunken-processors/src/main/java/com/drunkendev/nifi/processors/LookupAttributeFromRecord.java>

If y'all think this could be worth contributing I'll invest further time in neatening these
up and provide full test cases, atm these are just a PoC that's been proven.



> On 9 Jan 2018, at 18:50, Brett Ryan <brett.ryan@gmail.com> wrote:
> 
> 
>> On 7 Jan 2018, at 22:23, Brett Ryan <brett.ryan@gmail.com <mailto:brett.ryan@gmail.com>>
wrote:
>> 
>> (1) Write a processor that updates an attribute on a FF from a Record based LookupService.
Say I call it UpdateAttributeFromRecordLookup
> 
> Ok, I've implemented this now as a PoC and have tested it with the MongoDBLookupService
which is working great.
> 
> The processor takes a record based lookup service and any number of attributes. The processor
has a required attribute that specifies a key prefix (default = "key."), this prefix identifies
further attributes as a key lookup, any other attributes are considered names to include as
attribute/field pairs for the resulting FF, if none of these non-key custom attributes exist
then all fields from the Record are added.
> 
> if we take the (LookupRecord with MongoDB) as an example
> 
> Key Prefix = ".key"
> key.id_store = ${id_store}
> 
> We would end up with the following attributes added to the resulting FF
> 
> address = 177 Boulevard Haussmann, 75008 Paris
> address_city = Paris
> capacity = 464600
> id_store = 1
> manager = Jean Ricca
> Given all this, do you think I'm now on the right track for a more general purpose processor
that could be used for multiple lookups?
> 
> I could then implement the SQL Lookup Service to feed this.
> 
>   [1]: https://community.hortonworks.com/articles/146198/data-flow-enrichment-with-nifi-part-3-lookuprecord.html
<https://community.hortonworks.com/articles/146198/data-flow-enrichment-with-nifi-part-3-lookuprecord.html>
>> (2) Write a service SQLLookupService which given a query as a lookup coordinate would
return a Record for the first found entry.
>> (3) Write a processor that updates an attribute from a record lookup, call it UpdateAttributeFromRecordLookup.
>> 
>> It's a bit more long winded to achieve what I originally had with a single processor,
but I guess UpdateAttributeFromRecordLookup would have the benefit of updating attributes
from LookupService implementations that do not meet the LookupAttribute criteria.
>> 
>> If you think this is viable, please let me know and I'll have a look at starting
it tomorrow afternoon.
> 


Mime
View raw message