nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brett Ryan <>
Subject Re: Enrichment plugin for adding attributes from SQL
Date Tue, 09 Jan 2018 13:48:15 GMT
Have now implemented SQLLookupService which is a LookupService<Record> implementation.

This works as advertised to provide a service based method of assigning attributes to a flow
file based on a Lookup Service.

The matching LookupAttributeFromRecord

If y'all think this could be worth contributing I'll invest further time in neatening these
up and provide full test cases, atm these are just a PoC that's been proven.

> On 9 Jan 2018, at 18:50, Brett Ryan <> wrote:
>> On 7 Jan 2018, at 22:23, Brett Ryan < <>>
>> (1) Write a processor that updates an attribute on a FF from a Record based LookupService.
Say I call it UpdateAttributeFromRecordLookup
> Ok, I've implemented this now as a PoC and have tested it with the MongoDBLookupService
which is working great.
> The processor takes a record based lookup service and any number of attributes. The processor
has a required attribute that specifies a key prefix (default = "key."), this prefix identifies
further attributes as a key lookup, any other attributes are considered names to include as
attribute/field pairs for the resulting FF, if none of these non-key custom attributes exist
then all fields from the Record are added.
> if we take the (LookupRecord with MongoDB) as an example
> Key Prefix = ".key"
> key.id_store = ${id_store}
> We would end up with the following attributes added to the resulting FF
> address = 177 Boulevard Haussmann, 75008 Paris
> address_city = Paris
> capacity = 464600
> id_store = 1
> manager = Jean Ricca
> Given all this, do you think I'm now on the right track for a more general purpose processor
that could be used for multiple lookups?
> I could then implement the SQL Lookup Service to feed this.
>   [1]:
>> (2) Write a service SQLLookupService which given a query as a lookup coordinate would
return a Record for the first found entry.
>> (3) Write a processor that updates an attribute from a record lookup, call it UpdateAttributeFromRecordLookup.
>> It's a bit more long winded to achieve what I originally had with a single processor,
but I guess UpdateAttributeFromRecordLookup would have the benefit of updating attributes
from LookupService implementations that do not meet the LookupAttribute criteria.
>> If you think this is viable, please let me know and I'll have a look at starting
it tomorrow afternoon.

View raw message