nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brett Ryan <brett.r...@gmail.com>
Subject Re: Enrichment plugin for adding attributes from SQL
Date Tue, 09 Jan 2018 07:50:57 GMT

> On 7 Jan 2018, at 22:23, Brett Ryan <brett.ryan@gmail.com> wrote:
> 
> (1) Write a processor that updates an attribute on a FF from a Record based LookupService.
Say I call it UpdateAttributeFromRecordLookup

Ok, I've implemented this now as a PoC and have tested it with the MongoDBLookupService which
is working great.

The processor takes a record based lookup service and any number of attributes. The processor
has a required attribute that specifies a key prefix (default = "key."), this prefix identifies
further attributes as a key lookup, any other attributes are considered names to include as
attribute/field pairs for the resulting FF, if none of these non-key custom attributes exist
then all fields from the Record are added.

if we take the (LookupRecord with MongoDB) as an example

Key Prefix = ".key"
key.id_store = ${id_store}

We would end up with the following attributes added to the resulting FF

address = 177 Boulevard Haussmann, 75008 Paris
address_city = Paris
capacity = 464600
id_store = 1
manager = Jean Ricca
Given all this, do you think I'm now on the right track for a more general purpose processor
that could be used for multiple lookups?

I could then implement the SQL Lookup Service to feed this.

  [1]: https://community.hortonworks.com/articles/146198/data-flow-enrichment-with-nifi-part-3-lookuprecord.html

> (2) Write a service SQLLookupService which given a query as a lookup coordinate would
return a Record for the first found entry.
> (3) Write a processor that updates an attribute from a record lookup, call it UpdateAttributeFromRecordLookup.
> 
> It's a bit more long winded to achieve what I originally had with a single processor,
but I guess UpdateAttributeFromRecordLookup would have the benefit of updating attributes
from LookupService implementations that do not meet the LookupAttribute criteria.
> 
> If you think this is viable, please let me know and I'll have a look at starting it tomorrow
afternoon.


Mime
View raw message