nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Thomsen <mikerthom...@gmail.com>
Subject Re: Enrichment plugin for adding attributes from SQL
Date Sat, 06 Jan 2018 16:23:22 GMT
Take a look at the mongo lookup service. I think it could serve as a good
example here.
On Fri, Jan 5, 2018 at 10:49 PM Brett Ryan <brett.ryan@gmail.com> wrote:

> Looking at using a lookupservice, this doesn't seem to support sending
> multiple keys to the LookupService at the same time.
>
> What I was thinking of doing was implement a LookupService that took an
> attribute "sql.query" which would use this to evaluate the query but then
> pass in a map of key/value pairs for attribute-name/column-name to set the
> attributes.
>
> I could implement this as I imagined it to work, however it will evaluate
> the SQL expression multiple times for the same query on the same flow.
>
> I am also wondering why LookupService#getRequiredKeys must return a
> Set<String>, yet; this set must only contain one value.
>
>
> On 5 Jan 2018, at 08:32, Andy LoPresto <alopresto@apache.org> wrote:
>
> UpdateAttribute doesn’t pull from a database, it uses static or dynamic
> attribute values and supports NiFi Expression Language. In your original
> message, you didn’t mention any database interaction, so I thought you were
> just trying to accomplish "I wanted to add some attributes to a FlowFile
> while not altering the contents of that FlowFile”, which is indeed what
> UpdateAttribute does.
>
> If you need to retrieve those values from a database, as Joey mentions,
> the LookupService is the right tool.
>
> With your prior setup, the distributed map cache is as secure as the NiFi
> configuration — if using secured NiFi, the communication between that node
> and any other is over TLS, and within the node it’s a memory access.
>
> A big part of the NiFi philosophy is the same as the Unix philosophy —
> each tool should do one job very well, and to perform complicated tasks,
> chain those tools together. This helps with provenance reporting, usage
> reporting, debugging, flow development lifecycle, maintenance, etc. A
> processor which retrieves attributes from a database and updates the
> incoming flowfile with them is certainly useful in the use case you
> describe, but is not a generic pattern. There’s no intent to discourage
> custom development, and whatever makes your flow work is fine. Just
> explaining why you likely won’t see a solution like that in the NiFi
> bundled components.
>
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <alopresto.apache@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Jan 4, 2018, at 4:07 PM, Brett Ryan <brett.ryan@gmail.com> wrote:
>
> Thanks Andy, how would update attribute be able to get the value from sql?
>
> Consider a flow where a piece of information needs to be obtained from a
> DB but i do not want the contents of the current FF to be altered, using
> ExecuteSQL anywhere prior would not be possible due to replacing the FF
> contents.
>
> What I had was two seperate flows, one that updates an oauth key in a db
> keeping it fresh, the main flow would then read the db just before an
> invokehttp.
>
> I was originally using a distributed map cache but had concerns that it
> might not be secure and was also advised the cache server has been known to
> go down.
>
> On 5 Jan 2018, at 04:01, Joey Frazee <joey.frazee@icloud.com> wrote:
>
> Andy, Brett,
>
> Taking a quick glance at the code it looks like it's enriching attributes
> from a database according to a query.
>
> If that's correct, there's a LookupAttribute processor that delegates
> lookups to a "LookupService" and adds attributes without altering content.
> There are a variety of these LookupServices included. I think what you
> implemented would make sense as a LookupService and then you could just
> configure the processor to use that. It could also be used with
> LookupRecord then too so there'd be a double payoff.
>
> -joey
>
> On Jan 4, 2018, 8:44 AM -0800, Andy LoPresto <alopresto@apache.org>,
> wrote:
> Hi Brett,
>
> It’s great that you found it easy to write a new processor for Apache
> NiFi. It is probably an indicator that we need to improve
> education/evangelism/documentation, however, that you did not find
> UpdateAttribute [1], which should do exactly what you were looking for.
>
> [1]
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-update-attribute-nar/1.4.0/org.apache.nifi.processors.attributes.UpdateAttribute/index.html
>
> Andy LoPresto
> alopresto@apache.org
> alopresto.apache@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Jan 4, 2018, at 7:03 AM, Brett Ryan <brett.ryan@gmail.com> wrote:
>
> Hi all, having used NiFi for a couple days I wanted to add some attributes
> to a FlowFile while not altering the contents of that FlowFile.
>
> I had suggestions to use a script processor but that just sounded like a
> hack which could become a nuisance to replicate.
>
> Anyway, I figured I'd write a processor to do this, anyone interested you
> can find it here.
>
> https://github.com/brettryan/nifi-drunken-bundle
>
> Feel free to do with it what you please.
>
> I've published to maven central but it will take a day to appear in the
> search.
>
> <dependency>
>  <groupId>com.drunkendev</groupId>
>  <artifactId>nifi-drunken-nar</artifactId>
>  <version>1.0.0</version>
>  <type>nar</type>
> </dependency>
> <dependency>
>  <groupId>com.drunkendev</groupId>
>  <artifactId>nifi-drunken-processors</artifactId>
>  <version>1.0.0</version>
> </dependency>
> <dependency>
>  <groupId>com.drunkendev</groupId>
>  <artifactId>nifi-drunken-bundle</artifactId>
>  <version>1.0.0</version>
>  <type>pom</type>
> </dependency>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message