nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Bende <bbe...@gmail.com>
Subject Re: Import Kafka messages into Titan
Date Thu, 31 Mar 2016 16:15:42 GMT
For #2,  you can use templates to move the flow (or parts of it) to another
instance.
A possible approach is to organize the flow into process groups and create
a template per process group, making it potentially easier to update parts
of the flow independently.

This project might be helpful to look at in terms of automating deploying a
template from one instance to another:
https://github.com/aperepel/nifi-api-deploy

For properties that are environment specific, if the property supports
expression language, you can specify them in bootstrap.conf as -D
properties for each of your NiFi instances, and in your processors you can
reference them with Expression Language.
For example, in each bootstrap.conf there could be -Dkafka.topic=mytopic
and then in a PutKafka processor set the topic to ${kafka.topic}. This will
let your template be the same for each environment.
Unfortunately at a quick glance it looks like GetKafka topic name does not
support EL, which should probably be fixed to allow this.

In the future there is a plan to have a variable registry exposed through
the UI so that you wouldn't have to edit the bootstrap file to define these
properties.


On Thu, Mar 31, 2016 at 11:58 AM, Matt Burgess <mattyb149@gmail.com> wrote:

> I'll let someone else have a go at question 2 :)
>
> If you're using ExecuteScript with Groovy, you don't need
> EvaluateJsonPath, Groovy has a JSONSlurper that works nicely (examples on
> my blog).
>
> To put directly into Titan you don't need to convert the format, instead
> you'll want Gremlin (part of Apache Tinkerpop), point your processor's
> Module Path property at a folder containing the Gremlin JARs, then you can
> create the vertices and edges using the approach in the Titan documentation.
>
> This would make an excellent blog post, perhaps I'll give this a try
> myself but please feel welcome to share anything you learn along the way!
> If I get some spare time I'd like to write a PutGraph processor that does
> pretty much what we've outlined here.
>
> Regards,
> Matt
>
> Sent from my iPhone
>
> > On Mar 31, 2016, at 10:15 AM, idioma <corda.ilaria@gmail.com> wrote:
> >
> > Matt, thank you for this this is brilliant. So, as it is I am thinking
> that I
> > would like to use the following:
> >
> > GetKafka -> EvaluateJsonPath -> ExecuteScript+Groovy Script
> >
> > My questions are two:
> >
> > 1) How do I import the Titan-compliant file into Titan? I guess I can
> modify
> > the script and load it into it.
> > 2) my second quest is more naive and proves the fact my background is
> more
> > on Apache Camel with very little knowledge of NiFi. In a versionControl
> > environment, how do you push a process flow created with NiFi that mostly
> > involves standard components? Do you write customized version where you
> set
> > of kafka properties, for example?
> >
> > Thanks
> >
> >
> >
> > --
> > View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Import-Kafka-messages-into-Titan-tp8647p8667.html
> > Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message