nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lawrance, Greg J JTST:EX" <Greg.Lawra...@gov.bc.ca>
Subject Re: Apache NiFI - Atlas Bridge Builders Needed
Date Wed, 06 Sep 2017 16:13:40 GMT
Koji,

Thank you very much for this! It looks very promising - and may go a long
way towards resolving our problem. We will give it a more thorough review
/ trial over the next few days.

Greg


On 2017-09-05, 5:30 PM, "Koji Kawamura" <ijokarumawak@gmail.com> wrote:

>Hello Lawrance,
>
>I've been working on NIFI-3709, adding a Apache NiFi reporting task to
>report lineage to Apache Atlas.
>The original PullRequest did not use Provenance events, but I'm now
>adding it so that it can cover more use-cases.
>
>It's still under development and not ready to be reviewed fully, but
>you can look at the code to see if it covers your needs, I hope it
>does, if not, please let me know.
>https://github.com/ijokarumawak/nifi/tree/nifi-3709-2/nifi-nar-bundles/nif
>i-atlas-bundle/nifi-atlas-reporting-task
>
>The core part of analyzing NiFi Provenance events is implemented in
>this package, and extensible.
>An analyzer is picked by following matching rules:
>1. The component type that emitted the provenance event
>2. The transit URI of provenance event
>3. The event type of the provenance event
>
>For general protocols such as hdfs or hbase can be covered by #2.
>(http, jdbc and such can be done similarly in the future)
>If specific implementation is needed, it can be done as #1.
>
>https://github.com/ijokarumawak/nifi/tree/nifi-3709-2/nifi-nar-bundles/nif
>i-atlas-bundle/nifi-atlas-reporting-task/src/main/java/org/apache/nifi/atl
>as/provenance/analyzer
>
>The work also has a mechanism to resolve Atlas cluster name from
>hostname/ip address.
>User can define mapping of those at the ReportingTask's configuration.
>https://github.com/ijokarumawak/nifi/tree/nifi-3709-2/nifi-nar-bundles/nif
>i-atlas-bundle/nifi-atlas-reporting-task/src/main/java/org/apache/nifi/atl
>as/resolver
>
>Thanks,
>Koji
>
>
>On Wed, Sep 6, 2017 at 6:00 AM, Lawrance, Greg J JTST:EX
><Greg.Lawrance@gov.bc.ca> wrote:
>> We are working with Apache NiFi and Apache Atlas (within the Hortonworks
>> HDP framework) and we are looking for some paid help to develop
>>something
>> that meets our needs.  We posted the opportunity as an issue on GitHub
>> associated with the repo where code would be delivered
>>
>> https://github.com/bcgov/nifi-atlas/issues/1 .
>>
>>
>>  Any suggestions of where
>> we might advertise / share this information would be greatly
>>appreciated.
>>
>> Thanks in advance.
>>
>> Greg
>>


Mime
View raw message