nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Zhurakousky <ozhurakou...@hortonworks.com>
Subject Re: Nifi jobs support Mesos ?
Date Tue, 02 Feb 2016 01:08:38 GMT
Timothy

In short it’s very possible. I personally haven’t done that, but I have written some custom
code submitting jobs to YARN which I am assuming would be very similar. 
Would be happy and actually interested to take it a bit further. Could you elaborate on the
use case a bit more? For example; Where is the source of data? How does Mesos know about where
the data is? 
Sorry some of these question may be very dumb (not much experience with Mesos other then general
understanding) and what responsibility do you envision NiFi to have over such process?

I’ll share my example with NiFi and Tez where a custom NiFi Processor would submit job to
Tez and upon completion would create a FlowFile that would trigger another processor which
would invoke another Tez job and so on. Tez essentially is a processing wrapper that uses
YARN for resource management.
So I am assuming that something similar would be with Mesos. Am I on the right track?

Cheers
Oleg
 
> On Feb 1, 2016, at 4:53 PM, Timothy Findlay <tfindlay@prodevelop.com.au> wrote:
> 
> Hi Folks,
> 
> 
> Forgive me as I'm new to Nifi. The tool seems fantastic, the UI is super easy to use.
It was easy to extend and build a custom processor which is great.
> 
> 
> Looking at our processing, we're curious if we could use Nifi to replace some of our
batch processes. Setting up some of the basic routines, scheduled by cron is all working.
> 
> 
> However we would like the ability to scale jobs out, horizontally and distribute processing
across a number of Mesos nodes.
> 
> 
> Has anyone done this ? any notes, tips or documentation on how to set this up ?
> 
> 
> Thanks.

Mime
View raw message