nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aldrin Piri <aldrinp...@gmail.com>
Subject Re: Mutiple dataflow jobs management(lots of jobs)
Date Thu, 10 Mar 2016 18:27:11 GMT
Hi Yan,

We can get more into details and particulars if needed, but have you
experimented with expression language?  I could see a Cron driven approach
which covers your periodic efforts that feeds some number of ExecuteSQL
processors (perhaps one for each database you are communicating with) each
having a table.  This would certainly cut down on the need for the 30k
processors on a one-to-one basis with a given processor.

In terms of monitoring the dataflows, could you describe what else you are
searching for beyond the graph view?  NiFi tries to provide context for the
flow of data but is not trying to be a sole monitoring, we can give
information on a processor basis, but do not delve into specifics.  There
is a summary view for the overall flow where you can monitor stats about
the components and connections in the system. We support interoperation
with monitoring systems via push (ReportingTask) and pull (REST API [2])
semantics.

Any other details beyond your list of how this all interoperates might shed
some more light on what you are trying to accomplish.  It seems like NiFi
should be able to help with this.  With some additional information we may
be able to provide further guidance or at least get some insights on use
cases we could look to improve upon and extend NiFi to support.

Thanks!


[1]
http://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html
[2]
http://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#reporting-tasks
[3] http://nifi.apache.org/docs/nifi-docs/rest-api/index.html

On Sat, Mar 5, 2016 at 9:25 PM, 刘岩 <liuyan@richinfo.cn> wrote:

> Hi All
>
>
>     i'm trying to adapt Nifi to production but can not find an admin
> console which monitoring the dataflows
>
>
>    The scenarios is simple,
>
>
>    1.  we gather data from oracle database to hdfs and then to hive.
>
>    2.  residules/incrementals are updated daily or monthly via Nifi.
>
>    3.  full dump on some table are excuted daily or monthly via Nifi.
>
>
>     is it really simple ,  however , we have  7 oracle databases with over
> 30K  tables needs to implement the above scenario.
>
>
> which means that i will drag that ExcuteSQL  elements for like 30K time or
> so and also need to place them with a nice looking way on my little 21 inch
> screen .
>
>
> Just wondering if there is a table list like  ,groupable and searchable
> task control and monitoring feature for Nifi
>
>
>
> Thank you very much  in advance
>
>
>
> Yan Liu
>
> Hortonworks Service Division
>
> Richinfo, Shenzhen, China (PR)
>
> 06/03/2016
>
>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message