falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Srikanth Sundarrajan <srik...@hotmail.com>
Subject RE: XMLs in the falcon
Date Thu, 03 Apr 2014 05:51:40 GMT
Hi Jagat,    There are a lot of functionality that Falcon provides which would require this
structured metadata (in xmls). I do understand your pain in having to create these xml especially
when the number of feeds you would want to onboard is large. Since you seem to need very few
customisations, perhaps we can add a feature to do expand on standard cluster & feed definition
templates and overlay custom data from a property file or any other input to bootstrap the
xml creation. This will reduce the burden in creating these xmls, while still providing the
expressibility. Do you see this adequately addressing your concern? 
Patch for the same is welcome too. 
RegardsSrikanth Sundarrajan

> Date: Thu, 3 Apr 2014 09:28:55 +1100
> Subject: XMLs in the falcon
> From: jagatsingh@gmail.com
> To: dev@falcon.incubator.apache.org
> Hi,
> I was looking at the falcon basic userguide [1] and the recent blog post of
> same by Hortonworks [2]
> I was just wondering if there is some proposal to reduce the amount of XML
> code needed to ingest any new feed or process into the system.
> Can we have some properties globally defined in the system.
> Cluster A
> Cluster B etc
> Cluster A  temp dir
> Cluster B temp dir
> Cluster A hive parent dir
> Cluster B hive parent dir
> And for any new feed we just need to write something similar to what we do
> in cascading or pig script. 3-4 declarative steps what has to be done with
> that data.
> Write 3-4 lines of code and its all done.
> We can generate XMLs in the background if needed to make it working , but
> writing XMLs for ingesting every need feed is the most scary thing for me
> at this moment to use it in Production. Imagine we have 500 feeds and how
> many XMLs it will be needed to support
> What are your thoughts on this.
> Thanks,
> Jagat Singh
> [1] http://falcon.incubator.apache.org/docs/EntitySpecification.html
> [2]
> http://hortonworks.com/hadoop-tutorial/defining-processing-data-end-end-data-pipeline-apache-falcon/
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message