falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Samarth Gupta <samarthgupta...@gmail.com>
Subject Re: XMLs in the falcon
Date Thu, 03 Apr 2014 06:16:45 GMT
falcon QA team already use JAXB to generate various test xmls as runtime to
test different cases. We have a feed / process template which we modify at
runtime before submitting via api for testing.

some code might be reusable ....

below is the repo link :
https://github.com/samarthgupta437/falcon-regression

extended class with customized functions:
https://github.com/samarthgupta437/falcon-regression/blob/master/merlin-core/src/main/java/org/apache/falcon/regression/Entities/FeedMerlin.java

Thanks:
Samarth


On Thu, Apr 3, 2014 at 11:29 AM, Jean-Baptiste Onofré <jb@nanthrax.net>wrote:

> Hi,
>
> Even using the REST API, you need to provide the "init/descriptor" XML.
>
> Maybe you can create your own tooling (leverage falcon-client where JAXB
> descriptors are located).
> I started to create a tooling in Karaf to easily create XML:
>
> karaf@root()> falcon:cluster-create
> Write interface location:
> ...
>
> Especially the falcon:feed-create accepts a -r option to recursively
> browse a filesystem and create all feeds (for now, it's pretty basic).
>
> I plan to discuss with Srikanth and the team to "merge/donate" this
> tooling in Falcon (I tried to ping him yesterday but he was busy).
>
> My $0.02
>
> Regards
> JB
>
>
> On 04/03/2014 12:28 AM, Jagat Singh wrote:
>
>> Hi,
>>
>> I was looking at the falcon basic userguide [1] and the recent blog post
>> of
>> same by Hortonworks [2]
>>
>> I was just wondering if there is some proposal to reduce the amount of XML
>> code needed to ingest any new feed or process into the system.
>>
>> Can we have some properties globally defined in the system.
>>
>> Cluster A
>> Cluster B etc
>> Cluster A  temp dir
>> Cluster B temp dir
>> Cluster A hive parent dir
>> Cluster B hive parent dir
>>
>> And for any new feed we just need to write something similar to what we do
>> in cascading or pig script. 3-4 declarative steps what has to be done with
>> that data.
>>
>> Write 3-4 lines of code and its all done.
>>
>> We can generate XMLs in the background if needed to make it working , but
>> writing XMLs for ingesting every need feed is the most scary thing for me
>> at this moment to use it in Production. Imagine we have 500 feeds and how
>> many XMLs it will be needed to support
>>
>> What are your thoughts on this.
>>
>> Thanks,
>>
>> Jagat Singh
>>
>>
>>
>>
>>
>>
>>
>>
>> [1] http://falcon.incubator.apache.org/docs/EntitySpecification.html
>> [2]
>> http://hortonworks.com/hadoop-tutorial/defining-processing-
>> data-end-end-data-pipeline-apache-falcon/
>>
>>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message