falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ajay Yadava (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-2030) Enforce time partition pattern in the data location path in feed definition
Date Sat, 18 Jun 2016 15:04:05 GMT

    [ https://issues.apache.org/jira/browse/FALCON-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337964#comment-15337964

Ajay Yadava commented on FALCON-2030:

Hello [~me.venkatr],

Feed doesn't have any restriction that there should be a consumer. What if there are no consuming
processes? May be the user just uses falcon for periodic cleaning of a directory on Hadoop.
In other scenario the user may maintain only one copy of the metadata and uses it as a feed
for ad-hoc analysis. May be the users have aperiodic data and maintains versions of instances
but not in YYYY-MM-DD format.

The larger point is that retention today is possible for any particular type of data - periodic/aperiodic,
versioned/non-versioned and users don't need to model their data locations in a particular
pattern in order to onboard falcon. This is a great advantage. Forcing a particular pattern
in data location is restrictive IMO and the lack of this validation shouldn't cause any issues.

Hope it helps.

> Enforce time partition pattern in the data location path in feed definition 
> ----------------------------------------------------------------------------
>                 Key: FALCON-2030
>                 URL: https://issues.apache.org/jira/browse/FALCON-2030
>             Project: Falcon
>          Issue Type: Improvement
>          Components: feed
>            Reporter: Venkatesan Ramachandran
>            Assignee: Venkatesan Ramachandran
> In feed definition, data location can be specified without time series pattern like below:
>    <locations>
>         <location type="data" path="/tmp/falcon-regression/RetentionTest/testFolders/"/>
>         <location type="stats" path="/projects/falcon/clicksStats"/>
>         <location type="meta" path="/projects/falcon/clicksMetaData"/>
>     </locations>

This message was sent by Atlassian JIRA

View raw message