falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ajay Yadava (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-2030) Enforce time partition pattern in the data location path in feed definition
Date Wed, 15 Jun 2016 11:41:09 GMT

    [ https://issues.apache.org/jira/browse/FALCON-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331570#comment-15331570

Ajay Yadava commented on FALCON-2030:

There is one use case - a metadata snapshot whose only one instance is maintained. The limiting
factor is not Oozie as Oozie doesn't mandate it to have timeseries pattern. However, IIRC
retention will have problem in such cases but use case requires that metadata should never
be deleted so people use something like years(999).
It is not uncommon to have such metadata use cases e.g. typically in ad-networks metadata
like device metadata, ip metadata etc. are used like that.

[~me.venkatr] Won't it be the case with incremental snapshots in import as well?

I have been away for a while so not aware of the exact motivation to force this validation.
Are there any issues because of this?

> Enforce time partition pattern in the data location path in feed definition 
> ----------------------------------------------------------------------------
>                 Key: FALCON-2030
>                 URL: https://issues.apache.org/jira/browse/FALCON-2030
>             Project: Falcon
>          Issue Type: Improvement
>          Components: feed
>            Reporter: Venkatesan Ramachandran
>            Assignee: Venkatesan Ramachandran
> In feed definition, data location can be specified without time series pattern like below:
>    <locations>
>         <location type="data" path="/tmp/falcon-regression/RetentionTest/testFolders/"/>
>         <location type="stats" path="/projects/falcon/clicksStats"/>
>         <location type="meta" path="/projects/falcon/clicksMetaData"/>
>     </locations>

This message was sent by Atlassian JIRA

View raw message