falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ajay Yadava (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (FALCON-2030) Enforce time partition pattern in the data location path in feed definition
Date Wed, 15 Jun 2016 11:44:09 GMT

    [ https://issues.apache.org/jira/browse/FALCON-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331570#comment-15331570
] 

Ajay Yadava edited comment on FALCON-2030 at 6/15/16 11:43 AM:
---------------------------------------------------------------

There is one use case - a metadata snapshot whose only one instance is maintained. The limiting
factor is not Oozie as Oozie doesn't mandate it to have timeseries pattern. However, IIRC
retention will have problem in such cases but use case requires that metadata should never
be deleted so people use something like years(999).
It is not uncommon to have such metadata use cases e.g. typically in ad-networks metadata
like device metadata, ip metadata etc. are used like that.

[~me.venkatr] Won't it be the case with incremental import as well?

I have been away for a while so not aware of the exact motivation to force this validation.
Are there any issues because of this?



was (Author: ajayyadava):
There is one use case - a metadata snapshot whose only one instance is maintained. The limiting
factor is not Oozie as Oozie doesn't mandate it to have timeseries pattern. However, IIRC
retention will have problem in such cases but use case requires that metadata should never
be deleted so people use something like years(999).
It is not uncommon to have such metadata use cases e.g. typically in ad-networks metadata
like device metadata, ip metadata etc. are used like that.

[~me.venkatr] Won't it be the case with incremental snapshots in import as well?

I have been away for a while so not aware of the exact motivation to force this validation.
Are there any issues because of this?


> Enforce time partition pattern in the data location path in feed definition 
> ----------------------------------------------------------------------------
>
>                 Key: FALCON-2030
>                 URL: https://issues.apache.org/jira/browse/FALCON-2030
>             Project: Falcon
>          Issue Type: Improvement
>          Components: feed
>            Reporter: Venkatesan Ramachandran
>            Assignee: Venkatesan Ramachandran
>
> In feed definition, data location can be specified without time series pattern like below:
>    <locations>
>         <location type="data" path="/tmp/falcon-regression/RetentionTest/testFolders/"/>
>         <location type="stats" path="/projects/falcon/clicksStats"/>
>         <location type="meta" path="/projects/falcon/clicksMetaData"/>
>     </locations>



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message