falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Balu Vellanki (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-1372) Retention does not work in corner cases
Date Thu, 05 Nov 2015 16:21:27 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14991908#comment-14991908
] 

Balu Vellanki commented on FALCON-1372:
---------------------------------------

This seems to be an invalid bug. My retention coord is running on Nov 4th, 10 AM in San Francisco.
Feed frequency is monthly, Timezone is UTC and retention time is 4days. So, anything older
than 2015 Nov 1st 10 AM San Francisco time will be evicted. The tester is expecting 2015/10
to be deleted and 2015/11 to not be evicted. However 2015/11 is in UTC and it translates to
2015-10-31T16:00 pacific time, so it will be evicted.
A few more cases below.
- If feed frequency is daily and timezone is UTC, 2015/11/01 will translate to 2015-10-31T16:00
pacific time and will be evicted.
- If feed frequency is hourly and timezone is UTC, 2015/11/01/10 will translate to 2015-11-01T02:00
pacific time and will be evicted.
- If feed frequency is hourly and timezone is UTC, 2015/11/01/19 will translate to 2015-11-01T11:00
pacific time and will NOT be evicted.

> Retention does not work in corner cases 
> ----------------------------------------
>
>                 Key: FALCON-1372
>                 URL: https://issues.apache.org/jira/browse/FALCON-1372
>             Project: Falcon
>          Issue Type: Bug
>    Affects Versions: 0.6.1
>            Reporter: Balu Vellanki
>            Assignee: Balu Vellanki
>
> Originally reported by [~rostafiychuk].
> RetentionTest#testRetention (24, DAYS, true, MONTHLY, true)
> Uploaded feed:
> {code:xml}
> <?xml version="1.0" encoding="UTF-8"?><feed xmlns="uri:falcon:feed:0.1" name="A852708b0-3057547e"
description="clicks log">
>     <frequency>hours(1)</frequency>
>     <timezone>UTC</timezone>
>     <late-arrival cut-off="hours(6)"/>
>     <clusters>
>         <cluster name="A852708b0-cd30589f" type="source">
>             <validity start="2012-01-30T00:00Z" end="2099-03-31T23:59Z"/>
>             <retention limit="days(24)" action="delete"/>
>         </cluster>
>     </clusters>
>     <locations>
>         <location type="data" path="/tmp/falcon-regression/RetentionTest/testFolders/${YEAR}/${MONTH}"/>
>         <location type="stats" path="/projects/falcon/clicksStats"/>
>         <location type="meta" path="/projects/falcon/clicksMetaData"/>
>     </locations>
>     <ACL owner="hrt_qa" group="users" permission="*"/>
>     <schema location="/schema/clicks" provider="protobuf"/>
>     <properties/>
> </feed>
> {code}
> Created dirs:
> {code}
> dataDates = [2012/06, 2012/07, 2012/08, 2012/09, 2012/10, 2012/11, 2012/12, 2013/01,
2013/02, 2013/03, 2013/04, 2013/05, 2013/06, 2013/07, 2013/08, 2013/09, 2013/10, 2013/11,
2013/12, 2014/01, 2014/02, 2014/03, 2014/04, 2014/05, 2014/06, 2014/07, 2014/08, 2014/09,
2014/10, 2014/11, 2014/12, 2015/01, 2015/02, 2015/03, 2015/04, 2015/05, 2015/06, 2015/07,
2015/08, 2015/09, 2015/10, 2015/11, 2015/12, 2016/01, 2016/02, 2016/03, 2016/04, 2016/05,
2016/06, 2016/07, 2016/08, 2016/09, 2016/10, 2016/11, 2016/12, 2017/01, 2017/02, 2017/03,
2017/04, 2017/05, 2017/06, 2017/07, 2017/08, 2017/09, 2017/10, 2017/11, 2017/12, 2018/01,
2018/02, 2018/03, 2018/04, 2018/05, 2018/06, 2018/07] {code}
> expected for deletion:
> {code} [2012/06, 2012/07, 2012/08, 2012/09, 2012/10, 2012/11, 2012/12, 2013/01, 2013/02,
2013/03, 2013/04, 2013/05, 2013/06, 2013/07, 2013/08, 2013/09, 2013/10, 2013/11, 2013/12,
2014/01, 2014/02, 2014/03, 2014/04, 2014/05, 2014/06, 2014/07, 2014/08, 2014/09, 2014/10,
2014/11, 2014/12, 2015/01, 2015/02, 2015/03, 2015/04, 2015/05]
> {code}
> deleted:
> {code:javascript}[2012/06, 2012/07, 2012/08, 2012/09, 2012/10, 2012/11, 2012/12, 2013/01,
2013/02, 2013/03, 2013/04, 2013/05, 2013/06, 2013/07, 2013/08, 2013/09, 2013/10, 2013/11,
2013/12, 2014/01, 2014/02, 2014/03, 2014/04, 2014/05, 2014/06, 2014/07, 2014/08, 2014/09,
2014/10, 2014/11, 2014/12, 2015/01, 2015/02, 2015/03, 2015/04, 2015/05, 2015/06]
> {code}
> 24 days ago was 31th of May, but *2015/06* was deleted too.
> Another case:
> RetentionTest#testRetention (10080, HOURS, false, MONTHLY, true)
> {code:xml}
> <retention limit="hours(10080)" action="delete"/>
> {code}
> 10080 hours ago was 30th of April, but dir *"2014/05"* was deleted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message