falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Srikanth Sundarrajan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-1406) Effective time in Entity updates.
Date Tue, 22 Nov 2016 02:12:59 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15685433#comment-15685433

Srikanth Sundarrajan commented on FALCON-1406:

HI [~ajayyadava], I have been behind this feature for nearly 3 years now. That we go back
in time to reprocess data regularly is given. There are numerous scenarios when this may happen.
The most common / frequent one being bug in newly deployed code or un-handled data issues.
This in conjunction with problems relating to lack of deployment versioning makes it quite
hard for consumers to handle these scenarios, often resulting in hours / day of backfill handling.
To make matters worse, almost always this results in some kind of temporary feed & processes
created in the system, ending up fairly messing up the entities in the system. This jira essentially
attempts to address this issue cleanly. We attempted to get this solved few years back (FALCON-374
& FALCON-297), but had to roll it back as the implementation had an issue (copying of
all lib jars). IMHO, neither then nor now, it seems to be against any core principle/assumption
on which Falcon was build. It certainly is complex with numerous edge cases to consider (given
the nature of the feature), but that shouldn't be the reason for us to avoid this. Having
said this, it is quite possible that I might be missing something, if you call out specific
assumptions that this feature may possible be in violation then we can weigh in.

> Effective time in Entity updates.
> ---------------------------------
>                 Key: FALCON-1406
>                 URL: https://issues.apache.org/jira/browse/FALCON-1406
>             Project: Falcon
>          Issue Type: New Feature
>            Reporter: sandeep samudrala
>            Assignee: sandeep samudrala
>         Attachments: FALCON-1406-initial.patch, effective_time_in_entity_updates.pdf
> Effective time with entity updates needs to be provided even with past time too. There
was effective time capability provided in the past which gives the functionality to set an
effective time for an entity with only current or future time(now + delay), which could not
solve all the issues. 
> Following are few scenarios which would require effective time to be available with time
back in past.
> a) New code being deployed for an incompatible input data set which would leave instances
with old code and new data.
> b) Bad code being pushed for which, the entity should be able to go back in time to replay(rerun)
with new code.
> c) Orchestration level changes(good/bad) would need functionality to go back in time
to start with.
> For reference: Linking all the Jiras that have been worked upon around effective time
> https://issues.apache.org/jira/browse/FALCON-374
> https://issues.apache.org/jira/browse/FALCON-297

This message was sent by Atlassian JIRA

View raw message