falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ajay Yadava (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-1406) Effective time in Entity updates.
Date Tue, 22 Nov 2016 17:08:58 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15687308#comment-15687308

Ajay Yadava commented on FALCON-1406:

Hi [~sriksun] 

I know that you want to see this feature go live for a long time, but with all sincerest intentions
I believe that Falcon is better off without this change :) 

I think we all can agree that this change doesn't provide any new remarkable capabilities
(capabilities that would have helped someone evaluating falcon for their use case and found
it short) but just a way of doing things without creating temporary entities and probably
more conveniently.

Temporary entities are not created just for the reasons solved by this feature, they will
be created even after this change e.g. during colo migrations, avoiding restrictions of moving
start date to an earlier time and many more. It doesn't impact the system in any way, however,
if you prefer to keep system "clean", then providing a utiliity to "cleanse" the system of
such entities is much better and something which was proposed by the operations people who
are the most affected by this "mess".

The fact that an entity definition can change for a past instance is a rewrite of history
and in my view is a violation of fundamental "assumption". Apart from just being a general
bad practice IMHO, a lot of features developed in falcon took the immutability of entity definition
for past instances as given. I suspect the lack of details in design doc has kept you from
the true nature/impact of the change. I hope I am able to surface some of them once [~sandeep.samudrala]
updates the design doc with other qustions that I have asked.

Now, we can start fixing those issues on a case by case basis or we can just reevaluate the
trade off of this change in favor of a command to clean up(addresses much wider use cases)
or versioning of the entities.

> Effective time in Entity updates.
> ---------------------------------
>                 Key: FALCON-1406
>                 URL: https://issues.apache.org/jira/browse/FALCON-1406
>             Project: Falcon
>          Issue Type: New Feature
>            Reporter: sandeep samudrala
>            Assignee: sandeep samudrala
>         Attachments: FALCON-1406-initial.patch, effective_time_in_entity_updates.pdf
> Effective time with entity updates needs to be provided even with past time too. There
was effective time capability provided in the past which gives the functionality to set an
effective time for an entity with only current or future time(now + delay), which could not
solve all the issues. 
> Following are few scenarios which would require effective time to be available with time
back in past.
> a) New code being deployed for an incompatible input data set which would leave instances
with old code and new data.
> b) Bad code being pushed for which, the entity should be able to go back in time to replay(rerun)
with new code.
> c) Orchestration level changes(good/bad) would need functionality to go back in time
to start with.
> For reference: Linking all the Jiras that have been worked upon around effective time
> https://issues.apache.org/jira/browse/FALCON-374
> https://issues.apache.org/jira/browse/FALCON-297

This message was sent by Atlassian JIRA

View raw message