falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "sandeep samudrala (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-1406) Effective time in Entity updates.
Date Fri, 11 Nov 2016 05:01:58 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15656149#comment-15656149

sandeep samudrala commented on FALCON-1406:

It doesn't affect any functionality, it leverages new functionality to have for the capability
to hold the versioning of the libs. This feature is really needed as there are a lot of users
whenever they want to run the new libs they end up creating new entities(processes) each time
as they would need old entities's instances to be run with old libs. This results in a huge
number of entities being maintained for the same functionality but with different lib paths
being run, which also adds the burden on the user to keep maintaining each lib and each updated
process to be supplied with a different path having newer version of the libs.

The way I can see with these changes to features is to leverage this new flavor to each of
the user.
As when user wants to run these new instances from a given point of a time, the user ends
up creating a new process and then tracking the sla and backlog for the same and also kill
any instances in the past for the earlier entity, resulting to have unnecessary alerts where
the user already is aware of. It creates cumbersome to handle different processes which have
the similar functionality but except for the changes the user has made recently and would
need it from a specific point of time. 

On the point of extra burden for new features, as far as I can see any feature that listens
to configuration store listener are the ones that would have actually a problem. Even in the
current state of things, when user himself has killed the instances and started a new process
from some start point, would end up getting alerts. By making a simple change of accepting
efffectiveTime on listening to onChange(update)[onChange(Date effectiveTime)], the features
can reset their checks from(monitored time, status checks, backlogs, slas ) to start afresh
from the effectiveTime in case the effective time is some time in the past. This is no burden
to the features but adds a very rich functionality. I would look at this as the existing features
would add more functionality to themselves by falling in line with this change. The changes
to be made are also as minimal as to just update the monitored time to be reset to some past
time if the effective time is in the past. I have raised the jira for the same.[FALCON-2169].

As far as I can see the changes are as minimal as they can be, if you feel there is still
room for any optimization/changes, please feel free to comment on the same. 

> Effective time in Entity updates.
> ---------------------------------
>                 Key: FALCON-1406
>                 URL: https://issues.apache.org/jira/browse/FALCON-1406
>             Project: Falcon
>          Issue Type: New Feature
>            Reporter: sandeep samudrala
>            Assignee: sandeep samudrala
>         Attachments: FALCON-1406-initial.patch, effective_time_in_entity_updates.pdf
> Effective time with entity updates needs to be provided even with past time too. There
was effective time capability provided in the past which gives the functionality to set an
effective time for an entity with only current or future time(now + delay), which could not
solve all the issues. 
> Following are few scenarios which would require effective time to be available with time
back in past.
> a) New code being deployed for an incompatible input data set which would leave instances
with old code and new data.
> b) Bad code being pushed for which, the entity should be able to go back in time to replay(rerun)
with new code.
> c) Orchestration level changes(good/bad) would need functionality to go back in time
to start with.
> For reference: Linking all the Jiras that have been worked upon around effective time
> https://issues.apache.org/jira/browse/FALCON-374
> https://issues.apache.org/jira/browse/FALCON-297

This message was sent by Atlassian JIRA

View raw message