falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pallavi Rao" <pallavi....@inmobi.com>
Subject Re: Review Request 39844: FALCON-1233 Support co-existence of Oozie scheduler (coord) and Falcon native scheduler.
Date Wed, 25 Nov 2015 06:33:24 GMT


> On Nov. 4, 2015, 7:01 a.m., pavan kumar kolamuri wrote:
> > common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java, line
84
> > <https://reviews.apache.org/r/39844/diff/1/?file=1114449#file1114449line84>
> >
> >     Once entity is migrated to Native Scheduler. What will happen if user wants
to get status of instances of an entity with start time before migration time. (startTime
< migrationTime). Will user gets the consolidated view as per doc says ?

With this patch, the assumption is that user will delete older entity, create a new one and
schedule it on native scheduler. So, no history will be available. However, in the next phase,
I do plan to provide an option to move an existing entity to native scheduler which will preserve
the instance history.


> On Nov. 4, 2015, 7:01 a.m., pavan kumar kolamuri wrote:
> > common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java, line
102
> > <https://reviews.apache.org/r/39844/diff/1/?file=1114449#file1114449line102>
> >
> >     If entity is null shouldn't we throw exception ? In which case entity can be
null and still we will return Primary Engine.

With the previous API, the usage was WorkflowEngineFactory.getWorkflowEngine(). Entity was
not an argument at all. Just to preserve the backward compatibility of API for such usage,
if no entity is supplied, the default oozie engine is returned.


> On Nov. 4, 2015, 7:01 a.m., pavan kumar kolamuri wrote:
> > scheduler/src/test/java/org/apache/falcon/workflow/engine/WorkflowEngineFactoryTest.java,
line 103
> > <https://reviews.apache.org/r/39844/diff/1/?file=1114456#file1114456line103>
> >
> >     Other case props set to secondary should be added right ?

Ideally .. yes.. But, there is check in there where I invoke OozieWorkflowEngine.isAlive()
that actually makes a oozie call and it was too much pain to Mock the entire class. Didn't
think it was worth it.


- Pallavi


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39844/#review105036
-----------------------------------------------------------


On Nov. 25, 2015, 6:32 a.m., Pallavi Rao wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39844/
> -----------------------------------------------------------
> 
> (Updated Nov. 25, 2015, 6:32 a.m.)
> 
> 
> Review request for Falcon.
> 
> 
> Bugs: FALCON-1233
>     https://issues.apache.org/jira/browse/FALCON-1233
> 
> 
> Repository: falcon-git
> 
> 
> Description
> -------
> 
> The migration to Falcon native scheduler should be as seamless as possible and the user
should be able to migrate to the native scheduler in a phased manner (entity by entity).
> 
> This patch introduces the notion of primary and secondary workflow engines. It allows
users to schedule an entity on the secondary engine (primary, by default). The idea is that
initially, users will use Falcon native scheduler as the secondary engine to migrate their
entities one by one. When most entities have been migrated, the Oozie scheduler becomes secondary
(and eventually removed).
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java 49592ac

>   common/src/main/java/org/apache/falcon/workflow/WorkflowJobEndNotificationService.java
9d96fa3 
>   common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java
7b36b11 
>   docs/src/site/twiki/falconcli/Schedule.twiki 42192c7 
>   oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 724f646

>   prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java 16ef83a 
>   prism/src/main/java/org/apache/falcon/resource/AbstractInstanceManager.java fea2989

>   prism/src/main/java/org/apache/falcon/resource/AbstractSchedulableEntityManager.java
d317aa1 
>   scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java
d7d157f 
>   scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java ca2010b

>   scheduler/src/test/java/org/apache/falcon/execution/FalconExecutionServiceTest.java
bff92c9 
>   scheduler/src/test/java/org/apache/falcon/workflow/engine/WorkflowEngineFactoryTest.java
PRE-CREATION 
>   unit/pom.xml 8d9f443 
> 
> Diff: https://reviews.apache.org/r/39844/diff/
> 
> 
> Testing
> -------
> 
> UTs added. Manual testing done.
> 
> 
> Thanks,
> 
> Pallavi Rao
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message