helix-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HELIX-601) Allow work flow to schedule dependency jobs in parallel
Date Mon, 06 Jul 2015 23:11:05 GMT

    [ https://issues.apache.org/jira/browse/HELIX-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14615841#comment-14615841

ASF GitHub Bot commented on HELIX-601:

Github user jicongrui commented on the pull request:

    The comment says, "_parallelJobs would kind of break the job dependency, e.g: if job1
-> job2, but _parallelJobs == 2, then job1 and job2 could be scheduled at the same time".
    Parallel and dependency are semantically incompatible, and this diff allows user to put
"parallel" in a higher priority than "dependency". User must know "dependency" would be downgraded
to "preferred schedule order" when parallel count is bigger than 1.
    For 1, even there is node failure, job1 can still be finished before job0. This diff doesn't
commits the finishing order.
    For 2, job0 and job1 are scheduled to different instances and executed in parallel.

> Allow work flow to schedule dependency jobs in parallel
> -------------------------------------------------------
>                 Key: HELIX-601
>                 URL: https://issues.apache.org/jira/browse/HELIX-601
>             Project: Apache Helix
>          Issue Type: New Feature
>            Reporter: Congrui Ji
> Currently, Helix won't schedule dependency jobs in a same work flow. For example, if
Job2 depends on Job1, Job2 won't be scheduled until every partition of Job1 is completed.
> However, if some participant is very slow, then all dependency jobs is waiting for that
single participant.
> Helix should be able to schedule multiple jobs according to a parameter.
> A.C.
> 1. Introduce parallel count parameter in work flow and job queue.
> 2. Dependency jobs can be scheduled according to the parameter (Now the parameter is
always 1, so no parallel)
> 3. If Job2 depends on Job1, Job1 is scheduled before Job2.
> 4. No parallel jobs on the same instance. If a instance is running Job1, it won't run
Job2 until Job1 is finished. 

This message was sent by Atlassian JIRA

View raw message