lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joel Bernstein (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SOLR-10003) Add superstep Streaming Expression
Date Thu, 19 Jan 2017 18:46:26 GMT

     [ https://issues.apache.org/jira/browse/SOLR-10003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Joel Bernstein updated SOLR-10003:
----------------------------------
    Description: 
Certain parallel operations, for example *Pregel*, require all workers to complete a step
in the process before starting the next step. This is sometimes called a *superstep*.

This ticket will introduce a Streaming Expression that coordinates the superstep. The basic
structure of this syntax is:

{code}
superstep(parallel(daemon(...)))
{code} 

Each call to superstep.read() calls a parallel(daemon()) which sends a batch job to the workers.
The superstep will only return the tuple from read() when each parallel daemon has completed
the step.

Under the covers the superstep will poll the workers and check the state of the daemons on
the workers. New daemon states will be added so that processes can vote to end the entire
task. When all workers vote to end the task superstep will return the EOF tuple and the job
will end.


  was:
Certain parallel operations, for example *Pregel*, require all workers to complete a step
in the process before starting the next step. This is sometimes called a *superstep*.

This ticket will introduce a Streaming Expression thats coordinates the superstep. The basic
structure of this syntax is:

{code}
superstep(parallel(daemon(...)))
{code} 

Each call to superstep.read() calls a parallel(daemon()) which sends a batch job to the workers.
The superstep will only return the tuple from read() when each parallel daemon has completed
the step.

Under the covers the superstep will poll the workers and check the state of the daemons on
the workers. New daemon states will be added so that processes can vote to end the entire
task. When all workers vote to end the task superstep will return the EOF tuple and the job
will end.



> Add superstep Streaming Expression
> ----------------------------------
>
>                 Key: SOLR-10003
>                 URL: https://issues.apache.org/jira/browse/SOLR-10003
>             Project: Solr
>          Issue Type: New Feature
>      Security Level: Public(Default Security Level. Issues are Public) 
>            Reporter: Joel Bernstein
>
> Certain parallel operations, for example *Pregel*, require all workers to complete a
step in the process before starting the next step. This is sometimes called a *superstep*.
> This ticket will introduce a Streaming Expression that coordinates the superstep. The
basic structure of this syntax is:
> {code}
> superstep(parallel(daemon(...)))
> {code} 
> Each call to superstep.read() calls a parallel(daemon()) which sends a batch job to the
workers. The superstep will only return the tuple from read() when each parallel daemon has
completed the step.
> Under the covers the superstep will poll the workers and check the state of the daemons
on the workers. New daemon states will be added so that processes can vote to end the entire
task. When all workers vote to end the task superstep will return the EOF tuple and the job
will end.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message