lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dennis Gove (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SOLR-9096) Add PartitionStream to Streaming Expressions
Date Mon, 09 May 2016 22:47:12 GMT

     [ https://issues.apache.org/jira/browse/SOLR-9096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dennis Gove updated SOLR-9096:
------------------------------
    Description: 
The basic idea of a PartitionStream is to take a one or more input streams of tuples, partition
them out to a set of workers such that each worker can work with a subset of the tuples, and
then bring them all back into a single stream. This differs from a ParallelStream because
in ParallelStream the data is partitioned at the source whereas with a PartitionStream one
can take an existing stream and spread it out across workers.

The use-case here is for when one has a source stream (or more) which cannot be parallelized
at the source but which can be parallelized after some level of processing. I see this being
used most for parallelized sort, rollups, or graph searches.

{code}
                            /--- sort ----\
                           /               \                      /--- Collection A
                          / ---- sort ----- \                    /
Client <--- rollup  <----<                    <----- innerJoin <
                          \ ---- sort ----- /                    \
                           \               /                      \--- Collection B
                            \--- sort ----/
{code}

{code}
                            /--- sort -- rollup ----\
                           /                         \                      /--- Collection
A
                          / ---- sort -- rollup ----- \                    /
Client <-- innerJoin <---<                              <----- innerJoin <
                    \     \ ---- sort -- rollup ----- /                    \
                     \     \                         /                      \--- Collection
B
                      \     \--- sort -- rollup ----/
                       \
                        \
                         \ <--- jdbc source
{code}

{code}
                            /--- sort -- innerJoin ----\
                           /                            \  
                          / ---- sort -- innerJoin ----- \  <--- jdbc source
Client <-- innerJoin <---<                                | 
                    \     \ ---- sort -- innerJoin ----- /  <--- rollup <---- Collection
A
                     \     \                            /  
                      \     \--- sort -- innerJoin ----/
                       \
                        \
                         \ <--- jdbc source
{code}



I imagine partition expression would look something like this

{code}
partition(
  inputA=<source stream A>,
  inputB=<source stream B>,
  work=<stream for the workers>,
  over="fieldA,fieldB",
  workers=6,
  zkHost=<zk connection string>
)
{code}

for example

{code}
innerJoin(
  partition(
    inputA=jdbc(database1),
    inputB=rollup(
      search(collectionA, ...),
      ...
    ),
    work=sort(
      innerJoin(
        inputA,
        inputB,
        on="fieldA,fieldB"
      ),
      by="jdbcFieldC asc, collectionAFieldB desc"
    ),
    workers=6,
    zkHost=localhost:12345
  ),
  jdbc(database2),
  on="fieldZ"
)
{code}


  was:
The basic idea of a PartitionStream is to take a one or more input streams of tuples, partition
them out to a set of workers such that each worker can work with a subset of the tuples, and
then bring them all back into a single stream. This differs from a ParallelStream because
in ParallelStream the data is partitioned at the source whereas with a PartitionStream one
can take an existing stream and spread it out across workers.

{code}
                            /--- sort ----\
                           /               \                      /--- Collection A
                          / ---- sort ----- \                    /
Client <--- rollup  <----<                    <----- innerJoin <
                          \ ---- sort ----- /                    \
                           \               /                      \--- Collection B
                            \--- sort ----/
{code}

{code}
                            /--- sort -- rollup ----\
                           /                         \                      /--- Collection
A
                          / ---- sort -- rollup ----- \                    /
Client <-- innerJoin <---<                              <----- innerJoin <
                    \     \ ---- sort -- rollup ----- /                    \
                     \     \                         /                      \--- Collection
B
                      \     \--- sort -- rollup ----/
                       \
                        \
                         \ <--- jdbc source
{code}

{code}
                            /--- sort -- innerJoin ----\
                           /                            \  
                          / ---- sort -- innerJoin ----- \  <--- jdbc source
Client <-- innerJoin <---<                                | 
                    \     \ ---- sort -- innerJoin ----- /  <--- rollup <---- Collection
A
                     \     \                            /  
                      \     \--- sort -- innerJoin ----/
                       \
                        \
                         \ <--- jdbc source
{code}



I imagine partition expression would look something like this

{code}
partition(
  inputA=<source stream A>,
  inputB=<source stream B>,
  work=<stream for the workers>,
  over="fieldA,fieldB",
  workers=6,
  zkHost=<zk connection string>
)
{code}

for example

{code}
innerJoin(
  partition(
    inputA=jdbc(database1),
    inputB=rollup(
      search(collectionA, ...),
      ...
    ),
    work=sort(
      innerJoin(
        inputA,
        inputB,
        on="fieldA,fieldB"
      ),
      by="jdbcFieldC asc, collectionAFieldB desc"
    ),
    workers=6,
    zkHost=localhost:12345
  ),
  jdbc(database2),
  on="fieldZ"
)
{code}



> Add PartitionStream to Streaming Expressions
> --------------------------------------------
>
>                 Key: SOLR-9096
>                 URL: https://issues.apache.org/jira/browse/SOLR-9096
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Dennis Gove
>
> The basic idea of a PartitionStream is to take a one or more input streams of tuples,
partition them out to a set of workers such that each worker can work with a subset of the
tuples, and then bring them all back into a single stream. This differs from a ParallelStream
because in ParallelStream the data is partitioned at the source whereas with a PartitionStream
one can take an existing stream and spread it out across workers.
> The use-case here is for when one has a source stream (or more) which cannot be parallelized
at the source but which can be parallelized after some level of processing. I see this being
used most for parallelized sort, rollups, or graph searches.
> {code}
>                             /--- sort ----\
>                            /               \                      /--- Collection A
>                           / ---- sort ----- \                    /
> Client <--- rollup  <----<                    <----- innerJoin <
>                           \ ---- sort ----- /                    \
>                            \               /                      \--- Collection B
>                             \--- sort ----/
> {code}
> {code}
>                             /--- sort -- rollup ----\
>                            /                         \                      /--- Collection
A
>                           / ---- sort -- rollup ----- \                    /
> Client <-- innerJoin <---<                              <----- innerJoin
<
>                     \     \ ---- sort -- rollup ----- /                    \
>                      \     \                         /                      \--- Collection
B
>                       \     \--- sort -- rollup ----/
>                        \
>                         \
>                          \ <--- jdbc source
> {code}
> {code}
>                             /--- sort -- innerJoin ----\
>                            /                            \  
>                           / ---- sort -- innerJoin ----- \  <--- jdbc source
> Client <-- innerJoin <---<                                | 
>                     \     \ ---- sort -- innerJoin ----- /  <--- rollup <---- Collection
A
>                      \     \                            /  
>                       \     \--- sort -- innerJoin ----/
>                        \
>                         \
>                          \ <--- jdbc source
> {code}
> I imagine partition expression would look something like this
> {code}
> partition(
>   inputA=<source stream A>,
>   inputB=<source stream B>,
>   work=<stream for the workers>,
>   over="fieldA,fieldB",
>   workers=6,
>   zkHost=<zk connection string>
> )
> {code}
> for example
> {code}
> innerJoin(
>   partition(
>     inputA=jdbc(database1),
>     inputB=rollup(
>       search(collectionA, ...),
>       ...
>     ),
>     work=sort(
>       innerJoin(
>         inputA,
>         inputB,
>         on="fieldA,fieldB"
>       ),
>       by="jdbcFieldC asc, collectionAFieldB desc"
>     ),
>     workers=6,
>     zkHost=localhost:12345
>   ),
>   jdbc(database2),
>   on="fieldZ"
> )
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message