beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Kirpichov (JIRA)" <>
Subject [jira] [Created] (BEAM-92) Data-dependent sinks
Date Thu, 03 Mar 2016 17:08:18 GMT
Eugene Kirpichov created BEAM-92:

             Summary: Data-dependent sinks
                 Key: BEAM-92
             Project: Beam
          Issue Type: New Feature
            Reporter: Eugene Kirpichov

Current sink API writes all data to a single destination, but there are many use cases where
different pieces of data need to be routed to different destinations where the set of destinations
is data-dependent (so can't be implemented with a Partition transform).

One internally discussed proposal was an API of the form:
PCollection<Void> PCollection<T>.apply(
    Write.using(DoFn<T, SinkT> where,
                MapFn<SinkT, WriteOperation<WriteResultT, T>> how)

so an item T gets written to a destination (or multiple destinations) determined by "where";
and the writing strategy is determined by "how" that produces a WriteOperation (current API
- global init/write/global finalize hooks) for any given destination.

This API also has other benefits:
* allows the SinkT to be computed dynamically (in "where"), rather than specified at pipeline
construction time
* removes the necessity for a Sink class entirely
* is sequenceable w.r.t. downstream transforms (you can stick transforms onto the returned
PCollection<Void>, while the current returns a PDone)

This message was sent by Atlassian JIRA

View raw message