InMemoryWatermarkManager (Apache Beam SDK for Java 0.1.0-incubating API)

java.lang.Object
-
- org.apache.beam.runners.direct.InMemoryWatermarkManager
-

-
-
-
```
public class InMemoryWatermarkManager
-extends Object
```
-
Manages watermarks of PCollections and input and output watermarks of - AppliedPTransforms to provide event-time and completion tracking for - in-memory execution. InMemoryWatermarkManager is designed to update and return a - consistent view of watermarks in the presence of concurrent updates. - -
An InMemoryWatermarkManager is provided with the collection of root - AppliedPTransforms and a map of PCollections to - all the AppliedPTransforms that consume them at construction time. - -
Whenever a root transform produces elements, the - InMemoryWatermarkManager is provided with the produced elements and the output watermark - of the producing transform. The - watermark manager is responsible for computing the watermarks - of all transforms that consume one or more - PCollections. - -
Whenever a non-root AppliedPTransform finishes processing one or more in-flight - elements (referred to as the input bundle), the following occurs - atomically: -
- All of the in-flight elements are removed from the collection of pending elements for the - AppliedPTransform.
- All of the elements produced by the AppliedPTransform are added to the collection - of pending elements for each AppliedPTransform that consumes them.
- The input watermark for the AppliedPTransform becomes the maximum value of -
  - the previous input watermark
  - the minimum of -
    - the timestamps of all currently pending elements
    - all input PCollection watermarks
    -
  -
- The output watermark for the AppliedPTransform becomes the maximum of -
  - the previous output watermark
  - the minimum of -
    - the current input watermark
    - the current watermark holds
    -
  -
- The watermark of the output PCollection can be advanced to the output watermark of - the AppliedPTransform
- The watermark of all downstream AppliedPTransforms can be - advanced.
- -
The watermark of a PCollection is equal to the output watermark of the - AppliedPTransform that produces it. - -
The watermarks for a PTransform are updated as follows when output is committed:
```
- Watermark_In'  = MAX(Watermark_In, MIN(U(TS_Pending), U(Watermark_InputPCollection)))
- Watermark_Out' = MAX(Watermark_Out, MIN(Watermark_In', U(StateHold)))
- Watermark_PCollection = Watermark_Out_ProducingPTransform
- 
```
-

- -

Nested Class Summary

- - - - - - - - - - - - - - - - - - -

Nested Classes
Modifier and Type	Class and Description
`static class`	`InMemoryWatermarkManager.FiredTimers` - A pair of `TimerInternals.TimerData` and key which can be delivered to the appropriate - `AppliedPTransform`. -
`static class`	`InMemoryWatermarkManager.TimerUpdate` - A collection of newly set, deleted, and completed timers. -
`class`	`InMemoryWatermarkManager.TransformWatermarks` - A reference to the input and output watermarks of an `AppliedPTransform`. -

- -

- - -

Method Summary

- - - - - - - - - - - - - - - - - - - - - - - - - - -

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`static InMemoryWatermarkManager`	`create(Clock clock, - Collection<AppliedPTransform<?,?,?>> rootTransforms, - Map<PValue,Collection<AppliedPTransform<?,?,?>>> consumers)` - Creates a new `InMemoryWatermarkManager`. -
`Map<AppliedPTransform<?,?,?>,Map<org.apache.beam.runners.direct.StructuralKey<?>,InMemoryWatermarkManager.FiredTimers>>`	`extractFiredTimers()` - Returns a map of each `PTransform` that has pending timers to those timers. -
`Set<AppliedPTransform<?,?,?>>`	`getCompletedTransforms()`
`InMemoryWatermarkManager.TransformWatermarks`	`getWatermarks(AppliedPTransform<?,?,?> transform)` - Gets the input and output watermarks for an `AppliedPTransform`. -
`void`	`updateWatermarks(InProcessPipelineRunner.CommittedBundle<?> completed, - InMemoryWatermarkManager.TimerUpdate timerUpdate, - org.apache.beam.runners.direct.CommittedResult result, - org.joda.time.Instant earliestHold)` - Updates the watermarks of a transform with one or more inputs. -

- - -
Methods inherited from class java.lang.Object
-clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- -
- - - -
  Method Detail
  - - - -
  - -
    create
    -
```
public static InMemoryWatermarkManager create(Clock clock,
-                                              Collection<AppliedPTransform<?,?,?>> rootTransforms,
-                                              Map<PValue,Collection<AppliedPTransform<?,?,?>>> consumers)
```
    -
    Creates a new InMemoryWatermarkManager. All watermarks within the newly created - InMemoryWatermarkManager start at BoundedWindow.TIMESTAMP_MIN_VALUE, the - minimum watermark, with no watermark holds or pending elements.
    -
    -
    Parameters:
    -
    rootTransforms - the root-level transforms of the Pipeline
    -
    consumers - a mapping between each PCollection in the Pipeline to the - transforms that consume it as a part of their input
    -
    -
  - - - -
  - -
    getWatermarks
    -
```
public InMemoryWatermarkManager.TransformWatermarks getWatermarks(AppliedPTransform<?,?,?> transform)
```
    -
    Gets the input and output watermarks for an AppliedPTransform. If the - PTransform has not processed any elements, return a watermark of - BoundedWindow.TIMESTAMP_MIN_VALUE.
    -
    -
    Returns:
    -
    a snapshot of the input watermark and output watermark for the provided transform
    -
    -
  - - - -
  - -
    updateWatermarks
    -
```
public void updateWatermarks(@Nullable
-                             InProcessPipelineRunner.CommittedBundle<?> completed,
-                             InMemoryWatermarkManager.TimerUpdate timerUpdate,
-                             org.apache.beam.runners.direct.CommittedResult result,
-                             org.joda.time.Instant earliestHold)
```
    -
    Updates the watermarks of a transform with one or more inputs. - -
    Each transform has two monotonically increasing watermarks: the input watermark, which can, - at any time, be updated to equal: -
```
- MAX(CurrentInputWatermark, MIN(PendingElements, InputPCollectionWatermarks))
- 
```
    - and the output watermark, which can, at any time, be updated to equal: -
```
- MAX(CurrentOutputWatermark, MIN(InputWatermark, WatermarkHolds))
- 
```
    .
    -
    -
    Parameters:
    -
    completed - the input that has completed
    -
    timerUpdate - the timers that were added, removed, and completed as part of producing - this update
    -
    result - the result that was produced by processing the input
    -
    earliestHold - the earliest watermark hold in the transform's state. null if there - is no hold
    -
    -
  - - - -
  - -
    extractFiredTimers
    -
```
public Map<AppliedPTransform<?,?,?>,Map<org.apache.beam.runners.direct.StructuralKey<?>,InMemoryWatermarkManager.FiredTimers>> extractFiredTimers()
```
    -
    Returns a map of each PTransform that has pending timers to those timers. All of the - pending timers will be removed from this InMemoryWatermarkManager.
    -
  - - - -
  - -
    getCompletedTransforms
    -
```
public Set<AppliedPTransform<?,?,?>> getCompletedTransforms()
```
    -
  -
-

java.lang.Object
-
- org.apache.beam.runners.direct.InProcessBundleOutputManager
-

-
-
All Implemented Interfaces:
-
DoFnRunners.OutputManager
-
-
-
-
```
public class InProcessBundleOutputManager
-extends Object
-implements DoFnRunners.OutputManager
```
-
An DoFnRunners.OutputManager that outputs to Bundles used by the - InProcessPipelineRunner.
-

- -

- - -
Constructor Summary
- - - - - - - - -
Constructors
Constructor and Description
InProcessBundleOutputManager(Map<TupleTag<?>,InProcessPipelineRunner.UncommittedBundle<?>> bundles)
-

Constructors
Constructor and Description
`InProcessBundleOutputManager(Map<TupleTag<?>,InProcessPipelineRunner.UncommittedBundle<?>> bundles)`

- -

- - -

Method Summary

- - - - - - - - - - - - - - -

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`static InProcessBundleOutputManager`	`create(Map<TupleTag<?>,InProcessPipelineRunner.UncommittedBundle<?>> outputBundles)`
`<T> void`	`output(TupleTag<T> tag, - WindowedValue<T> output)` - Outputs a single element to the receiver indicated by the given `TupleTag`. -

- - -
Methods inherited from class java.lang.Object
-clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- -

- - -

Constructor Detail

- - - -

InProcessBundleOutputManager

public InProcessBundleOutputManager(Map<TupleTag<?>,InProcessPipelineRunner.UncommittedBundle<?>> bundles)

- -

- - -

Method Detail

- - - -

create

public static InProcessBundleOutputManager create(Map<TupleTag<?>,InProcessPipelineRunner.UncommittedBundle<?>> outputBundles)

- - - -

-
output
-
```
public <T> void output(TupleTag<T> tag,
-                       WindowedValue<T> output)
```
-
Description copied from interface: DoFnRunners.OutputManager
-
Outputs a single element to the receiver indicated by the given TupleTag.
-
-
Specified by:
-
output in interface DoFnRunners.OutputManager
-
-

-
-
All Superinterfaces:
-
ApplicationNameOptions, HasDisplayData, PipelineOptions
-
-
-
-
```
public interface InProcessPipelineOptions
-extends PipelineOptions, ApplicationNameOptions
```
-
Options that can be used to configure the InProcessPipelineRunner.
-

- -

- - -
Nested Class Summary
-
- - - -
  Nested classes/interfaces inherited from interface org.apache.beam.sdk.options.PipelineOptions
  -PipelineOptions.CheckEnabled
-

- -

- - -

Method Summary

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

All Methods Instance Methods Abstract Methods
Modifier and Type	Method and Description
`Clock`	`getClock()` - Gets the `Clock` used by this pipeline. -
`ExecutorServiceFactory`	`getExecutorServiceFactory()` - Gets the `ExecutorServiceFactory` to use to create instances of `ExecutorService` - to execute `PTransforms`. -
`boolean`	`isBlockOnRun()`
`boolean`	`isShutdownUnboundedProducersWithMaxWatermark()`
`boolean`	`isTestImmutability()`
`void`	`setBlockOnRun(boolean b)`
`void`	`setClock(Clock clock)`
`void`	`setExecutorServiceFactory(ExecutorServiceFactory executorService)`
`void`	`setShutdownUnboundedProducersWithMaxWatermark(boolean shutdown)`
`void`	`setTestImmutability(boolean test)`

- - -
Methods inherited from interface org.apache.beam.sdk.options.ApplicationNameOptions
-getAppName, setAppName

- - -
Methods inherited from interface org.apache.beam.sdk.options.PipelineOptions
-as, cloneAs, getRunner, getStableUniqueNames, getTempLocation, setRunner, setStableUniqueNames, setTempLocation

- - -
Methods inherited from interface org.apache.beam.sdk.transforms.display.HasDisplayData
-populateDisplayData

- -
- - - -
  Method Detail
  - - - -
  - -
    getExecutorServiceFactory
    -
```
@Validation.Required
- @Hidden
- @Default.InstanceFactory(value=org.apache.beam.runners.direct.FixedThreadPoolExecutorServiceFactory.class)
-ExecutorServiceFactory getExecutorServiceFactory()
```
    -
    Gets the ExecutorServiceFactory to use to create instances of ExecutorService - to execute PTransforms. - -
    Note that ExecutorServices returned by the factory must ensure that - it cannot enter a state in which it will not schedule additional pending work unless currently - scheduled work completes, as this may cause the Pipeline to cease processing. - -
    Defaults to a FixedThreadPoolExecutorServiceFactory, which produces instances of - Executors.newCachedThreadPool().
    -
  - - - -
  - -
    setExecutorServiceFactory
    -
```
void setExecutorServiceFactory(ExecutorServiceFactory executorService)
```
    -
  - - - -
  - -
    getClock
    -
```
@Default.InstanceFactory(value=NanosOffsetClock.Factory.class)
- @Validation.Required
- @Hidden
-Clock getClock()
```
    -
    Gets the Clock used by this pipeline. The clock is used in place of accessing the - system time when time values are required by the evaluator.
    -
  - - - -
  - -
    setClock
    -
```
void setClock(Clock clock)
```
    -
  - - - -
  - -
    isShutdownUnboundedProducersWithMaxWatermark
    -
```
@Default.Boolean(value=false)
-boolean isShutdownUnboundedProducersWithMaxWatermark()
```
    -
  - - - -
  - -
    setShutdownUnboundedProducersWithMaxWatermark
    -
```
void setShutdownUnboundedProducersWithMaxWatermark(boolean shutdown)
```
    -
  - - - -
  - -
    isBlockOnRun
    -
```
@Default.Boolean(value=true)
-boolean isBlockOnRun()
```
    -
  - - - -
  - -
    setBlockOnRun
    -
```
void setBlockOnRun(boolean b)
```
    -
  - - - -
  - -
    isTestImmutability
    -
```
@Default.Boolean(value=true)
-boolean isTestImmutability()
```
    -
  - - - -
  - -
    setTestImmutability
    -
```
void setTestImmutability(boolean test)
```
    -
  -
-

Class InMemoryWatermarkManager

Nested Class Summary

Method Summary

Methods inherited from class java.lang.Object

Method Detail

create

getWatermarks

updateWatermarks

extractFiredTimers

getCompletedTransforms

Class InProcessBundleOutputManager

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Constructor Detail

InProcessBundleOutputManager

Method Detail

create

output

Interface InProcessPipelineOptions

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.beam.sdk.options.PipelineOptions

Method Summary

Methods inherited from interface org.apache.beam.sdk.options.ApplicationNameOptions

Methods inherited from interface org.apache.beam.sdk.options.PipelineOptions

Methods inherited from interface org.apache.beam.sdk.transforms.display.HasDisplayData

Method Detail

getExecutorServiceFactory

setExecutorServiceFactory

getClock

setClock

isShutdownUnboundedProducersWithMaxWatermark

setShutdownUnboundedProducersWithMaxWatermark

isBlockOnRun

setBlockOnRun

isTestImmutability

setTestImmutability