http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/8797d044/content/learn/runners/capability-matrix/index.html ---------------------------------------------------------------------- diff --git a/content/learn/runners/capability-matrix/index.html b/content/learn/runners/capability-matrix/index.html new file mode 100644 index 0000000..28ed044 --- /dev/null +++ b/content/learn/runners/capability-matrix/index.html @@ -0,0 +1,1716 @@ + + + + + + + + + Apache Beam Capability Matrix + + + + + + + + + + + + + + + + + + + + + +
+ +
+

Beam Capability Matrix

+

Last updated: 2016-08-02 10:42 PDT

+ +

Apache Beam (incubating) provides a portable API layer for building sophisticated data-parallel processing engines that may be executed across a diversity of exeuction engines, or runners. The core concepts of this layer are based upon the Beam Model (formerly referred to as the Dataflow Model), and implemented to varying degrees in each Beam runner. To help clarify the capabilities of individual runners, we’ve created the capability matrix below.

+ +

Individual capabilities have been grouped by their corresponding What / Where / When / How question:

+ + + +

For more details on the What / Where / When / How breakdown of concepts, we recommend reading through the Streaming 102 post on O’Reilly Radar.

+ +

Note that in the future, we intend to add additional tables beyond the current set, for things like runtime characterstics (e.g. at-least-once vs exactly-once), performance, etc.

+ + + +
+ + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Beam ModelGoogle Cloud DataflowApache FlinkApache Spark
ParDo
+
+
+
+
GroupByKey
+
+
+
~
+
Flatten
+
+
+
+
Combine
+
+
+
+
Composite Transforms
+
~
+
~
+
~
+
Side Inputs
+
+
~ (BEAM-102)
+
~
+
Source API
+
+
+
+
Aggregators
~
+
~
+
~
+
~
+
Keyed State
✕ (BEAM-25)
+
+
+
+
Beam ModelGoogle Cloud DataflowApache FlinkApache Spark
Global windows
+
+
+
+
Fixed windows
+
+
+
~
+
Sliding windows
+
+
+
~
+
Session windows
+
+
+
+
Custom windows
+
+
+
+
Custom merging windows
+
+
+
+
Timestamp control
+
+
+
+
Beam ModelGoogle Cloud DataflowApache FlinkApache Spark
Configurable triggering
+
+
+
+
Event-time triggers
+
+
+
+
Processing-time triggers
+
+
+
+
Count triggers
+
+
+
+
[Meta]data driven triggers
✕ (BEAM-101)
+
+
+
+
Composite triggers
+
+
+
+
Allowed lateness
+
+
+
+
Timers
✕ (BEAM-27)
+
+
+
+
Beam ModelGoogle Cloud DataflowApache FlinkApache Spark
Discarding
+
+
+
+
Accumulating
+
+
+
+
Accumulating & Retracting
✕ (BEAM-91)
+
+
+
+
+
+ + + + + + + + + + + +
+ +
+ + +
+
+
+ +
+
+ +
+ + + + + http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/8797d044/content/learn/runners/dataflow/index.html ---------------------------------------------------------------------- diff --git a/content/learn/runners/dataflow/index.html b/content/learn/runners/dataflow/index.html new file mode 100644 index 0000000..a45e677 --- /dev/null +++ b/content/learn/runners/dataflow/index.html @@ -0,0 +1,153 @@ + + + + + + + + + Cloud Dataflow Runner + + + + + + + + + + + + + + + + + + + + + +
+ +
+

Using the Cloud Dataflow Runner

+ +

This page is under construction (BEAM-508).

+ +
+ + +
+
+
+ +
+
+ +
+ + + + + http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/8797d044/content/learn/runners/direct/index.html ---------------------------------------------------------------------- diff --git a/content/learn/runners/direct/index.html b/content/learn/runners/direct/index.html new file mode 100644 index 0000000..8a8ab40 --- /dev/null +++ b/content/learn/runners/direct/index.html @@ -0,0 +1,153 @@ + + + + + + + + + Apache Direct Runner + + + + + + + + + + + + + + + + + + + + + +
+ +
+

Using the Direct Runner

+ +

This page is under construction (BEAM-505).

+ +
+ + +
+
+
+ +
+
+ +
+ + + + + http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/8797d044/content/learn/runners/flink/index.html ---------------------------------------------------------------------- diff --git a/content/learn/runners/flink/index.html b/content/learn/runners/flink/index.html new file mode 100644 index 0000000..ad28860 --- /dev/null +++ b/content/learn/runners/flink/index.html @@ -0,0 +1,153 @@ + + + + + + + + + Apache Flink Runner + + + + + + + + + + + + + + + + + + + + + +
+ +
+

Using the Apache Flink Runner

+ +

This page is under construction (BEAM-506).

+ +
+ + +
+
+
+ +
+
+ +
+ + + + +