beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amit Sela (JIRA)" <j...@apache.org>
Subject [jira] [Created] (BEAM-542) Spark batch interval should be a configuration instead of an interpretation of the Pipeline's windows
Date Wed, 10 Aug 2016 10:26:20 GMT
Amit Sela created BEAM-542:
------------------------------

             Summary: Spark batch interval should be a configuration instead of an interpretation
of the Pipeline's windows
                 Key: BEAM-542
                 URL: https://issues.apache.org/jira/browse/BEAM-542
             Project: Beam
          Issue Type: Bug
          Components: runner-spark
            Reporter: Amit Sela
            Assignee: Amit Sela


Currently, the SparkRunner extracts the batch interval from the duration of the first window.

This is wrong in several ways:
# GlobalWindow pipelines
# It's an engine specific property and should not be expressed as a part of the logic but
rather as a configuration for execution of the pipeline.
# Effectively forces the definition of Fixed/SlidingWindows even when they are not needed
(stateless processing), which also makes the pipeline code not portable.    



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message