beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aviem Zur (JIRA)" <j...@apache.org>
Subject [jira] [Created] (BEAM-2847) Stop streaming pipeline when global watermark reaches +INF
Date Wed, 06 Sep 2017 13:36:00 GMT
Aviem Zur created BEAM-2847:
-------------------------------

             Summary: Stop streaming pipeline when global watermark reaches +INF 
                 Key: BEAM-2847
                 URL: https://issues.apache.org/jira/browse/BEAM-2847
             Project: Beam
          Issue Type: Improvement
          Components: runner-spark
            Reporter: Aviem Zur
            Assignee: Amit Sela


Stop streaming pipeline when global watermark reaches +INF as suggested in BEAM-849 comments.

We can do this similarly to the solution in [TestSparkRunner#awaitWatermarksOrTimeout|https://github.com/apache/beam/blob/v2.1.0/runners/spark/src/main/java/org/apache/beam/runners/spark/TestSparkRunner.java#L158-L179]
by polling {code}
SparkTimerInternals.global(GlobalWatermarkHolder.get(...))
{code}

We can also remove this code from {{TestSparkRunner}} if we make sure all sources in test
behave in a way which shifts the watermark to +INF at the end of the test.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message