flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chengzhi Zhao <w.zhaocheng...@gmail.com>
Subject Multiple Streams Connect Watermark
Date Wed, 25 Apr 2018 18:20:26 GMT
Hi, everyone,

I am trying to do some join-like pipeline using flink connect operator and
CoProcessFunction, I have use case that I need to connect 3+ streams. So I
am having something like this:

A
    ===> C
B                 ==> E
              D

So two streams A and B connect at first with 3 hours late on low watermark,
after data has been emitted (the output C stream), a new stream D connect
to C and emitted E as final output. I was wondering how the downstream
watermark should be defined. Should I give C stream a new watermark for 3
hours delay again? or when I connect stream D, everything will be 6 hours
late on low watermark.

I am using BoundedOutOfOrdernessGenerator[1] with maxOutOfOrderness 3 hours

Thanks for your tips and help in advance.

Best,
Chengzhi

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/event_timestamps_watermarks.html#with-periodic-watermarks

Mime
View raw message