flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Filipe Couto <Filipe.Co...@i2s.pt>
Subject Watermark and multiple streams
Date Tue, 10 Apr 2018 16:32:58 GMT
Hello.

I'm joining several data streams, using ConnectedStreams. Let's say something like A connect
B which outputs AB, and then I join AB with C, which outputs ABC.

However, the relationship between A and B, or AB and C may be of 1 to many, or 1 to 1, depending
on the case. For the 1 to 1, it's expected to produce an output as soon as I obtain both records
that match the same key, but for the 1 to many, it's not so simple, there's no event that
guarantees that I have obtained all of my <many> records. For this, I've searched and
found out that I have to implement an onTimer method, in a Process Function. I also read that
an event timer is registered in a queue and when a new watermark arrives, the event timers
that <passed> will be triggered.

Given this, I'm running into trouble when generating new watermarks like in the example: https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/event_timestamps_watermarks.html#with-periodic-watermarks

The solution I've reached basically processes (on the onTimer method) all the records that
I obtained previously when the first watermark arrives, but after that, it stops triggering.

I read that watermarks are global, so how can I create a new watermark when working with several
streams?

Thank you

Mime
View raw message