flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Merging N parallel/partitioned WindowedStreams together, one-to-one, into a global window stream
Date Thu, 06 Oct 2016 20:50:00 GMT
Maybe this can be done by assigning the same window id to each of the N
local windows, and do a

.keyBy(windowId)
.countWindow(N)

This should create a new global window for each window id and collect all N
windows.

Best, Fabian

2016-10-06 22:39 GMT+02:00 AJ Heller <aj@drfloob.com>:

> The goal is:
>  * to split data, random-uniformly, across N nodes,
>  * window the data identically on each node,
>  * transform the windows locally on each node, and
>  * merge the N parallel windows into a global window stream, such that one
> window from each parallel process is merged into a "global window" aggregate
>
> I've achieved all but the last bullet point, merging one window from each
> partition into a globally-aggregated window output stream.
>
> To be clear, a rolling reduce won't work because it would aggregate over
> all previous windows in all partitioned streams, and I only need to
> aggregate over one window from each partition at a time.
>
> Similarly for a fold.
>
> The closest I have found is ParallelMerge for ConnectedStreams, but I have
> not found a way to apply it to this problem. Can flink achieve this? If so,
> I'd greatly appreciate a point in the right direction.
>
> Cheers,
> -aj
>

Mime
View raw message