samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zach Cox <zcox...@gmail.com>
Subject Stream Progress
Date Tue, 12 Jan 2016 23:08:03 GMT
I'm curious - has anyone built any Samza-based systems that use any notion
of stream progress, e.g. low watermarks, punctuations, or heartbeats? These
are described in the stream-processing literature [1] [2] [3] and
implemented in MillWheel [4] and Dataflow [5] but I have not seen any
mention of these techniques related to Samza (except for briefly in
Samza-552 [6]).

The purpose of something like a low watermark would include handling
out-of-order events, outputting the result of a stateful operation after
all relevant events have been processed, and cleaning up internal state
that will never again be updated to avoid unbounded growth.

Just wondering if techniques like these would be useful in Samza job
pipelines, or if there are various approaches in Samza that make them
unnecessary.

Thanks,
Zach

[1] http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=1198390
[2] http://dl.acm.org/citation.cfm?id=1055596
[3] http://dl.acm.org/citation.cfm?id=1453890
[4] http://research.google.com/pubs/pub41378.html
[5] http://research.google.com/pubs/pub43864.html
[6] https://issues.apache.org/jira/browse/SAMZA-552

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message