flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Dimiduk <ndimi...@gmail.com>
Subject Error handling
Date Wed, 11 Nov 2015 20:49:14 GMT

I don't see a section in the online manual dedicated to this topic, so I
want to raise the question here: How should errors be handled? Specifically
I'm thinking about streaming jobs, which are expected to "never go down".
For example, errors can be raised at the point where objects are serialized
to/from sources/sinks, and UDFs. Cascading provides failure traps [0] where
erroneous tuples are saved off for post-processing. Is there any such
functionality in Flink?

I started down the road of implementing a Maybe/Optional type, a POJO
Generic triple of <IN, OUT, Throwable> for capturing errors at each stage
of a pipeline. However, Java type erasure means even though it compiles,
the job is rejected at submission time.

How are other people handling errors in their stream processing?


[0]: http://docs.cascading.org/cascading/1.2/userguide/html/ch06s03.html

View raw message