flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai" <tzuli...@apache.org>
Subject Re: Tracking deserialization errors
Date Thu, 19 Apr 2018 13:56:35 GMT
Sorry about that, that would be my mistake. I’ll close FLINK-9204 as a duplicate and leave
my thoughts on FLINK-9155. Thanks for pointing out!

On 19 April 2018 at 2:00:51 AM, Elias Levy (fearsome.lucidity@gmail.com) wrote:

Either proposal would work.  In the later case, at a minimum we'd need a way to identify
the source within the metric.  The basic error metric would then allow us to go into the
logs to determine the cause of the error, as we already record the message causing trouble
in the log. 

On Mon, Apr 16, 2018 at 4:42 AM, Fabian Hueske <fhueske@gmail.com> wrote:
Thanks for starting the discussion Elias.

I see two ways to address this issue.

1) Add an interface that a deserialization schema can implement to register metrics. Each
source would need to check for the interface and call it to setup metrics.
2) Check for null returns in the source functions and increment a respective counter.

In both cases, we need to touch the source connectors.

I see that passing information such as topic name, partition, and offset are important debugging
information. However, I don't think that metrics would be good to capture them.
In that case, log files might be a better approach.

I'm not sure to what extend the source functions (Kafka, Kinesis) support such error tracking.
Adding Gordon to the thread who knows the internals of the connectors.

View raw message