On 19 April 2018 at 2:00:51 AM, Elias Levy (firstname.lastname@example.org) wrote:
Either proposal would work. In the later case, at a minimum we'd need a way to identify the source within the metric. The basic error metric would then allow us to go into the logs to determine the cause of the error, as we already record the message causing trouble in the log.On Mon, Apr 16, 2018 at 4:42 AM, Fabian Hueske <email@example.com> wrote:
In that case, log files might be a better approach.I see that passing information such as topic name, partition, and offset are important debugging information. However, I don't think that metrics would be good to capture them.In both cases, we need to touch the source connectors.2) Check for null returns in the source functions and increment a respective counter.Thanks for starting the discussion Elias.I see two ways to address this issue.
1) Add an interface that a deserialization schema can implement to register metrics. Each source would need to check for the interface and call it to setup metrics.
I'm not sure to what extend the source functions (Kafka, Kinesis) support such error tracking.Adding Gordon to the thread who knows the internals of the connectors.