kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Srikanth <srikanth...@gmail.com>
Subject Re: skipped-records-rate vs skippedDueToDeserializationError-rate metric in streams app
Date Sun, 28 Jan 2018 16:54:56 GMT
Guozhang,

While I understand that this metric is meaningless when handler is set to
FAIL, in my case I'm actually using LogAndContinueExceptionHandler.
In this case, app needs to report such occurrences. What I noticed is that
only skipped-records is set.
The granularity offered by skippedDueToDeserializationError is missing.

Srikanth

On Fri, Jan 26, 2018 at 10:45 PM, Guozhang Wang <wangguoz@gmail.com> wrote:

> Hi Srikanth,
>
> Looked at the source code once again and discussing with other committer I
> now remembered why we designed it that way: when you set the
> HandlerResponse to FAIL, it means that once a "poison record" is received,
> stop the world by throwing this exception all the way up. And hence at that
> time the application would be stopped anyways so we would not need to
> record this metric.
>
> So in other words, I think it is rather a documentation improvement that we
> should do.
>
>
> Guozhang
>
>
> On Fri, Jan 26, 2018 at 8:56 AM, Guozhang Wang <wangguoz@gmail.com> wrote:
>
> > Helo Srikanth,
> >
> > Thanks for reporting this, as I checked the code
> skippedDueToDeserializati
> > onError is effectively only recorded when the DeserializationHandlerResp
> > onse is not set to FAIL. I agree it is not exactly matching the
> > documentation's guidance, and will try to file a JIRA and fix it.
> >
> > As for skippedDueToDeserializationError and skipped-records-rate, there
> > is an open JIRA discussing about this: https://issues.apache.
> > org/jira/browse/KAFKA-6376, just FYI.
> >
> >
> > Could you share on which version of Kafka did you observe this issue?
> >
> > Guozhang
> >
> > On Fri, Jan 26, 2018 at 6:30 AM, Srikanth <srikanth.ht@gmail.com> wrote:
> >
> >> Hello,
> >>
> >> As per doc when LogAndContinueExceptionHandler is used it will set
> >> skippedDueToDeserializationError-rate metric to indicate
> deserialization
> >> error.
> >> I notice that it is never set. Instead skipped-records-rate is set. My
> >> understanding was that skipped-records-rate is set due to timestamp
> >> extraction errors.
> >>
> >> Ex, I sent a few invalid records to a topic and was able to see
> exception
> >> during deserialization.
> >>
> >> org.apache.kafka.common.errors.SerializationException: Error
> >> deserializing
> >> Avro message for id -1
> >> Caused by: org.apache.kafka.common.errors.SerializationException:
> Unknown
> >> magic byte!
> >> 18/01/24 06:50:09 WARN StreamThread: Exception caught during
> >> Deserialization, taskId: 0_0, topic: docker.event.1, partition: 0,
> offset:
> >> 3764
> >>
> >> These incremented skipped-records-[rate|total].
> >>
> >> Thanks,
> >> Srikanth
> >>
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
> -- Guozhang
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message