kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: Streams - Manage Json SerDe error
Date Tue, 05 Jul 2016 21:40:20 GMT
Hello Gaspar,

In your case, a single topic can have messages in different format, and my
guess is that they usually have different semantics (e.g. one format for
data record, and another format for control message / error log / etc).

In this case, I'd suggest similar solutions as you mentioned, to either
"filter" out the non-related messages or "branch" this topic into multiple
streams based on their data format with a String-with-manual-Json serde
first, before doing the actual processing. Kafka Streams has the
corresponding filter / branch operators in the high-level DSL for these
purposes.

Guozhang


On Mon, Jul 4, 2016 at 1:46 AM, Gaspar Muñoz <gmunoz@stratio.com> wrote:

> Hi there,
>
> my question is about Kafka Streams. I'm writting an application using
> Streams. I read JSON from Kafka topic and I make some transformations.
>
> I'm using
>
> Serde<JsonNode> jsonNodeSerder = Serdes.serdeFrom(new JsonSerializer(), new
> JsonDeserializer());
> KStream<String, JsonNode> kStream = builder.stream(Serdes.String(),
> jsonNodeSerder, topic);
>
> If, unluckily, we receive a non-json message, JsonDeserializer launch a
>
> throw new SerializationException("Error serializing JSON message", e);
>
> and streams threads die so the entire application dies. In order to make
> robust my application  I have to skip this errors and send the error
> message to another topic to be analyzed.
>
> Currently I've implemented MyOwnJSONDeserializer extends Kafka
> JsonDeserializer and I catch and manage the exception but this don't seems
> really clean to me.
>
> How would you manage this errors? I thought I might have used String SerDe
> and "manually" convert to JsonNode, is there any better solution?
>
> Thanks.
>



-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message