kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: KStreams.reduceByKey passing nulls to my Deserializer?
Date Fri, 10 Jun 2016 22:47:21 GMT
Hello Avi,

Yes, this is possible: although we checked nullable keys when doing reduce
/ aggregations:

https://github.com/apache/kafka/blob/0.10.0/streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamReduce.java#L67

We do not check if the there are any values returned from the underlying
state store, and when we cannot find such a match, null is returned to
deserializer.


I think in general the library should guard this case instead of letting
desers worry about it. Do you want to file a JIRA reporting this bug so we
can follow-up?


Guozhang

On Fri, Jun 10, 2016 at 11:27 AM, Avi Flax <avi.flax@parkassist.com> wrote:

>
> > On Jun 10, 2016, at 14:24, Avi Flax <avi.flax@parkassist.com> wrote:
> >
> > Hi, I’m using Kafka Streams (0.10.0) with JRuby, most of my
> scripts/nodes are working well at this point, except for one which is using
> reduceByKey.
>
> Whoops, I should probably share my code as well!
>
> Here’s the topology:
>
>
> builder.stream(key_serde, val_serde, 'visit-update-events')
>        .reduceByKey(-> (a, b) { a.merge b }, key_serde, val_serde,
> 'intermediate-visits')
>        .to(key_serde, val_serde, 'visits’)
>
>
> This is using Ruby syntax, but hopefully it’s fairly readable. I’ve added
> it to this gist as well:
> https://gist.github.com/aviflax/3428cdbaa18aca9bf0a958c6d5eac2bd
>
> Thanks!
> Avi




-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message