kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Damian Guy <damian....@gmail.com>
Subject Re: what is the key used for change log topic backed by windowed store
Date Fri, 23 Dec 2016 17:29:48 GMT
Hi Sachin,

The key is a combination of the record key + window start time + a sequence
number. The timestamp is 8 bytes and the sequence number is 4 bytes.

Thanks,
Damian

On Thu, 22 Dec 2016 at 15:26 Sachin Mittal <sjmittal@gmail.com> wrote:

> Hi All,
> Our stream is something like
>
> builder.stream()
>         .groupByKey()
>         .aggregate(Initializer, Aggregator, TimeWindows, valueSerde,
> "table-name')
>
> So This creates a changelog topic.
> I was wondering what would be the key used for this topic.
>
> Would it be they key we use to group by or a compounded key of (our key,
> window key).
>
> I am asking this because what is observed that when stream runs for few
> days we start getting the exception where message size is greater than the
> allowable message size.
>
> However our messages are windowed on hourly windows, so size of a message
> should be far less than the max message size.
>
> Thanks
> Sachin
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message