kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: Consume more than produce
Date Fri, 01 Aug 2014 15:07:30 GMT
What is the ack value used in the producer?


On Fri, Aug 1, 2014 at 1:28 AM, Guy Doulberg <Guy.Doulberg@perion.com>
wrote:

> Hey,
>
>
> After a year or so I have Kafka as my streaming layer in my production, I
> decided it is time to audit, and to test how many events do I lose, if I
> lose events at all.
>
>
> I discovered something interesting which I can't explain.
>
>
> The producer produces less events that the consumer group consumes.
>
>
> It is not much more, it is about 0.1% more events
>
>
> I use the Consumer API (not the simple consumer API)
>
>
> I was thinking I might had rebalancing going on in my system, but it
> doesn't look like that.
>
>
> Did anyone see such a behaviour
>
>
> In order to audit, I calculated for each event the minute it arrived, and
> assigned this value to the event, I used statsd do to count all events from
> all my producer cluster, and all consumer group cluster.
>
>
> I must say that it is not a happening for every minute,
>
>
> Thanks, Guy
>
>
>


-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message