kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: consumer may lose data
Date Tue, 12 Jun 2012 16:44:55 GMT
Corbin,

This indicates a bug in Kafka. Which version of Kafka are you using? Is
there an easy way to reproduce this problem? Your usage of Kafka consumer
seems to be normal.

Thanks,

Jun

On Tue, Jun 12, 2012 at 9:14 AM, Corbin Hoenes <corbin@tynt.com> wrote:

> Hi there this is my first experience w/kafka.  We've deployed it in
> production (soft release) and using it to create a realtime stream of
> data--we love it so far.
>
> Running in production we are seeing these types of messages every once in
> a while:
>
> [2012-06-09 09:02:36,051] ERROR [pool-1-thread-4] (ConsumerIterator.scala
> 74) - consumed offset: 22013667532 doesn't match fetch offset: 21008498593
> for firehose:1-23: fetched offset = 22013667532: consumed offset =
> 22013667532;
>  Consumer may lose data
> [2012-06-09 09:22:48,520] ERROR [pool-1-thread-4] (ConsumerIterator.scala
> 74) - consumed offset: 21013192930 doesn't match fetch offset: 21475567914
> for firehose:1-3: fetched offset = 22021419503: consumed offset =
> 21013192930;
>  Consumer may lose data
> [2012-06-09 09:42:34,342] ERROR [pool-1-thread-1] (ConsumerIterator.scala
> 74) - consumed offset: 21017992042 doesn't match fetch offset: 21477363255
> for firehose:1-5: fetched offset = 22029075985: consumed offset =
> 21017992042;
>  Consumer may lose data
> [2012-06-09 09:46:50,599] ERROR [pool-1-thread-1] (ConsumerIterator.scala
> 74) - consumed offset: 21017498912 doesn't match fetch offset: 21476883323
> for firehose:1-7: fetched offset = 22022716494: consumed offset =
> 21017498912;
>  Consumer may lose data
> [2012-06-09 09:50:54,912] ERROR [pool-1-thread-1] (ConsumerIterator.scala
> 74) - consumed offset: 21016428723 doesn't match fetch offset: 21475750245
> for firehose:1-4: fetched offset = 22027573299: consumed offset =
> 21016428723;
>  Consumer may lose data
> [2012-06-09 09:58:29,709] ERROR [pool-1-thread-1] (ConsumerIterator.scala
> 74) - consumed offset: 21017643906 doesn't match fetch offset: 21477006308
> for firehose:1-6: fetched offset = 22025778964: consumed offset =
> 21017643906;
>  Consumer may lose data
> [2012-06-09 09:59:04,622] ERROR [pool-1-thread-4] (ConsumerIterator.scala
> 74) - consumed offset: 21017419393 doesn't match fetch offset: 21476749439
> for firehose:1-23: fetched offset = 22025584690: consumed offset =
> 21017419393;
>
> I am a bit unsure what kafka does when the consumed offset doesn't match
> the fetch offset.  We are using a pool of threads to consume each stream
> created by ConsumerConnector.createMessageStreams().  Is this kosher?
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message