kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: Contiguous Offsets on non-compacted topics
Date Tue, 23 Jan 2018 21:20:33 GMT
In general offsets should be consecutive. However, this is no "official
guarantee" and you should not build application that rely on consecutive
offsets.

Also note, with Kafka 0.11 and transactions, commit/abort markers
require on offset in the partitions and thus, having "offset gaps" is
normal for this case.

Not sure atm, why you have a "offset gap" as your 0.9 log format does
not support transactions.


-Matthias


On 1/23/18 9:52 AM, Justin Miller wrote:
> Greetings, 
> 
> We’ve seen a strange situation where-in the topic is not compacted but the offset numbers
inside the partition (#93) are not contiguous. This only happens once a day though, on a topic
with billions of messages per day.
> 
> next offset = 1786997223
> next offset = 1786997224
> next offset = 1786997226
> next offset = 1786997227
> next offset = 1786997228
> 
> I was wondering if this still holds with Kafka 0.10, 0.11, 1.0:	http://grokbase.com/t/kafka/users/12bpnexg1m/dumb-question-about-offsets
<http://grokbase.com/t/kafka/users/12bpnexg1m/dumb-question-about-offsets>
> 
> Specifically: “In Kafka 0.8, each message is assigned a monotonically increasing, contiguous
sequence number per partition,starting with 1.”
> 
> We’re on Kafka 1.0 with logs at version 0.9.0.0.
> 
> Thanks!
> Justin
> 


Mime
View raw message