kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 向河林 <xian...@mediav.com>
Subject Re: In what condition does getoffsetbefore (latest) not return latest offset?
Date Tue, 09 Dec 2014 08:31:58 GMT
Thanks Guozhang,

I checked the server log, and I am 90% sure that no leader movement
happened at that moment.

Could it be that the data was in the memory but yet not flushed to the disk?
I read the doc in the wiki
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Replication,
and said

The high watermark (HW) is the offset of the last committed message.
Each log is periodically synced to disks. Data before the flushed
offset is guaranteed to be persisted on disks. As we will see, the
flush offset can be before or after HW.

since the https://issues.apache.org/jira/browse/KAFKA-501 said

 For regular consumers, getOffset should return highWatermark as the
latest offset.

Thanks again.
​

On Tue, Dec 9, 2014 at 5:28 AM, Guozhang Wang <wangguoz@gmail.com> wrote:

> Helin,
>
> Is there a leader movement just before the get latest offset call? If your
> follower is not synced and it then becomes the leader due to some reason,
> it will not have the complete partition data.
>
> Guozhang
>
> On Mon, Dec 8, 2014 at 3:02 AM, Helin Xiang <xkeegan@gmail.com> wrote:
>
> > 1 additional information we found in the kafka’s application log since
> the
> > MAGIC time:
> >
> > 2014-12-04 09:59:36,726 [kafka-scheduler-2] INFO
> > kafka.cluster.Partition  - Partition [a.s.3,26] on broker 5: Shrinking
> > ISR for partition [a.s.3,26] from 5,4 to 5
> > 2014-12-04 09:59:36,728 [kafka-scheduler-2] ERROR kafka.utils.ZkUtils$
> >  - Conditional update of path
> > /brokers/topics/a.s.3/partitions/26/state with data
> > {"controller_epoch":2,"leader":5,"version":1,"leader_epoch":4,"isr":[5]}
> > and expected version 675 failed due to
> > org.apache.zookeeper.KeeperException$BadVersionException:
> > KeeperErrorCode = BadVersion for
> > /brokers/topics/a.s.3/partitions/26/state
> >
> > ​
> >
> > On Mon, Dec 8, 2014 at 6:59 PM, Helin Xiang <xkeegan@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > We have currently upgraded our kafka cluster from 0.7.2 to 0.8.1.1.
> > >
> > > In one of our application, we want to get all partitions' latest
> offsets,
> > > so we use getoffsetbefore java API (latest).
> > >
> > > We believe at some time, 1 of the partition's latest offset we got is
> > much
> > > smaller than its real latest offset,(we saw in the application's log
> that
> > > the partition's offset is much smaller than other partitions'). Since
> the
> > > real data file of that partition was already deleted, we cannot prove
> our
> > > guess, but we found some clue in the kafka's application log which
> helps
> > us
> > > to conclude that the partition's latest offset at that moment did have
> a
> > > much larger number.
> > >
> > > some additional useful information: the partition have 1 additional
> > > replica(follower), and at that time, it was not synced with the leader
> > > partition(far away behind the leader).
> > >
> > > Does any one have the same issue? In what condition could lead to this
> > > situation?
> > >
> > > Thanks.
> > >
> > > --
> > >
> > >
> > > *Best Regards向河林*
> > >
> >
> >
> >
> > --
> >
> >
> > *Best Regards向河林*
> >
>
>
>
> --
> -- Guozhang
>



-- 

*向河林*


*MV AD **聚效广告*       *上海* · 北京 · 广州 · 杭州
_______________________________________________________________

上海市闸北区天目中路585号新梅大厦4楼  200070

MOB:18621368491

TEL:021-52559088(分机:8133)

EMAIL:xianghl@mediav.com <yaoyj@mediav.com>

HTTP:www.mediav.com


-------------------------------CONFIDENTIAL ------------------------------
------------

本邮件载有秘密信息,请您恪守保密义务,勿向第三人透露。谢谢合作。

This email communication is confidential. Recipient(s) named above is(are)
obligated to maintain secrecy and is(are) not permitted to disclose the
contents of this communication to others. Thank you.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message