kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Kreps <jay.kr...@gmail.com>
Subject Re: watermark location
Date Fri, 15 Jun 2012 16:27:54 GMT
There may be some confusion of terminology
  hw mark == highest offset the server will give to consumers, based on
what is sync'd to disk (0.7) or replicated to multiple machines (0.8)
  consumer offset == consumer's position in the stream (always less than hw
mark, by definition)

As Jun says in 0.7 we take the HW mark from the log and cache it in memory.

The consumer offset is stored in zookeeper.

Hope that helped clarify.

-Jay

On Fri, Jun 15, 2012 at 7:12 AM, Jun Rao <junrao@gmail.com> wrote:

> In 0.7, HW is only kept in memory in the broker. In 0.8, it will be
> checkpointed to disk too.
>
> Thanks,
>
> Jun
>
> On Fri, Jun 15, 2012 at 3:51 AM, Raymond Ng <raymondair@gmail.com> wrote:
>
> > Hi
> >
> > I'm trying to locate where the watermark is written to, is it on the
> kafka
> > server / zookeeper or where kafka spouts are deployed?
> >
> > --
> > Rgds
> > Ray
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message