kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Liam Clarke-Hutchinson <liam.cla...@adscale.co.nz>
Subject Re: reliable way to count number of messages
Date Mon, 08 Jun 2020 10:26:51 GMT
Hi Fabio,

-1 is shorthand for latest when passed as --time to GetOffsetShell (-2 is
earliest), so the output is telling you that the latest offset of partition
0 of the topic is 47252.

However, the earliest offset in the topic may not be zero - as topic
retention times are hit and messages removed, offsets aren't changed.

So likely you'll find the earliest offset is 23626 or similar if you run
GetOffsetShell with --time -2.

Cheers,

Liam Clarke-Hutchinson

On Mon, 8 Jun. 2020, 8:42 pm Fabio Pardi, <f.pardi@portavita.eu> wrote:

> Hi there,
>
> I have one topic with one partition and i want to know how many messages
> are there in the topic.
>
> I noticed that if i run:
>
> kafka-console-consumer --topic mytopic  --bootstrap-server [..]:9092
> --from-beginning
>
> [..]
> Processed a total of 23626 messages
>
>
> If I instead run:
>
>  kafka.tools.GetOffsetShell --broker-list [..]:9092 --topic mytopic --time
> -1
>
> mytopic:0:47252
>
>
> So the 2 commands return different numbers and the first returns exactly
> half the amount the second does.
>
> Why the 2 commands do not return the same amount and which one is right?
>
>
> kafka-console-consumer --version
> 5.4.1-ccs (Commit:fd1e543386b47352)
>
> kafka-run-class -version
> openjdk version "1.8.0_212"
> OpenJDK Runtime Environment (Zulu 8.38.0.13-CA-linux64) (build
> 1.8.0_212-b04)
> OpenJDK 64-Bit Server VM (Zulu 8.38.0.13-CA-linux64) (build 25.212-b04,
> mixed mode)
>
> regards,
>
> fabio pardi
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message