kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From William Yu <...@unified.com>
Subject Message polling behavior when subscribed to a topic pattern
Date Thu, 11 May 2017 18:42:46 GMT
Hi,

I'm trying to understand the behavior of consumer poll function when
subscribed to multiple topics using a RegEx topic pattern. I was under the
assumption the poll function would pull messages from each of the topics I
was subscribed to, but from a test program I wrote it looks like it will
pull from a given topic until it has been drained.

Sample program:
https://gist.github.com/wiyu/afb860aacd0382f84601a3512a21e9d1

Test :
- 2 Topics: foo and bar
- published 10 msgs to each topic.

Output:
records to process: 10
[foo] partition = 2, offset = 83, value = 4
[foo] partition = 2, offset = 84, value = 8
[foo] partition = 3, offset = 82, value = 2
[foo] partition = 3, offset = 83, value = 6
[foo] partition = 3, offset = 84, value = 10
[foo] partition = 0, offset = 111, value = 3
[foo] partition = 0, offset = 112, value = 7
[foo] partition = 1, offset = 104, value = 1
[foo] partition = 1, offset = 105, value = 5
[foo] partition = 1, offset = 106, value = 9
committed: 10
records to process: 10
[bar] partition = 1, offset = 80, value = 2
[bar] partition = 1, offset = 81, value = 4
[bar] partition = 1, offset = 82, value = 6
[bar] partition = 1, offset = 83, value = 8
[bar] partition = 1, offset = 84, value = 10
[bar] partition = 0, offset = 80, value = 1
[bar] partition = 0, offset = 81, value = 3
[bar] partition = 0, offset = 82, value = 5
[bar] partition = 0, offset = 83, value = 7
[bar] partition = 0, offset = 84, value = 9

Thanks,
William

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message