kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: Offsets/Lags for global state stores not shown
Date Tue, 06 Nov 2018 19:07:28 GMT
The topics of global stores are not included by design.

The "problem" is, that each instance needs to consume *all*
topic-partitions from and thus topis, we thus they cannot be include
into the consumer group that would assign each partition to exactly one
instance. Hence, an additional consumer is used that uses partition
assignment (instead of subscription) and this consumer does not commit
any offset to Kafka.

Note that global stores are bootstrapped before processing begins
though, and are expected to be low throughput topic anyway.


-Matthias

On 11/6/18 2:03 AM, Patrik Kleindl wrote:
> Hello
> 
> Am I doing something wrong or is it by design that global state stores and
> their consumers do not show up under the consumer-groups?
> With the consumer group command (and in control center as well) I don't get
> any output for the group:
> ./kafka-consumer-groups --bootstrap-server broker:9092 --group somegroup
> --describe
> Note: This will not show information about old Zookeeper-based consumers.
> 
> If I query for the state I get a response that members are present:
> ./kafka-consumer-groups --bootstrap-server broker:9092 --group somegroup
> --describe --state
> Note: This will not show information about old Zookeeper-based consumers.
> 
> COORDINATOR (ID)                 ASSIGNMENT-STRATEGY
> STATE                #MEMBERS
> broker:9092 (1) stream                    Stable               2
> 
> This is quite irritating as we cannot see if a global state store has
> caught up with a backlog of messages.
> 
> Code to reproduce:
>         builder.globalTable(TOPIC_NAME, Materialized
>                 .<String, String, KeyValueStore<Bytes, byte[]>>as(STORENAME)
>                 .withKeySerde(Serdes.String())
>                 .withValueSerde(Serdes.String()));
> 
> Nothing fancy.
> 
> Logs:
> 2018-11-05 21:25:56 INFO  AbstractCoordinator:442 - (Re-)joining group
> 2018-11-05 21:25:56 INFO  StreamPartitionAssignor:481 - Assigned tasks to
> clients as {e0250aa5-e1c6-4d33-a746-bc9357c66965=[activeTasks: ([])
> standbyTasks: ([]) assignedTasks: ([]) prevActiveTasks: ([])
> prevAssignedTasks: ([]) capacity: 1]}.
> 2018-11-05 21:25:56 WARN  ConsumerCoordinator:376 - The following
> subscribed topics are not assigned to any members: [storetopic]
> 2018-11-05 21:25:56 INFO  AbstractCoordinator:409 - Successfully joined
> group with generation 3
> 2018-11-05 21:25:56 INFO  ConsumerCoordinator:256 - Setting newly assigned
> partitions []
> 
> The store works after this, but it is not shown.
> 
> Any input is appreciated
> 
> best regards
> 
> Patrik
> 
> PS: The customer will forward this to the Confluent support too, but I'm
> asking here for public visibility
> 


Mime
View raw message