kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From SJTU胡昊坤 <geeku...@gmail.com>
Subject Re: How to manipulate client settings through AdminAPI?
Date Fri, 12 Oct 2018 02:43:54 GMT
Hi Will,
Unfortunately, you can't use AdminClient to do that. Because Kafka use
zookeeper to keep the user quotas. And there is no API for this.
You can implement this feature by writing raw data to zookeeper. The
protocol is simple:
first, write data to:
 znode: ${kafka_rootdir}/config/users/${user_id}/clients/${client_id}
value: {"version": 1, "config": {"producer_byte_rate": "1024",
"consumer_byte_rate": "1024"}}
second, write config change with sequence to:
znode: ${kafka_rootdir}/config/changes/config_change_${sequence} value:
{"version": 2, "entity_path": "users/${user_id}/clients/${client_id}"}

You can find more impelment detail in these source files:
src/main/scala/kafka/admin/ConfigCommand.scala
src/main/scala/kafka/zk/AdminZkClient.scala
src/main/scala/kafka/server/DynamicConfigManager.scala ( The comments
helps!)

from DynamicConfigManager.scala:

Config is stored under the path: /config/entityType/entityName
*   E.g. /config/topics/<topic_name> and /config/clients/<clientId>
* This znode stores the overrides for this entity in properties format
with defaults stored using entityName "<default>".
* Multiple entity names may be specified (eg. <user, client-id>
quotas) using a hierarchical path:
*   E.g. /config/users/<user>/clients/<clientId>
*
* To avoid watching all topics for changes instead we have a notification path
*   /config/changes
* The DynamicConfigManager has a child watch on this path.
*
* To update a config we first update the config properties. Then we
create a new sequential
* znode under the change path which contains the name of the
entityType and entityName that was updated, say
*   /config/changes/config_change_13321
* The sequential znode contains data in this format: {"version" : 1,
"entity_type":"topic/client", "entity_name" : "topic_name/client_id"}
* This is just a notification--the actual config change is stored only
once under the /config/entityType/entityName path.
* Version 2 of notifications has the format: {"version" : 2,
"entity_path":"entity_type/entity_name"}
* Multiple entities may be specified as a hierarchical path (eg.
users/<user>/clients/<clientId>).
*


However, I suggest not relying on this implement for a long time. Because
it may change in new version of Kafka. It's not part of public API.

Hope this will help :)

Will Weber <rwaweber@gmail.com> 于2018年10月8日周一 下午11:59写道:

> Hey all,
>
> Working on a tool for Kafka lifecycle management and one of the features
> we're keen to try to wrap is user quotas.
>
> This capability definitely exists through sets of scripts included with
> Kafka, case and point: https://kafka.apache.org/documentation/#quotas
>
> For context:
>
> > bin/kafka-configs.sh  --zookeeper localhost:2181 --alter --add-config
> 'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200'
> --entity-type users --entity-name user1 --entity-type clients --entity-name
> clientA
>
> Updated config for entity: user-principal 'user1', client-id 'clientA'.
>
> Which would set limits on a given client.
>
> Based on this command, it leads me to believe that the
> AdminClient#alterConfigs
> <
> https://kafka.apache.org/20/javadoc/org/apache/kafka/clients/admin/AdminClient.html#alterConfigs-java.util.Map-
> >
> method
> is probably the way to go.
>
> However, I noticed that in the ConfigResource.Type
> <
> https://kafka.apache.org/20/javadoc/org/apache/kafka/common/config/ConfigResource.Type.html
> >
>  enum, there are only Broker, Unknown and Topic options. Naively, I imagine
> there to be a corresponding Enum to match the entity-type flag options as
> well, but no such luck.
>
> Anyone have recommendations as to how I could go about enforcing quotas
> through the admin api?
>
> Mostly just looking for a nudge in the right direction, any help is
> appreciated!
>
> Best,
>
> -Will
>


-- 

Haokun Hu



E-mail: geekun94@gmail.com or 18817874087@163.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message