kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Behrang Saeedzadeh <behran...@gmail.com>
Subject None of [consumer.]max.partition.fetch.bytes, [consumer.]max.message.bytes are picked up by Connect
Date Fri, 01 Sep 2017 02:48:21 GMT
Hi,

I have added the following configuration properties to a standalone Connect
server that is running a sink connector, but they are not getting picked up
by Connect:

max.partition.fetch.bytes=8192
max.message.bytes=8192
consumer.max.partition.fetch.bytes=8192
consumer.max.message.bytes=8192

In the following graph:
https://lh3.googleusercontent.com/-QLcLC5jCqcI/WajIXo-p1dI/AAAAAAAAEk0/pB94eB1WxPUMHq4QBXdqya2i1rsx6QmtQCLcBGAs/s1600/kinesis-kafka-data-rate.png

The blue line shows bytes per second received from Kafka (captured using
Wireshark). I have set the limit to 8KB and the source topic has 40
partitions, so I was expecting a maximum read rate of 320KB per second, but
I am receiving about 14MB/second.

According to the docs (
http://docs.confluent.io/current/connect/userguide.html#overriding-producer-consumer-settings)
the consumer.max.partition.fetch.bytes property should just do that. Am I
missing something here?



Thanks in advance,
Behrang
-- 
Best regards,
Behrang Saeedzadeh

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message