kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alberto Coello <coloalcol...@gmail.com>
Subject Does Kafka Connect handles well having lots of connectors in the same cluster?
Date Wed, 09 Dec 2020 16:17:39 GMT
So, I have the intention to deploy a lot of connectors(An unbounded number,
like hundreds, thousands or more) in my Kafka Connect cluster, each of them
would be of the same class(org.kaliy.kafka.connect.rss.RssSourceConnector)
but with different configuration, the difference would be in the RSS URI
they read from. I mean to  create, and manage, them in an automated way
implementing a process in charge of managing their status via Connect REST
API. BUT as all Connect configuration is saved in three
topics: config.storage.topic, offset.storage.topic and
status.storage.topic; I am doubting about its capacity to handle this
number of connectors. Any other problems you can higlight about this
approach would be appreciated, thanks a lot!.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message