kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ning Zhang <ning2008w...@gmail.com>
Subject Re: Does Kafka Connect handles well having lots of connectors in the same cluster?
Date Sun, 13 Dec 2020 19:52:14 GMT
As org.kaliy.kafka.connect.rss.RssSourceConnector is open-source, it seems a better idea to
extend it to read from multiple RSS URI, so you may still enjoy the fine-granularity of automated
management of a set of URI within one connector, while not creating too many connectors

On 2020/12/09 16:17:39, Alberto Coello <coloalcoloal@gmail.com> wrote: 
> So, I have the intention to deploy a lot of connectors(An unbounded number,
> like hundreds, thousands or more) in my Kafka Connect cluster, each of them
> would be of the same class(org.kaliy.kafka.connect.rss.RssSourceConnector)
> but with different configuration, the difference would be in the RSS URI
> they read from. I mean to  create, and manage, them in an automated way
> implementing a process in charge of managing their status via Connect REST
> API. BUT as all Connect configuration is saved in three
> topics: config.storage.topic, offset.storage.topic and
> status.storage.topic; I am doubting about its capacity to handle this
> number of connectors. Any other problems you can higlight about this
> approach would be appreciated, thanks a lot!.

View raw message