kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephane Maarek <steph...@simplemachines.com.au>
Subject One big kafka connect cluster or many small ones?
Date Thu, 05 Jan 2017 23:12:30 GMT

We like to operate in micro-services (dockerize and ship everything on ecs)
and I was wondering which approach was preferred.
We have one kafka cluster, one zookeeper cluster, etc, but when it comes to
kafka connect I have some doubts.

Is it better to have one big kafka connect with multiple nodes, or many
small kafka connect clusters or standalone, for each connector / etl ?

The issues I’m trying to address are :
 - Integration with our CI/CD pipeline
 - Efficient resources utilisation
 - Easily add new jar files that connectors depend on with minimal downtime
 - Monitoring operations

Thanks for your guidance


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message