kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From D <subharaj.ma...@gmail.com>
Subject Fair Usage of Kafka Queues
Date Fri, 27 Nov 2015 09:54:48 GMT
Hi,

We have a ElasticSearch-Logstash-Kibana deployment in which multiple 
logstash-forwarders (from multiple service logs) pushes log to logstash 
which then sends it to Kafka and then one more logstash pulls those logs 
from Kafka and indexes them in ElasticSearch cluster. Right now we are 
using single kafka queue in which all the logs from all the services are 
going.

Can we configure Kafka in such a way that a single logstash-forwarder 
(of a single service) does not hog the entire set-up? We want to ensure 
that all services can use the set-up fairly. So basically I want to 
ensure the Kafka queues are not always filled up by only one type of 
messages.

Thanks,
Debraj

Mime
View raw message