kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Compton <d...@danielcompton.net>
Subject Re: Design for Kafka
Date Tue, 24 Jun 2014 05:54:34 GMT
Hi there

One architecture pattern could be to have a number of app servers to receive the messages
from the devices and send them to the Kafka cluster. This would let you handle authentication
and authorisation from these devices, I don’t know your exact scenario but I don’t think
you’d want 300K devices having direct write access to your Kafka cluster.

A second issue which would be more problematic is (as I understand it) every time a producer
connects or disconnects to a cluster, it causes a rebalance of the producer -> partition
mapping. Transient disconnections or restarts would cause some issues with that number of
devices connecting and disconnecting.  

Daniel.


On Tuesday, 24 June 2014 at 12:07 pm, 229hgcqi8+ag6x9a7o@guerrillamail.com wrote:

> Hi, I’m new. We are considering using Kafka for our next generation product. We have
300K devices that are connected to our cloud backend. We are currently using a set of web
services for our headless client devices to send state information. We are moving to a message-based
interface.  
>  
> Does Kafka support a use case of many thousands of client/producers? I can’t seem to
find anyone using Kafka in this way. It appears most user have the small number of producers
inside the data center.
>  
> It also seems the producers need to know a lot of state information about the Kafka cluster.
Why should a client have to be involved selecting the right partition for a given topic? 

>  
> Or maybe I am total confused on how to correctly design for Kafka.
>  
> Any help / guidance is appreciated.
>  
>  
>  
>  
>  
> ----
> Sent using GuerrillMail.com (http://GuerrillMail.com)
> Block or report abuse: https://www.guerrillamail.com/abuse/?a=VFJxFx4gSLUTgw%2F68W4ecRzCA8WC1Q%3D%3D
>  
>  



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message