kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paolo Patierno <ppatie...@live.com>
Subject RE: Increase number of topic in Kafka leads zookeeper fail
Date Mon, 16 May 2016 12:30:58 GMT
I agree with Tom but ... 

... to reply Christina question I guess that Anas thought about this kind of solution in relation
to the simplicity to read data from a specific devices from a backend service point of view.

Using one topic per device means that on the backend side you know exactly which is the topic
to read to get data from device X.
When you start using topic with more partitions and a key for device to determine the partition
destination, on the backend side you have to read the entire topic (with data across different
device) and get information about the key to understand which is the device sender.

I'm only guessing that it could be the reason ...

Btw ... +1 what Tom said

Paolo PatiernoSenior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoTMicrosoft Azure Advisor 
Twitter : @ppatierno
Linkedin : paolopatierno
Blog : DevExperience

> Date: Mon, 16 May 2016 05:26:31 -0700
> Subject: Re: Increase number of topic in Kafka leads zookeeper fail
> From: christian.posta@gmail.com
> To: users@kafka.apache.org
> 
> +1 what Tom said.
> 
> Curious though Anas, what motivated you to try a topic per device? was
> there something regarding management or security that you believe you can
> achieve with topic per device?
> 
> On Mon, May 16, 2016 at 4:11 AM, Tom Crayford <tcrayford@heroku.com> wrote:
> 
> > Hi there,
> >
> > Generally you don't use a single topic per device in this use case, but one
> > topic with some number of partitions and the key distribution based on
> > device id. Kafka isn't designed for millions of low volume topics, but a
> > few high volume ones.
> >
> > Thanks
> >
> > Tom Crayford
> > Heroku Kafka
> >
> > On Mon, May 16, 2016 at 5:23 AM, Anas A <anas.24aj@gmail.com> wrote:
> >
> > > We plan to use kafka as a message broker for IoT use case, where each
> > > device is considered as unique topic. when I simulated 10 message per
> > > second to 10 thousand topics zookeeper is getting bottle neck,all Kafka
> > > monitoring tools fails to read the throughput values and number of topics
> > > from JMX port because of that. will tuning zookeeper will solve the
> > issues.
> > > where In IoT use case there will be millions of device polling data to
> > > millions of topics. I want to make sure the approach is perfect to go.
> > > Please suggest.
> > >
> > >
> > > *Thanks & Regards,*
> > >
> > >
> > > Anas A
> > > DBA, Trinity Mobility
> > > [image: facebook] <https://www.facebook.com/anas.24aj> [image: twitter]
> > > <https://twitter.com/anas24aj> [image: linkedin]
> > > <http://in.linkedin.com/in/anas24aj> [image: googleplus]
> > > <https://plus.google.com/u/0/+anasA24aj/>
> > > +917736368236
> > > anas.24aj@gmail.com
> > > Bangalore
> > >
> >
> 
> 
> 
> -- 
> *Christian Posta*
> twitter: @christianposta
> http://www.christianposta.com/blog
> http://fabric8.io
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message