kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Johan Lundahl <johan.lund...@gmail.com>
Subject Producer questions and more
Date Fri, 14 Dec 2012 12:42:56 GMT

I'm trying to promote Kafka for our centralized log aggregation/metrics
system and have set up a proof of concept based on 0.7.2 which seems to
work very well for our purposes but the improvements in 0.8 looks too
important for us to go live without them. After studying the presentation
material and videos I have some questions:

1) It's mentioned by Jay in one of the videos that Kafka is designed for <
1000 topics. I understand the fundamentals of what a topic is meant to be
but are there any real system limits in regards to this? In our case, we
have around 100 clusters running our different (java only) applications
with a guesstimate average size of 40 nodes each. We have around 30
different types of logs plus some other metrics so this would give us 100
clusters * 35 types = 3500 topics. Furthermore, it's likely that the number
of clusters will increase in the future. Is this something that could cause
us trouble or are this figure of < 1000 topics just a guideline?

2) The KafkaLog4jAppender is a very convenient way for us to stream our
logs since no changes in the application code will be needed but is it
possible to build a lightweight jar with only the KafkaLog4jAppender
producer that we easily could deploy on our production servers? I'm not an
sbt expert but I could only manage to build a full package including broker
and everything which is a lot heavier.

3) As our applications are pure java, it would be nice to avoid the scala
runtime on the producer side. Would it be feasible to implement the
KafkaLog4jAppender in java? With 0.8, the dependency on Zookeeper should
not be needed on producer side either if I understand correctly right?

4) How do you handle non application logs, for example webserver logs? Is
there something like an Apache httpd mod_kafka? OS metrics?

5) In general I think it's somewhat tricky to follow the status of the
different Kafka versions. It seems like 0.8 has been postponed a bit
relative to original plans but are there newer estimations of when it can
be considered "stable"? Is there a summary of the important changes for the

6) I've seen a few mails recently about git migration. Would it be enough
to only use git from 0.8 or would I still need svn for anything?

7) Has there been discussions about creating a logo for Kafka? My
conceptual system diagrams look a bit empty on the Kafka parts in the
promo-slides I've made...(the same thing applies to the Storm parts)

Thanks a lot in advance for your help!

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message