kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Stein <crypt...@gmail.com>
Subject Re: WELCOME to users@kafka.apache.org
Date Mon, 05 Aug 2013 11:58:18 GMT
Yes, depending on your implementation of the event processing system you
may simply be able to have each topic be the type of event.

So you would have a topic called "TypeA" and then setup a consumer group
and those consumers (if you really only needed 1 consumer set the
partitions to 1) would get everything from the "TypeA" topic.  If you had
more event types then just setup more topics and then consumers group for a
consumer for those topics.

Now depending on what you need to-do you may need topics to not be by type
perhaps be another value and in that case you can still "pin" data to a
consumer in that case use Semantic Partitioning
*Semantic partitioning*

"Consider an application that would like to maintain an aggregation of the
number of profile visitors for each member. It would like to send all
profile visit events for a member to a particular partition and, hence,
have all updates for a member to appear in the same stream for the same
consumer thread. The producer has the capability to be able to semantically
map messages to the available kafka nodes and partitions. This allows
partitioning the stream of messages with some semantic partition function
based on some key in the message to spread them over broker machines. The
partitioning function can be customized by providing an implementation of
the kafka.producer.Partitioner interface, default being the random
partitioner. For the example above, the key would be member_id and the
partitioning function would be hash(member_id)%num_partitions."

Take a look at KeyedMessage.scala, the ConsoleProducer.scala example uses
the overloaded constructor but you can use the constructor which makes the
partitions random by instead passin in the key (the type) and also set
your key-serializer to kafka.serializer.StringEncoder or make your own

In this case you might need to have a partition for each topic unless you
can have a consumer read different event types again, it all depends on
your implementation of your event processing system.

Hope this helps getting you started some, thanks!

 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>

On Mon, Aug 5, 2013 at 5:44 AM, masoom alam

> Hi every one,
> i am new to kafka and desiging an Event Processing System. Is this possible
> that Kafka  Broker can do some event dependency handling so that for
> example events of type A only goes to Consumer1.
> I hope I was able to explain my problem
> Thanks.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message