kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <ma...@modelcitizen.com>
Subject random access performance of messages.
Date Fri, 21 Oct 2011 16:02:56 GMT
I would like to use Kafka to process messages that need to be immutably
stored for a N-days, and during that period the msgs need to be indexed,
searched, as well as retrieval of msg data that is queried. 


One approach is to read messages from Kafka and store the messages in a
secondary db for query and data retrieval.  Once the messages are read and
processed into the secondary db, then the messages can be discarded from the
Kafka queue.


Another approach is to read the messages, build an external index for
searching that directly references the message data by Kafka-key in the
Kafka queue itself.  In this case the Kafka becomes the message store for
the life of the message/data.


The latter would be ideal for me if the performance of query-by-key and
message data retrieval is very good.


Is random query of message+data good for Kafka?  Is this an appropriate
usecase for Kafka?


Thank you.





  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message