kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neha Narkhede <neha.narkh...@gmail.com>
Subject Re: random access performance of messages.
Date Fri, 21 Oct 2011 17:02:23 GMT
Marko,

I agree with Sharad. Instead of using kafka for random message
lookups, you could use it as the persistent message bus between the
publishers of the messages and your indexing system.
Using the low level consumer API (SimpleConsumer), you could set up
your indexer processes to pull from the broker partitions for a topic.
You would have to checkpoint your Kafka
offsets to match the data indexed and flushed to disk, and re-fetch
data from Kafka, if/when the indexer fails.

Thanks,
Neha

On Fri, Oct 21, 2011 at 9:47 AM, Sharad Agarwal <sharad.apache@gmail.com> wrote:
> kafka is more suited for sequential message reads. Not really meant for
> random message lookups.
>
> Also using kafka as *long* term message store is not a good usecase.
>
> On Fri, Oct 21, 2011 at 9:32 PM, <marko@modelcitizen.com> wrote:
>
>> I would like to use Kafka to process messages that need to be immutably
>> stored for a N-days, and during that period the msgs need to be indexed,
>> searched, as well as retrieval of msg data that is queried.
>>
>>
>>
>> One approach is to read messages from Kafka and store the messages in a
>> secondary db for query and data retrieval.  Once the messages are read and
>> processed into the secondary db, then the messages can be discarded from
>> the
>> Kafka queue.
>>
>>
>>
>> Another approach is to read the messages, build an external index for
>> searching that directly references the message data by Kafka-key in the
>> Kafka queue itself.  In this case the Kafka becomes the message store for
>> the life of the message/data.
>>
>>
>>
>> The latter would be ideal for me if the performance of query-by-key and
>> message data retrieval is very good.
>>
>>
>>
>> Is random query of message+data good for Kafka?  Is this an appropriate
>> usecase for Kafka?
>>
>>
>>
>> Thank you.
>>
>>
>>
>> Marko.
>>
>> .
>>
>>
>>
>>
>
>
> --
> Thanks
> Sharad Agarwal
> Hadoop and Avro Committer
> Technology Platforms, InMobi
> *Disclaimer: Opinions expressed here are my own and do not represent past or
> present employers.*
>

Mime
View raw message