kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sharninder Khera <sharnin...@gmail.com>
Subject Re: Storing Kafka Message JSON to deep storage like S3
Date Tue, 06 Dec 2016 11:35:46 GMT
Why not just have a parallel consumer read all messages from whichever topics you're interested
in and store them wherever you want to? You don't need to "backup" Kafka messages. 

From: Aseem Bansal <asmbansal2@gmail.com>
Sent: Tuesday, December 6, 2016 4:55 PM
Subject: Storing Kafka Message JSON to deep storage like S3
To:  <users@kafka.apache.org>


Has anyone done a storage of Kafka JSON messages to deep storage like S3.
We are looking to back up all of our raw Kafka JSON messages for
Exploration. S3, HDFS, MongoDB come to mind initially.

I know that it can be stored in kafka itself but storing them in Kafka
itself does not seem like a good option as we won't be able to query it and
the configurations of machines containing kafka will have to be increased
as we go. Something like S3 we won't have to manage.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message