kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tauzell, Dave" <Dave.Tauz...@surescripts.com>
Subject RE: Kafka as a database/repository question
Date Thu, 15 Dec 2016 18:06:10 GMT
What is the plan for backup and recovery of the kafka data?


-----Original Message-----
From: Susheel Kumar [mailto:susheel2777@gmail.com]
Sent: Thursday, December 15, 2016 12:00 PM
To: users@kafka.apache.org
Subject: Kafka as a database/repository question

Hello Folks,

I am going thru an existing design where Kafka is planned to be utilised in below manner

   1. Messages will pushed to Kafka by producers
   2. There will be updates to existing messages on ongoing basis.  The
   expectation is that all the updates are consolidated in Kafka and the
   latest and greatest version/copy is kept
   3. Consumers will read the messages from Kafka and push to Solr for
   ingestion purposes
   4. There will be no purging/removal of messages since it is expected to
   replay the messages in the future and perform full-re-ingestion.  So
   messages will be kept in Kafka for indefinite period similar to database
   where data once stored remains there and can be used later in teh future.

Do you see any pitfalls / any issue with this design especially wrt to storing the messages

This e-mail and any files transmitted with it are confidential, may contain sensitive information,
and are intended solely for the use of the individual or entity to whom they are addressed.
If you have received this e-mail in error, please notify the sender by reply e-mail immediately
and destroy all copies of the e-mail and any attachments.
View raw message