kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryanne Dolan <ryannedo...@gmail.com>
Subject Re: Event Sourcing question
Date Wed, 08 May 2019 19:44:28 GMT
Pavel, one thing I'd recommend: don't jam multiple event types into a
single topic. You are better served with multiple topics, each with a
single schema and event type. In your case, you might have a received topic
and a transformed topic, with an app consuming received and producing
transformed.

If your transformer process consumes, produces, and commits in the right
order, your app can crash and restart without skipping records. Consider
using Kafka Streams for this purpose, as it takes care of the semantics you
need to do this correctly.

Ryanne

On Wed, May 8, 2019 at 12:06 PM Pavel Molchanov <
pavel.molchanov@infodesk.com> wrote:

> I have an architectural question.
>
> I am planning to create a data transformation pipeline for document
> transformation. Each component will send processing events to the Kafka
> 'events' topic.
>
> It will have the following steps:
>
> 1) Upload data to the repository (S3 or other storage). Get public URL to
> the uploaded document. Create 'received' event with the document URL and
> send the event to the Kafka 'events' topic.
>
> 2) Tranformer process will be listening to the Kafka 'events' topic. It
> will react on the 'received' event in the 'events' topic, will download the
> document, transform it, push the transformed document to the repository (S3
> or other storage), create 'transformed' event and send 'transformed' event
> to the same 'events' topic.
>
> Tranformer process can break in the middle (exception, died, crashed,
> etc.). Upon startup, Tranformer process needs to check 'events' topic for
> documents that were received but not transformed.
>
> Should it read all events from the 'events' topic? Should it join
> 'received' and 'transformed' events somehow to understand what was received
> but not transformed?
>
> I don't have a clear idea of how it should behave.
>
> Please help.
>
> *Pavel Molchanov*
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message