samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <criccom...@apache.org>
Subject Re: Store changelog
Date Thu, 02 Apr 2015 17:36:52 GMT
Hey Dan,

I think you might have a misunderstanding in how changelogs work with
Samza. Suppose you have a job with two tasks, and a single kv-store is
configured with a changelog attached. The changelog, in Kafka, will have
two partitions. Each task will use one partition of the changelog topic.
You only need one topic per-changelog (and no prefix) because there are
multiple partitions per changelog, and there's a 1:1 mapping between a task
and its changelog partition.

Cheers,
Chris

On Thu, Apr 2, 2015 at 10:30 AM, Dan <danharvey42@gmail.com> wrote:

> Hi all,
>
> We're just starting out using Samza to process streams we've already got in
> Kafka. Some of the jobs we've written are using the per task KV store which
> are being persisted to a changelog topic in Kafka. As you need a different
> changelog topic per task we are wondering how people are dealing with
> ensuring that each task's store has a separate changelog.
>
> I think we could define multiple stores in the properties file, then pick
> the correct one for each task index. But that seems quite a verbose way to
> go about that?
>
> If Samza could use a prefix in the properties file then generate a topic
> name for each task it would simplify using that. Maybe there's something
> I'm missing from this?
>
> Thanks,
> Dan
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message