kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Enrico Olivelli - Diennea <enrico.olive...@diennea.com>
Subject Kafka as Transaction-log - Producer fencing
Date Fri, 29 Apr 2016 08:32:37 GMT
I would like to use Kafka as transaction log in order to support a case of replicated state
machine, but actually (using 0.9.x) there is a feature I would like to have.
I'm using Apache BookKeeper and this feature (fencing) is native, but I have some cases of
customers which already use Kafka and I would like to let them use Kafka as log manager.

My scenario is the following:
- I have a group of processes which compete to acquire leadership (for instance using ZooKeeper
- The only one process which is the leader can write to a shared commit log (a topic with
a single partition)
- The other processes follow the leader, "tailing" the commit log and replaying the log to
the local state

When the leader loses its role, for instance ZK session expires, I want the "log manager"
to prevent it from writing new entries (fencing).
This action will guarantee that only one service can really "act" as leader in the group,
that is to decide changes to the global state of the system.

I do not known Kafka internals but I think that a "producer fencing" feature would be simple
to implement if I use a single topic with a single partition.

As there is only one leader Broker per partition, this leader could enforce the constraint
that only any producer can be active at a time on the partition.

- Producer A (leader) logs in and writes entries
- Producer B (new leader) logs in, writes a message with a flag "close other producers for
this partition"
- Producer A gets a "ProducerFencedException" and known that someone else became the new leader

That do you think?
Is it interesting for others ?
Is there any way to achive this goal using actual Kafka features ?

Thank you

Enrico Olivelli
Software Development Manager @Diennea
Tel.: (+39) 0546 066100 - Int. 925
Viale G.Marconi 30/14 - 48018 Faenza (RA)

MagNews - E-mail Marketing Solutions
Diennea - Digital Marketing Solutions


Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email marketing! http://www.magnews.it/newsletter/

The information in this email is confidential and may be legally privileged. If you are not
the intended recipient please notify the sender immediately and destroy this email. Any unauthorized,
direct or indirect, disclosure, copying, storage, distribution or other use is strictly forbidden.
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message