kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Kreps <jay.kr...@gmail.com>
Subject Re: Kafak Mirroring Questions
Date Tue, 13 Dec 2011 04:49:05 GMT
Hi Olivier,

No the mirrored cluster is essentially a completely different cluster. It
may have a different number of partitions or servers and there is no
correspondence between offsets.

-Jay

On Mon, Dec 12, 2011 at 8:35 PM, Olivier Pomel <oli@datadoghq.com> wrote:

> One more question: will the offsets for individual messages in the
> master and mirror always be the same? In other words, would a failover
> be completely transparent to consumers that may persist state linked
> to specific offsets?
> Thanks,
> Olivier.
>
> On Mon, Dec 12, 2011 at 8:18 PM, Jun Rao <junrao@gmail.com> wrote:
> > You can find more information in the patch of this jira:
> > https://issues.apache.org/jira/browse/KAFKA-199
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Dec 12, 2011 at 11:57 AM, Carey, Lynda
> > <Lynda.Carey@six3systems.com>wrote:
> >
> >> Hi!  I've been tasked with setting up a kafka cluster and mirroring.
>  I've
> >> set up a small cluster of just two machines.  I am running zookeeper,
> two
> >> brokers, 1 producer and 3 consumers.  Each consumer is in it's own group
> >> and all are reading from the same topic.  So, with this setup, I'm using
> >> simple console producer that reads from the command line and writes to
> >> kafka.  The consumer read the messages fine.
> >>
> >> Now, I need to set up a mirroring cluster.  I've read the wiki document
> >> about kafka mirroring and it leaves me a little confused.  I understand
> >> that the mirror cluster uses an embedded consumer to read from the
> source
> >> cluster and writes the messages to kafka on the mirrored cluster.
>  What's
> >> not clear to me is what else needs to be established on the mirrored
> >> cluster?  Do I only need one embedded consumer and producer on the
> mirror
> >> to get all the messages (regardless of topic/broker/etc)?  of do I need
> a
> >> pair of embedded consumer and producer for each topic?  What is reading
> the
> >> messages on the mirrored cluster? -- do I need to deploy the same
> consumers
> >> there as on the source cluster?  Is there any other documentation
> regarding?
> >>
> >> Any information you can give me would be awesome.  I'm just not getting
> it
> >> from the documentation alone.
> >>
> >> Thanks
> >> _______________________________
> >> Lynda Carey, Software Engineer
> >> Six3 Systems, Enterprise Systems Division
> >>
> >> This e-mail message is for the sole use of the intended recipient(s) and
> >> may contain Six3 Systems Private or Six3 Systems Proprietary
> information.
> >> Any unauthorized review, use, disclosure, or distribution is
> prohibited. If
> >> you are not an intended recipient, please contact the sender by reply
> >> e-mail and destroy all copies of the original message.
> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message