kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carey, Lynda" <Lynda.Ca...@Six3Systems.com>
Subject RE: Kafak Mirroring Questions
Date Tue, 13 Dec 2011 17:07:06 GMT
So, I've read all the documents that were suggested and now I'm trying to follow the instructions
for "How to set up a Mirror" on this site: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring.
 The instructions state that you provide the broker with the 3 properties files.  However,
when I try to submit three properties files, I get a usage error (see below).  

[kafka %]  JMX_PORT=8888 bin/kafka-server-start.sh config/mirror-server.properties config/mirror-consumer.properties
config/mirror-producer.properties

USAGE: java [options] KafkaServer server.properties [consumer.properties


I am using kafka-0.6, downloaded from http://sna-projects.com/kafka/downloads.php


_______________________________
Lynda Carey, Software Engineer
Six3 Systems, Enterprise Systems Division
301-206-6000 (Office)
410-262-4942(Cell)

This e-mail message is for the sole use of the intended recipient(s) and may contain Six3
Systems Private or Six3 Systems Proprietary information. Any unauthorized review, use, disclosure,
or distribution is prohibited. If you are not an intended recipient, please contact the sender
by reply e-mail and destroy all copies of the original message.


________________________________________
From: Jay Kreps [jay.kreps@gmail.com]
Sent: Monday, December 12, 2011 11:49 PM
To: kafka-users@incubator.apache.org
Subject: Re: Kafak Mirroring Questions

Hi Olivier,

No the mirrored cluster is essentially a completely different cluster. It
may have a different number of partitions or servers and there is no
correspondence between offsets.

-Jay

On Mon, Dec 12, 2011 at 8:35 PM, Olivier Pomel <oli@datadoghq.com> wrote:

> One more question: will the offsets for individual messages in the
> master and mirror always be the same? In other words, would a failover
> be completely transparent to consumers that may persist state linked
> to specific offsets?
> Thanks,
> Olivier.
>
> On Mon, Dec 12, 2011 at 8:18 PM, Jun Rao <junrao@gmail.com> wrote:
> > You can find more information in the patch of this jira:
> > https://issues.apache.org/jira/browse/KAFKA-199
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Dec 12, 2011 at 11:57 AM, Carey, Lynda
> > <Lynda.Carey@six3systems.com>wrote:
> >
> >> Hi!  I've been tasked with setting up a kafka cluster and mirroring.
>  I've
> >> set up a small cluster of just two machines.  I am running zookeeper,
> two
> >> brokers, 1 producer and 3 consumers.  Each consumer is in it's own group
> >> and all are reading from the same topic.  So, with this setup, I'm using
> >> simple console producer that reads from the command line and writes to
> >> kafka.  The consumer read the messages fine.
> >>
> >> Now, I need to set up a mirroring cluster.  I've read the wiki document
> >> about kafka mirroring and it leaves me a little confused.  I understand
> >> that the mirror cluster uses an embedded consumer to read from the
> source
> >> cluster and writes the messages to kafka on the mirrored cluster.
>  What's
> >> not clear to me is what else needs to be established on the mirrored
> >> cluster?  Do I only need one embedded consumer and producer on the
> mirror
> >> to get all the messages (regardless of topic/broker/etc)?  of do I need
> a
> >> pair of embedded consumer and producer for each topic?  What is reading
> the
> >> messages on the mirrored cluster? -- do I need to deploy the same
> consumers
> >> there as on the source cluster?  Is there any other documentation
> regarding?
> >>
> >> Any information you can give me would be awesome.  I'm just not getting
> it
> >> from the documentation alone.
> >>
> >> Thanks
> >> _______________________________
> >> Lynda Carey, Software Engineer
> >> Six3 Systems, Enterprise Systems Division
> >>
> >> This e-mail message is for the sole use of the intended recipient(s) and
> >> may contain Six3 Systems Private or Six3 Systems Proprietary
> information.
> >> Any unauthorized review, use, disclosure, or distribution is
> prohibited. If
> >> you are not an intended recipient, please contact the sender by reply
> >> e-mail and destroy all copies of the original message.
> >>
> >>
>

Mime
View raw message