kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kuldeep Kamboj <kuldeep.kam...@osscube.com>
Subject Re: How to build strategy for particular setup of kafka
Date Fri, 29 Apr 2016 11:48:46 GMT
Sorry for wrong php code. It should be like that

3. In php code, Instead of using partition = 0, Use App1 = 0, App2 = 1,
App3 = 2

   In App 1 Producer Script :
       $producer->setMessages($queue, 0, array($data));

   In App 2 Producer Script :
       $producer->setMessages($queue, 1, array($data));

   In Consumer 1 Script :
       $consumer->setPartition($queue, 0);

   In Consumer 2 Script :
       $consumer->setPartition($queue, 1);


On Fri, Apr 29, 2016 at 5:17 PM, Kuldeep Kamboj <kuldeep.kamboj@osscube.com>
wrote:

> I restarted brainstroming for Kafka setup. I found link
> http://www.michael-noll.com/blog/2013/03/13/running-a-multi-broker-apache-kafka-cluster-on-a-single-node/
> for kafka cluster setup (Thanks Lohith Samaga for valuable suggestion) .
> After try on local machine, I got this is what I need.
>
> Now I am implementing following approach.
>
> 1. First create 3 brokers
>     cd /opt/kafka
>     cp config/server.properties config/broker1.properties
>         broker.id=1
>         port=9092
>         log.dir=/tmp/kafka-logs-1
>
>     cp config/server.properties config/broker2.properties
>         broker.id=2
>         port=9093
>         log.dir=/tmp/kafka-logs-2
>
>     cp config/server.properties config/broker3.properties
>         broker.id=3
>         port=9094
>         log.dir=/tmp/kafka-logs-3
>
>
> 2. Create topics with 3 partitions & 3 replications
>              /opt/kafka/bin/kafka-topics.sh --zookeeper localhost:2181
> --create --topic category --partitions 3            --replication-factor
> 3
>         ...
>
> 3. In php code, Instead of using partition = 0, Use App1 = 1, App2 = 2,
> App3 = 3
>
>    In App 1 Producer Script :
>        $producer->setMessages($queue, 0, array($data));
>
>    In App 2 Producer Script :
>        $producer->setMessages($queue, 2, array($data));
>
>    In Consumer 1 Script :
>        $consumer->setPartition($queue, 0);
>
>    In Consumer 2 Script :
>        $consumer->setPartition($queue, 1);
>
> I expect, it should work as my requirements.
>
>
> On Fri, Apr 22, 2016 at 3:34 PM, Lohith Samaga M <
> Lohith.Samaga@mphasis.com> wrote:
>
>> Hi,
>>         Please set up a Kafka cluster. So, you can get high throughput as
>> well as high availability.
>>
>> Best regards / Mit freundlichen Grüßen / Sincères salutations
>> M. Lohith Samaga
>>
>>
>> -----Original Message-----
>> From: Gaurav Agarwal [mailto:gaurav130403@gmail.com]
>> Sent: Friday, April 22, 2016 12.44
>> To: users@kafka.apache.org
>> Subject: Re: How to build strategy for particular setup of kafka
>>
>> Hi
>> You can have one or two instances of Kafka but you can have one or two
>> Kafka topic dedicated to each application according to the need. Partition
>> will have u in increasing the throughput and consumer group id can help u
>> to make queue as topic or queue.
>> On Apr 22, 2016 12:37 PM, "Kuldeep Kamboj" <kuldeep.kamboj@osscube.com>
>> wrote:
>>
>> > Thanks for reply,
>> >
>> > I understand the your point, But my whole strategy depend on first
>> > issue and that is how I can integrate Apps in architecture. Partition
>> > / Consumer groups have different purpose. If I need to setup three
>> > kafka instances each for App ?
>> >
>> > On Fri, Apr 22, 2016 at 12:30 PM, Lohith Samaga M <
>> > Lohith.Samaga@mphasis.com
>> > > wrote:
>> >
>> > > Hi,
>> > >         It is better NOT to share topics among applications. You may
>> > > have a wrapper application reading from the queue/topic and routing
>> > > it to the correct application, but it is simpler for each
>> > > application to read from its own topic.
>> > >
>> > > Best regards / Mit freundlichen Grüßen / Sincères salutations M.
>> > > Lohith Samaga
>> > >
>> > >
>> > >
>> > > -----Original Message-----
>> > > From: Kuldeep Kamboj [mailto:kuldeep.kamboj@osscube.com]
>> > > Sent: Friday, April 22, 2016 12.08
>> > > To: users@kafka.apache.org
>> > > Subject: How to build strategy for particular setup of kafka
>> > >
>> > > I have three applications let named AppA, AppB, AppC. All could have
>> > > several some message queue named like
>> > >
>> > >     AppA
>> > >        MsgQueueA
>> > >        MsgQueueB
>> > >     AppB
>> > >        MsgQueueA
>> > >        MsgQueueB
>> > >        MsgQueueC
>> > >     AppC
>> > >        MsgQueueA
>> > >        MsgQueueB
>> > >        MsgQueueD
>> > >
>> > > I have very little exposure for Apache Kafka and even for message
>> > > queue concepts.
>> > >
>> > > I just thinking to create topics for MsgQueueA, MsgQueueB and so on.
>> > > But
>> > I
>> > > am confused how to integrate AppA, AppB, AppC. Partition and
>> > > Consumer
>> > group
>> > > look like wrong choices for them.
>> > >
>> > > Also If separation of queue/topic for Apps is better idea instead of
>> > > re-using them for different Apps for better debugging Like below
>> > >
>> > >
>> > >     AppA
>> > >        AppAMsgQueueA
>> > >        AppAMsgQueueB
>> > >     AppB
>> > >        AppBMsgQueueA
>> > >        AppBMsgQueueB
>> > >        AppBMsgQueueC
>> > >     AppC
>> > >        AppCMsgQueueA
>> > >        AppCMsgQueueB
>> > >        AppCMsgQueueD
>> > >
>> > >
>> > >
>> > >
>> > > --
>> > > Kuldeep Kamboj
>> > > Information transmitted by this e-mail is proprietary to Mphasis,
>> > > its associated companies and/ or its customers and is intended for
>> > > use only by the individual or entity to which it is addressed, and
>> > may
>> > > contain information that is privileged, confidential or exempt from
>> > > disclosure under applicable law. If you are not the intended
>> > > recipient or it appears that this mail has been forwarded to you
>> > > without proper authority, you are notified that any use or
>> > > dissemination of this information in any manner is strictly
>> > > prohibited. In such cases, please notify us immediately at
>> > > mailmaster@mphasis.com and delete this mail from your records.
>> > >
>> >
>> >
>> >
>> > --
>> > Kuldeep Kamboj
>> > Mob. 9871662849
>> >
>> Information transmitted by this e-mail is proprietary to Mphasis, its
>> associated companies and/ or its customers and is intended
>> for use only by the individual or entity to which it is addressed, and
>> may contain information that is privileged, confidential or
>> exempt from disclosure under applicable law. If you are not the intended
>> recipient or it appears that this mail has been forwarded
>> to you without proper authority, you are notified that any use or
>> dissemination of this information in any manner is strictly
>> prohibited. In such cases, please notify us immediately at
>> mailmaster@mphasis.com and delete this mail from your records.
>>
>
>
>
> --
> Kuldeep Kamboj
> Mob. 9871662849
>



-- 
Kuldeep Kamboj
Mob. 9871662849

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message