kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sa Li <sal...@gmail.com>
Subject Re: Kafka System test
Date Fri, 23 Jan 2015 22:42:52 GMT
Also I found ./kafka/system_test/cluster_config.json is duplicated on each
directory ./kafka/system_test/replication_testsuite/testcase_xxxx/

When I change the ./kafka/system_test/cluster_config.json, do I need to
overwrite it each
./kafka/system_test/replication_testsuite/testcase_xxxx/cluster_config.json
?

Thanks

AL



On Fri, Jan 23, 2015 at 1:39 PM, Sa Li <salicn@gmail.com> wrote:

> Thanks for reply. Ewen, pertaining to your statement "... hostname setting
> being a list instead of a single host," are you saying entity_id 1 or 0,
>
> "entity_id": "0",
>             "hostname":
> "10.100.70.28,10.100.70.29,10.100.70.30,10.100.70.31,10.100.70.32",
>
> "entity_id": "1",
>             "hostname": "10.100.70.28",
>
> I thought the role zookeeper has multiple hosts, so I list all the IPs of
> ensemble. While entity 1 is only about 1 broker (my design about production
> cluster to fire up one broker for each host, so 3 nodes with 3 brokers), so
> I specify one hostname IP only here. How do I change?
>
>
> thanks
>
> AL
>
> On Fri, Jan 23, 2015 at 1:22 PM, Ewen Cheslack-Postava <ewen@confluent.io>
> wrote:
>
>> 1. Except for that hostname setting being a list instead of a single host,
>> the changes look reasonable. That is where you want to customize settings
>> for your setup.
>>
>> 2 & 3. Yes, you'll want to update those files as well. They top-level ones
>> provide defaults, the ones in specific test directories provide overrides
>> for that specific test. But they aren't combined in any way, i.e. the more
>> specific one is just taken as a whole rather than being like a diff, so
>> you
>> do have to update both.
>>
>> You might want to take a look at
>> https://issues.apache.org/jira/browse/KAFKA-1748. Currently if you want
>> to
>> run all tests it's a pain to change the hosts they're running on since it
>> requires manually editing all those files. The patch gets rid of
>> cluster_config.json and provides a couple of different ways of configuring
>> the cluster -- run everything on localhost, get cluster info from a single
>> json file, or get the ssh info from Vagrant.
>>
>>
>>
>> On Fri, Jan 23, 2015 at 11:50 AM, Sa Li <salicn@gmail.com> wrote:
>>
>> > Hi, All
>> >
>> > From my last ticket (Subject: kafka production server test), Guozhang
>> > kindly point me the system test package come with kafka source build
>> which
>> > is really cool package. I took a look at this package, things are clear
>> is
>> > I run it on localhost, I don't need to change anything, say,
>> > cluster_config.json defines entities, and system test reads
>> > testcase_xxxx_properties.json to override the properties in
>> > cluster_config.json. For example, cluster_config.json defaults hostname
>> as
>> > localhost, and three brokers, I assume it will create 3 brokers in
>> > localhost and continue the test.
>> >
>> > Currently I install the package on a vagrant VM, and like to run the
>> system
>> > test on VM and remotely access production to test production cluster.
>> The
>> > production cluster has 3 nodes. kafka production cluster is on top of a
>> > 5-node zookeeper ensemble.  My questions is how to effectively change
>> the
>> > properties on vagrant system test package.
>> >
>> > 1. change on cluster_config.json, like
>> >     {
>> >             "entity_id": "0",
>> >             "hostname":
>> > "10.100.70.28,10.100.70.29,10.100.70.30,10.100.70.31,10.100.70.32",
>> >             "role": "zookeeper",
>> >             "cluster_name": "target",
>> >             "kafka_home": "/etc/kafka",
>> >             "java_home": "/usr/lib/jvm/java-7-openjdk-amd64/jre",
>> >             "jmx_port": "9990"
>> >         },
>> >         {
>> >             "entity_id": "1",
>> >             "hostname": "10.100.70.28",
>> >             "role": "broker",
>> >             "cluster_name": "target",
>> >             "kafka_home": "/etc/kafka",
>> >             "java_home": "/usr/lib/jvm/java-7-openjdk-amd64/jre",
>> >             "jmx_port": "9991"
>> >         },
>> >
>> >  Here because I want to test remote servers, so I need to change the
>> > cluster_name as "target", right?
>> >
>> > 2.  In directory ./replication_testsuite/config/ , for all the
>> properties
>> > files, do I need to change them all to be the same as the properties on
>> > production servers?
>> >
>> > 3. in ./replication_testsuite/testcase_xxxx/, seems I need to make
>> > corresponding changes as well to keep consistent with
>> > ./config/....properties, such as
>> > "log.dir": "/tmp/kafka_server_1_logs" will be change to the log.dir in
>> my
>> > production server.properties, is that right?
>> >
>> >
>> > Hope someone who has done the system test on remote server can share
>> some
>> > experience, thanks
>> >
>> >
>> >
>> > AL
>> >
>> > --
>> >
>> > Alec Li
>> >
>>
>>
>>
>> --
>> Thanks,
>> Ewen
>>
>
>
>
> --
>
> Alec Li
>



-- 

Alec Li

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message