kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Parviz deyhim <dey...@gmail.com>
Subject Re: Kafka tests on a remote cluster
Date Thu, 27 Dec 2018 20:57:06 GMT
+dev@kafka.apache.org

On Wed, Dec 26, 2018 at 8:53 PM Parviz deyhim <deyhim@gmail.com> wrote:

> Thanks fair points. Probably best if I simplify the question: How does
> Kafka community run tests besides using mocked local Kafka components?
> Surely there are tests to confirm different failure scenarios such as
> losing a broker in a real clustered environment (multi node cluster with
> Ip, port, hostnsmes and etc). The answer would be a good starting point for
> me.
>
> On Wed, Dec 26, 2018 at 6:11 PM Stephen Powis <spowis@salesforce.com>
> wrote:
>
>> Without looking into how the integration tests work my best guess is
>> within
>> the context they were written to run in, it doesn't make sense to run them
>> against a remote cluster.  The "internal" cluster is running the same
>> code,
>> so why require having to coordinate with an external dependency?
>>
>> For the use case you gave, and I'm not sure if tests exist that cover this
>> behavior or not -- Running the brokers locally in the context of the tests
>> mean that those tests have control over the brokers (IE shut them off,
>> restart them, etc.. programmatically) and validate behavior.  To
>> coordinate
>> these operations on a remote broker would be significantly more difficult.
>>
>> Not sure this helps...but perhaps you're either asking the wrong questions
>> or trying to go about solving your problem using the wrong set of tools?
>> My gut feeling says if you want to do a full scale multi-server load / HA
>> test, Kafka's test suite is not the best place to start.
>>
>> Stephen
>>
>>
>>
>> On Thu, Dec 27, 2018 at 10:53 AM Parviz deyhim <deyhim@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > I'm looking to see who has done this before and get some guidance. On
>> > frequent basis I like to run basic tests on a remote Kafka cluster while
>> > some random chaos/faults are being performed. In other words I like to
>> run
>> > chaos engineering tasks (network outage, disk outage, etc) and see how
>> > Kafka behaves. For example:
>> >
>> > 1) bring some random Broker node down
>> > 2) send 2000 messages
>> > 3) consumes messages
>> > 4) confirm there's no data loss
>> >
>> > My questions: I'm pretty sure most of the scenarios I'm looking to test
>> > have been covered under Kafka's integration,unit and other existing
>> tests.
>> > What I cannot figure out is how to run those tests on a remote cluster
>> vs.
>> > a local one which the tests seems to run on. For example I like to run
>> the
>> > following command but the tests to be executed on a remote cluster of my
>> > choice:
>> >
>> > ./gradlew cleanTest integrationTest
>> >
>> > Any guidance/help would be appreciated.
>> >
>> > Thanks
>> >
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message