kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Parviz deyhim <dey...@gmail.com>
Subject Re: Kafka tests on a remote cluster
Date Thu, 27 Dec 2018 04:53:01 GMT
Thanks fair points. Probably best if I simplify the question: How does
Kafka community run tests besides using mocked local Kafka components?
Surely there are tests to confirm different failure scenarios such as
losing a broker in a real clustered environment (multi node cluster with
Ip, port, hostnsmes and etc). The answer would be a good starting point for
me.

On Wed, Dec 26, 2018 at 6:11 PM Stephen Powis <spowis@salesforce.com> wrote:

> Without looking into how the integration tests work my best guess is within
> the context they were written to run in, it doesn't make sense to run them
> against a remote cluster.  The "internal" cluster is running the same code,
> so why require having to coordinate with an external dependency?
>
> For the use case you gave, and I'm not sure if tests exist that cover this
> behavior or not -- Running the brokers locally in the context of the tests
> mean that those tests have control over the brokers (IE shut them off,
> restart them, etc.. programmatically) and validate behavior.  To coordinate
> these operations on a remote broker would be significantly more difficult.
>
> Not sure this helps...but perhaps you're either asking the wrong questions
> or trying to go about solving your problem using the wrong set of tools?
> My gut feeling says if you want to do a full scale multi-server load / HA
> test, Kafka's test suite is not the best place to start.
>
> Stephen
>
>
>
> On Thu, Dec 27, 2018 at 10:53 AM Parviz deyhim <deyhim@gmail.com> wrote:
>
> > Hi,
> >
> > I'm looking to see who has done this before and get some guidance. On
> > frequent basis I like to run basic tests on a remote Kafka cluster while
> > some random chaos/faults are being performed. In other words I like to
> run
> > chaos engineering tasks (network outage, disk outage, etc) and see how
> > Kafka behaves. For example:
> >
> > 1) bring some random Broker node down
> > 2) send 2000 messages
> > 3) consumes messages
> > 4) confirm there's no data loss
> >
> > My questions: I'm pretty sure most of the scenarios I'm looking to test
> > have been covered under Kafka's integration,unit and other existing
> tests.
> > What I cannot figure out is how to run those tests on a remote cluster
> vs.
> > a local one which the tests seems to run on. For example I like to run
> the
> > following command but the tests to be executed on a remote cluster of my
> > choice:
> >
> > ./gradlew cleanTest integrationTest
> >
> > Any guidance/help would be appreciated.
> >
> > Thanks
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message