kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Behrang <behran...@gmail.com>
Subject Re: Consumer group intermittently can not read any records from a cluster with 3 nodes that has one node down
Date Tue, 20 Feb 2018 11:50:32 GMT
Hi Sandor,

Thanks for your reply. I am not at work right now, but I still am a bit
confused about what happened at work:

1- One thing that I confirmed was that one the 3 nodes was definitely down.
We were unable to telnet into its Kafka port from anywhere. The other two
nodes were up and we could telnet into their Kafka port.

2- I modified my app a bit and implemented a means for sending
DescribeCluster requests to the cluster, setting bootrstrap-servers to all
the 3 nodes. The result indicated that the controller node (
https://kafka.apache.org/0110/javadoc/org/apache/kafka/clients/admin/DescribeClusterResult.html#controller())
had an id that was not amongst the nodes (
https://kafka.apache.org/0110/javadoc/org/apache/kafka/clients/admin/DescribeClusterResult.html#nodes()).
It was the same node that was down (i.e. I could telnet into the other
nodes but not the controller node). And this was always the same, even
after a few minutes, the controller node's id was still the same.

3- Despite that, when running my app from my machine, I could get records
from the topics I had subscribed to, but from another machine, no records
were getting sent to the app. The app running on the other machine had a
different consumer groups though.

4- The cluster had three nodes and when the controller node was done, most
of the time I was getting a message like this: *"Connection to node -N
could not be established. Broker may not be available."* where N was either
-1, -2, or -3 but at one point in my app's logs I found a handful of
entries in which N was a very large number (e.g. 2156987456).

I assume our cluster was misbehaving, but still can't explain why my app
was working like this.


Best regards,
Behrang Saeedzadeh

On 20 February 2018 at 19:22, Sandor Murakozi <smurakozi@gmail.com> wrote:

> Hi Behrang,
>
> All reads and writes of a partition go through the leader of that
> partition.
> If the leader of a partition is down you will not be able to
> produce/consume data in it until a new leader is elected. Typically it
> happens in a few seconds, after that you should be able to use that
> partition again. If your problem persists I recommend figuring out why
> leader election does not happen.
> You might be able to work with other partitions, at least those that have
> leaders on brokers that are up.
>
> Cheers,
> Sandor Murakozi
>
> On Tue, Feb 20, 2018 at 9:00 AM, Behrang <behrangsa@gmail.com> wrote:
>
> > Hi,
> >
> > I have a Kafka cluster with 3 nodes.
> >
> > I pass the nodes in the cluster to a consumer app I am building as
> > bootstrap servers.
> >
> > When one of the nodes in the cluster is down, the consumer group
> sometimes
> > CAN read records from the server but sometimes CAN NOT.
> >
> > In both cases, the same Kafka node is down.
> >
> > Is this behavior normal? Isn't it enough to only have one of the nodes in
> > the Kafka cluster be up and running? I have not delved much into setup
> and
> > administration of Kafka clusters, but I thought Kafka uses the nodes for
> HA
> > and as long as one node is up and running, the cluster remains healthy
> and
> > working.
> >
> > Best regards,
> > Behrang Saeedzadeh
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message