kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Praveen Kumar K S <prav...@securelyshare.com>
Subject Re: Client session timed out
Date Tue, 20 Oct 2020 18:07:18 GMT
Hello,

I'm not sure if I can add my issue in this thread. But it seems like I'm
facing the same problem.

KAFKA_VERSION=2.5.1
ZK_VERSION=3.5.8

I run 3 node zookeeper cluster and 3 node kafka cluster as docker
containers in docker swarm environment. When I install it for first time,
everything goes well. Zookeeper and Kafka are able to form the cluster.
Services are healthy.

But when I issue docker update command, kafka is not coming up though the
zookeeper cluster is healthy. Below is the sequence of steps.



*docker service update one_zookeeper --image x.x.x/v1/zookeeper:latest
--forcedocker service update one_zookeeper1 --image
x.x.x/v1/zookeeper:latest --forcedocker service update one_zookeeper2
--image x.x.x/v1/zookeeper:latest --force*

Zookeeper is healthy now. I'm able to query leader and follower.

Now, I'm updating kafka and it doesn't work.
docker service update one_kafka --image x.x.com/v1/kafka:latest --force

PFA Kafka log.

While kafka update has failed, I see that kafka1 and kafka2 are running and
healthy.


*docker service ls | grep kafka*
*one_kafka            replicated          0/1*
*one_kafka1           replicated          1/1*
*one_kafka2           replicated          1/1*

To cross verify, I have just brought down the services zookeeper and kafka
without data loss. I preserve zookeeperdata,zookeeperlogs,zookeepertxns and
kafkadata,kafkalogs.

*docker stack remove one*
*docker stack deploy -c cluster-zookeeper.yml one*
*docker stack deploy -c cluster-kafka.yml one*

Now, all the services are healthy.

I'm not sure why kafka deployment is failing only during update. There is
no change in the configuration in either zookeeper or kafka.

Please help me resolve this issue and let me know if you need any
additional details.

Regards,
Praveen Kumar K S
+91-9986855625


On Tue, Oct 20, 2020 at 3:54 PM Sabina Marx <sabina.marx@sneo.io> wrote:

> Yes, it's the same problem.
>
> Am 19.10.20, 19:50 schrieb "Mich Talebzadeh" <mich.talebzadeh@gmail.com>:
>
>     can you try to disable automatic start and on the node just booted,
> start
>     zookeeper first, check the log that it is connected and then start
> Kafka?
>
>     I assume everything is set-up OK including in
>     $KAFKA_HOME/config/server<N>.properties values for broker.id,
> hostname,
>     zookeeper.connect=<server1>:2181,,server2>:2181, <serverN>:2181  and
> also
>     zookeeper.connection.timeout.ms=6000 (default)
>
>     HTH
>
>
>
>
>
>     LinkedIn *
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>     <
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> >*
>
>
>
>
>
>     *Disclaimer:* Use it at your own risk. Any and all responsibility for
> any
>     loss, damage or destruction of data or any other property which may
> arise
>     from relying on this email's technical content is explicitly
> disclaimed.
>     The author will in no case be liable for any monetary damages arising
> from
>     such loss, damage or destruction.
>
>
>
>
>     On Mon, 19 Oct 2020 at 18:17, Sabina Marx <sabina.marx@sneo.io> wrote:
>
>     > Yes, you have it
>     >
>     > Holen Sie sich Outlook für iOS<https://aka.ms/o0ukef>
>     > ________________________________
>     > Von: Mich Talebzadeh <mich.talebzadeh@gmail.com>
>     > Gesendet: Monday, October 19, 2020 7:09:53 PM
>     > An: users@kafka.apache.org <users@kafka.apache.org>
>     > Betreff: Re: Client session timed out
>     >
>     > Ok I think it is clearer now.
>     >
>     > As I understand all your Zookeepers and Kafkas are running. (5 nodes
>     > meaning 5 physical hosts?). Then you have to reboot one physical
> host. You
>     > still have the redundancy. But when the physical host comes up and
> your
>     > zookeeper and then Kafka come up, you have Kafka timing out and not
>     > connecting to the existing Kafka cluster?
>     >
>     > Does that make sense?
>     >
>     >
>     >
>     >
>     >
>     >
>     > LinkedIn *
>     >
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>     > <
>     >
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>     > >*
>     >
>     >
>     >
>     >
>     >
>     > *Disclaimer:* Use it at your own risk. Any and all responsibility
> for any
>     > loss, damage or destruction of data or any other property which may
> arise
>     > from relying on this email's technical content is explicitly
> disclaimed.
>     > The author will in no case be liable for any monetary damages
> arising from
>     > such loss, damage or destruction.
>     >
>     >
>     >
>     >
>     > On Mon, 19 Oct 2020 at 17:59, Sabina Marx <sabina.marx@sneo.io>
> wrote:
>     >
>     > > No, sorry I‘m not so good in explaining.
>     > > The scenario is: the complete Cluster is running, all zookeepers
> and all
>     > > kafkas. And then I restart one server, the others are still
> running.
>     > >
>     > > Holen Sie sich Outlook für iOS<https://aka.ms/o0ukef>
>     > > ________________________________
>     > > Von: Mich Talebzadeh <mich.talebzadeh@gmail.com>
>     > > Gesendet: Monday, October 19, 2020 6:46:49 PM
>     > > An: users@kafka.apache.org <users@kafka.apache.org>
>     > > Betreff: Re: Client session timed out
>     > >
>     > > Can you pls clarify when you say you start one pair (Zookeeper and
>     > Kafka),
>     > > what happens to the others. Do you keep them down?
>     > >
>     > >
>     > >
>     > >
>     > > LinkedIn *
>     > >
>     >
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>     > > <
>     > >
>     >
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>     > > >*
>     > >
>     > >
>     > >
>     > >
>     > >
>     > > *Disclaimer:* Use it at your own risk. Any and all responsibility
> for any
>     > > loss, damage or destruction of data or any other property which
> may arise
>     > > from relying on this email's technical content is explicitly
> disclaimed.
>     > > The author will in no case be liable for any monetary damages
> arising
>     > from
>     > > such loss, damage or destruction.
>     > >
>     > >
>     > >
>     > >
>     > > On Mon, 19 Oct 2020 at 17:33, Sabina Marx <sabina.marx@sneo.io>
> wrote:
>     > >
>     > > > Yes, it's a new setup. The kafka and zookeeper runs as services
> and are
>     > > > enabled, so they should start at system start. And kafka starts
> after
>     > > > zookeeper.
>     > > > And if I stop everything and start it, then it works. But then,
> if I
>     > > > restart one server with zookeeper and kafka the kafka gets the
>     > timeouts.
>     > > >
>     > > > Am 19.10.20, 18:16 schrieb "Mich Talebzadeh" <
>     > mich.talebzadeh@gmail.com
>     > > >:
>     > > >
>     > > >     OK so the issue seems to be kafka cluster. Is this a new
> setup?
>     > > >
>     > > >     HTH
>     > > >
>     > > >
>     > > >
>     > > >     LinkedIn *
>     > > >
>     > >
>     >
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>     > > >     <
>     > > >
>     > >
>     >
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>     > > > >*
>     > > >
>     > > >
>     > > >
>     > > >
>     > > >
>     > > >     *Disclaimer:* Use it at your own risk. Any and all
> responsibility
>     > for
>     > > > any
>     > > >     loss, damage or destruction of data or any other property
> which may
>     > > > arise
>     > > >     from relying on this email's technical content is explicitly
>     > > > disclaimed.
>     > > >     The author will in no case be liable for any monetary damages
>     > arising
>     > > > from
>     > > >     such loss, damage or destruction.
>     > > >
>     > > >
>     > > >
>     > > >
>     > > >     On Mon, 19 Oct 2020 at 17:11, Sabina Marx <
> sabina.marx@sneo.io>
>     > > wrote:
>     > > >
>     > > >     > Thanks for your answer, but the zookeeper ensemble is
> running and
>     > > > ports
>     > > >     > are ok.
>     > > >     >
>     > > >     > Am 19.10.20, 17:38 schrieb "Mich Talebzadeh" <
>     > > > mich.talebzadeh@gmail.com>:
>     > > >     >
>     > > >     >     Start the zookeeper ensemble first before starting
> Kafka
>     > > > cluster. They
>     > > >     > need
>     > > >     >     to select a leader and ensure that they all come
> online OK.
>     > > > Check port
>     > > >     >     2181, 2888, 3888 using
>     > > >     >
>     > > >     >      netstat -plten|egrep '2181|2888|3888'
>     > > >     >
>     > > >     >     tcp        0      0 :::2181                     :::*
>     > > >     >     LISTEN      1005       9934134    29170/java
>     > > >     >     tcp        0      0 ::ffff:50.140.197.217:2888  :::*
>     > > >     >     LISTEN      1005       9935496    29170/java
>     > > >     >     tcp        0      0 ::ffff:50.140.197.217:3888  :::*
>     > > >     >     LISTEN      1005       9935493    29170/java
>     > > >     >
>     > > >     >
>     > > >     >
>     > > >     >     P.S. I assume you are talking about Apache Kafks here.
>     > > >     >
>     > > >     >     HTH
>     > > >     >
>     > > >     >
>     > > >     >
>     > > >     >     LinkedIn *
>     > > >     >
>     > > >
>     > >
>     >
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>     > > >     >     <
>     > > >     >
>     > > >
>     > >
>     >
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>     > > >     > >*
>     > > >     >
>     > > >     >
>     > > >     >
>     > > >     >
>     > > >     >
>     > > >     >     *Disclaimer:* Use it at your own risk. Any and all
>     > > > responsibility for
>     > > >     > any
>     > > >     >     loss, damage or destruction of data or any other
> property
>     > which
>     > > > may
>     > > >     > arise
>     > > >     >     from relying on this email's technical content is
> explicitly
>     > > >     > disclaimed.
>     > > >     >     The author will in no case be liable for any monetary
> damages
>     > > > arising
>     > > >     > from
>     > > >     >     such loss, damage or destruction.
>     > > >     >
>     > > >     >
>     > > >     >
>     > > >     >
>     > > >     >     On Mon, 19 Oct 2020 at 14:42, Sabina Marx <
>     > sabina.marx@sneo.io
>     > > >
>     > > > wrote:
>     > > >     >
>     > > >     >     > Hi,
>     > > >     >     >
>     > > >     >     > I have a 5 nodes kafka cluster with 3 zookeepers.
If
> I
>     > > restart
>     > > > 1 node
>     > > >     >     > (zookeeper and kafka) the kafka gets a Client
> session timed
>     > > > out,
>     > > >     > have not
>     > > >     >     > heard from server in 6007ms for sessionid 0x0
>     > > >     >     > (org.apache.zookeeper.ClientCnxn)
>     > > >     >     > Client session timed out, have not heard from server
> in
>     > > 6007ms
>     > > > for
>     > > >     >     > sessionid 0x0, closing socket connection and
> attempting
>     > > > reconnect
>     > > >     >     > (org.apache.zookeeper.ClientCnxn)
>     > > >     >     > And my kafka service do not start.
>     > > >     >     > I have set the tickTime=6000 in the
> zookeeper.properties
>     > but
>     > > > that
>     > > >     > didn’t
>     > > >     >     > help. What can I do?
>     > > >     >     >
>     > > >     >     > Many thanks for your help.
>     > > >     >     > Sabina
>     > > >     >     >
>     > > >     >
>     > > >     >
>     > > >
>     > > >
>     > >
>     >
>
>

Mime
View raw message