kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Praveen Kumar K S <prav...@securelyshare.com>
Subject Re: Client session timed out
Date Fri, 23 Oct 2020 08:48:16 GMT
Hello,

Can someone please help me to understand what is the issue ?

Regards,
Praveen Kumar K S
+91-9986855625


On Thu, Oct 22, 2020 at 6:52 AM Praveen Kumar K S <praveen@securelyshare.com>
wrote:

> Hello Experts,
>
> Any help to debug and resolve this issue is highly appreciated.
>
> Regards,
> Praveen
>
> On Wed, 21 Oct, 2020, 11:26 Sabina Marx, <sabina.marx@sneo.io> wrote:
>
>> Hi Praveen,
>>
>> it seems to be the same problem, your log looks quite similar to mine.
>> But I have no solution until now.
>>
>> Regards
>> Sabina
>>
>> Von: Praveen Kumar K S <praveen@securelyshare.com>
>> Antworten an: "users@kafka.apache.org" <users@kafka.apache.org>
>> Datum: Dienstag, 20. Oktober 2020 um 20:07
>> An: "users@kafka.apache.org" <users@kafka.apache.org>
>> Betreff: Re: Client session timed out
>>
>> Hello,
>>
>> I'm not sure if I can add my issue in this thread. But it seems like I'm
>> facing the same problem.
>>
>> KAFKA_VERSION=2.5.1
>> ZK_VERSION=3.5.8
>>
>> I run 3 node zookeeper cluster and 3 node kafka cluster as docker
>> containers in docker swarm environment. When I install it for first time,
>> everything goes well. Zookeeper and Kafka are able to form the cluster.
>> Services are healthy.
>>
>> But when I issue docker update command, kafka is not coming up though the
>> zookeeper cluster is healthy. Below is the sequence of steps.
>>
>> docker service update one_zookeeper --image x.x.x/v1/zookeeper:latest
>> --force
>> docker service update one_zookeeper1 --image x.x.x/v1/zookeeper:latest
>> --force
>> docker service update one_zookeeper2 --image x.x.x/v1/zookeeper:latest
>> --force
>>
>> Zookeeper is healthy now. I'm able to query leader and follower.
>>
>> Now, I'm updating kafka and it doesn't work.
>> docker service update one_kafka --image x.x.com/v1/kafka:latest<
>> http://x.x.com/v1/kafka:latest> --force
>>
>> PFA Kafka log.
>>
>> While kafka update has failed, I see that kafka1 and kafka2 are running
>> and healthy.
>>
>> docker service ls | grep kafka
>> one_kafka            replicated          0/1
>> one_kafka1           replicated          1/1
>> one_kafka2           replicated          1/1
>>
>> To cross verify, I have just brought down the services zookeeper and
>> kafka without data loss. I preserve
>> zookeeperdata,zookeeperlogs,zookeepertxns and kafkadata,kafkalogs.
>>
>> docker stack remove one
>> docker stack deploy -c cluster-zookeeper.yml one
>> docker stack deploy -c cluster-kafka.yml one
>>
>> Now, all the services are healthy.
>>
>> I'm not sure why kafka deployment is failing only during update. There is
>> no change in the configuration in either zookeeper or kafka.
>>
>> Please help me resolve this issue and let me know if you need any
>> additional details.
>>
>> Regards,
>> Praveen Kumar K S
>> +91-9986855625
>>
>>
>> On Tue, Oct 20, 2020 at 3:54 PM Sabina Marx <sabina.marx@sneo.io<mailto:
>> sabina.marx@sneo.io>> wrote:
>> Yes, it's the same problem.
>>
>> Am 19.10.20, 19:50 schrieb "Mich Talebzadeh" <mich.talebzadeh@gmail.com
>> <mailto:mich.talebzadeh@gmail.com>>:
>>
>>     can you try to disable automatic start and on the node just booted,
>> start
>>     zookeeper first, check the log that it is connected and then start
>> Kafka?
>>
>>     I assume everything is set-up OK including in
>>     $KAFKA_HOME/config/server<N>.properties values for broker.id<
>> http://broker.id>, hostname,
>>     zookeeper.connect=<server1>:2181,,server2>:2181, <serverN>:2181
 and
>> also
>>     zookeeper.connection.timeout.ms<
>> http://zookeeper.connection.timeout.ms>=6000 (default)
>>
>>     HTH
>>
>>
>>
>>
>>
>>     LinkedIn *
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>     <
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> >*
>>
>>
>>
>>
>>
>>     *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any
>>     loss, damage or destruction of data or any other property which may
>> arise
>>     from relying on this email's technical content is explicitly
>> disclaimed.
>>     The author will in no case be liable for any monetary damages arising
>> from
>>     such loss, damage or destruction.
>>
>>
>>
>>
>>     On Mon, 19 Oct 2020 at 18:17, Sabina Marx <sabina.marx@sneo.io
>> <mailto:sabina.marx@sneo.io>> wrote:
>>
>>     > Yes, you have it
>>     >
>>     > Holen Sie sich Outlook für iOS<https://aka.ms/o0ukef>
>>     > ________________________________
>>     > Von: Mich Talebzadeh <mich.talebzadeh@gmail.com<mailto:
>> mich.talebzadeh@gmail.com>>
>>     > Gesendet: Monday, October 19, 2020 7:09:53 PM
>>     > An: users@kafka.apache.org<mailto:users@kafka.apache.org> <
>> users@kafka.apache.org<mailto:users@kafka.apache.org>>
>>     > Betreff: Re: Client session timed out
>>     >
>>     > Ok I think it is clearer now.
>>     >
>>     > As I understand all your Zookeepers and Kafkas are running. (5 nodes
>>     > meaning 5 physical hosts?). Then you have to reboot one physical
>> host. You
>>     > still have the redundancy. But when the physical host comes up and
>> your
>>     > zookeeper and then Kafka come up, you have Kafka timing out and not
>>     > connecting to the existing Kafka cluster?
>>     >
>>     > Does that make sense?
>>     >
>>     >
>>     >
>>     >
>>     >
>>     >
>>     > LinkedIn *
>>     >
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>     > <
>>     >
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>     > >*
>>     >
>>     >
>>     >
>>     >
>>     >
>>     > *Disclaimer:* Use it at your own risk. Any and all responsibility
>> for any
>>     > loss, damage or destruction of data or any other property which may
>> arise
>>     > from relying on this email's technical content is explicitly
>> disclaimed.
>>     > The author will in no case be liable for any monetary damages
>> arising from
>>     > such loss, damage or destruction.
>>     >
>>     >
>>     >
>>     >
>>     > On Mon, 19 Oct 2020 at 17:59, Sabina Marx <sabina.marx@sneo.io
>> <mailto:sabina.marx@sneo.io>> wrote:
>>     >
>>     > > No, sorry I‘m not so good in explaining.
>>     > > The scenario is: the complete Cluster is running, all zookeepers
>> and all
>>     > > kafkas. And then I restart one server, the others are still
>> running.
>>     > >
>>     > > Holen Sie sich Outlook für iOS<https://aka.ms/o0ukef>
>>     > > ________________________________
>>     > > Von: Mich Talebzadeh <mich.talebzadeh@gmail.com<mailto:
>> mich.talebzadeh@gmail.com>>
>>     > > Gesendet: Monday, October 19, 2020 6:46:49 PM
>>     > > An: users@kafka.apache.org<mailto:users@kafka.apache.org> <
>> users@kafka.apache.org<mailto:users@kafka.apache.org>>
>>     > > Betreff: Re: Client session timed out
>>     > >
>>     > > Can you pls clarify when you say you start one pair (Zookeeper and
>>     > Kafka),
>>     > > what happens to the others. Do you keep them down?
>>     > >
>>     > >
>>     > >
>>     > >
>>     > > LinkedIn *
>>     > >
>>     >
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>     > > <
>>     > >
>>     >
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>     > > >*
>>     > >
>>     > >
>>     > >
>>     > >
>>     > >
>>     > > *Disclaimer:* Use it at your own risk. Any and all responsibility
>> for any
>>     > > loss, damage or destruction of data or any other property which
>> may arise
>>     > > from relying on this email's technical content is explicitly
>> disclaimed.
>>     > > The author will in no case be liable for any monetary damages
>> arising
>>     > from
>>     > > such loss, damage or destruction.
>>     > >
>>     > >
>>     > >
>>     > >
>>     > > On Mon, 19 Oct 2020 at 17:33, Sabina Marx <sabina.marx@sneo.io
>> <mailto:sabina.marx@sneo.io>> wrote:
>>     > >
>>     > > > Yes, it's a new setup. The kafka and zookeeper runs as services
>> and are
>>     > > > enabled, so they should start at system start. And kafka starts
>> after
>>     > > > zookeeper.
>>     > > > And if I stop everything and start it, then it works. But then,
>> if I
>>     > > > restart one server with zookeeper and kafka the kafka gets the
>>     > timeouts.
>>     > > >
>>     > > > Am 19.10.20, 18:16 schrieb "Mich Talebzadeh" <
>>     > mich.talebzadeh@gmail.com<mailto:mich.talebzadeh@gmail.com>
>>     > > >:
>>     > > >
>>     > > >     OK so the issue seems to be kafka cluster. Is this a new
>> setup?
>>     > > >
>>     > > >     HTH
>>     > > >
>>     > > >
>>     > > >
>>     > > >     LinkedIn *
>>     > > >
>>     > >
>>     >
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>     > > >     <
>>     > > >
>>     > >
>>     >
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>     > > > >*
>>     > > >
>>     > > >
>>     > > >
>>     > > >
>>     > > >
>>     > > >     *Disclaimer:* Use it at your own risk. Any and all
>> responsibility
>>     > for
>>     > > > any
>>     > > >     loss, damage or destruction of data or any other property
>> which may
>>     > > > arise
>>     > > >     from relying on this email's technical content is explicitly
>>     > > > disclaimed.
>>     > > >     The author will in no case be liable for any monetary
>> damages
>>     > arising
>>     > > > from
>>     > > >     such loss, damage or destruction.
>>     > > >
>>     > > >
>>     > > >
>>     > > >
>>     > > >     On Mon, 19 Oct 2020 at 17:11, Sabina Marx <
>> sabina.marx@sneo.io<mailto:sabina.marx@sneo.io>>
>>     > > wrote:
>>     > > >
>>     > > >     > Thanks for your answer, but the zookeeper ensemble is
>> running and
>>     > > > ports
>>     > > >     > are ok.
>>     > > >     >
>>     > > >     > Am 19.10.20, 17:38 schrieb "Mich Talebzadeh" <
>>     > > > mich.talebzadeh@gmail.com<mailto:mich.talebzadeh@gmail.com>>:
>>     > > >     >
>>     > > >     >     Start the zookeeper ensemble first before starting
>> Kafka
>>     > > > cluster. They
>>     > > >     > need
>>     > > >     >     to select a leader and ensure that they all come
>> online OK.
>>     > > > Check port
>>     > > >     >     2181, 2888, 3888 using
>>     > > >     >
>>     > > >     >      netstat -plten|egrep '2181|2888|3888'
>>     > > >     >
>>     > > >     >     tcp        0      0 :::2181                     :::*
>>     > > >     >     LISTEN      1005       9934134    29170/java
>>     > > >     >     tcp        0      0 ::ffff:50.140.197.217:2888<
>> http://50.140.197.217:2888>  :::*
>>     > > >     >     LISTEN      1005       9935496    29170/java
>>     > > >     >     tcp        0      0 ::ffff:50.140.197.217:3888<
>> http://50.140.197.217:3888>  :::*
>>     > > >     >     LISTEN      1005       9935493    29170/java
>>     > > >     >
>>     > > >     >
>>     > > >     >
>>     > > >     >     P.S. I assume you are talking about Apache Kafks
here.
>>     > > >     >
>>     > > >     >     HTH
>>     > > >     >
>>     > > >     >
>>     > > >     >
>>     > > >     >     LinkedIn *
>>     > > >     >
>>     > > >
>>     > >
>>     >
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>     > > >     >     <
>>     > > >     >
>>     > > >
>>     > >
>>     >
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>     > > >     > >*
>>     > > >     >
>>     > > >     >
>>     > > >     >
>>     > > >     >
>>     > > >     >
>>     > > >     >     *Disclaimer:* Use it at your own risk. Any and all
>>     > > > responsibility for
>>     > > >     > any
>>     > > >     >     loss, damage or destruction of data or any other
>> property
>>     > which
>>     > > > may
>>     > > >     > arise
>>     > > >     >     from relying on this email's technical content is
>> explicitly
>>     > > >     > disclaimed.
>>     > > >     >     The author will in no case be liable for any monetary
>> damages
>>     > > > arising
>>     > > >     > from
>>     > > >     >     such loss, damage or destruction.
>>     > > >     >
>>     > > >     >
>>     > > >     >
>>     > > >     >
>>     > > >     >     On Mon, 19 Oct 2020 at 14:42, Sabina Marx <
>>     > sabina.marx@sneo.io<mailto:sabina.marx@sneo.io>
>>     > > >
>>     > > > wrote:
>>     > > >     >
>>     > > >     >     > Hi,
>>     > > >     >     >
>>     > > >     >     > I have a 5 nodes kafka cluster with 3 zookeepers.
>> If I
>>     > > restart
>>     > > > 1 node
>>     > > >     >     > (zookeeper and kafka) the kafka gets a Client
>> session timed
>>     > > > out,
>>     > > >     > have not
>>     > > >     >     > heard from server in 6007ms for sessionid 0x0
>>     > > >     >     > (org.apache.zookeeper.ClientCnxn)
>>     > > >     >     > Client session timed out, have not heard from
>> server in
>>     > > 6007ms
>>     > > > for
>>     > > >     >     > sessionid 0x0, closing socket connection and
>> attempting
>>     > > > reconnect
>>     > > >     >     > (org.apache.zookeeper.ClientCnxn)
>>     > > >     >     > And my kafka service do not start.
>>     > > >     >     > I have set the tickTime=6000 in the
>> zookeeper.properties
>>     > but
>>     > > > that
>>     > > >     > didn’t
>>     > > >     >     > help. What can I do?
>>     > > >     >     >
>>     > > >     >     > Many thanks for your help.
>>     > > >     >     > Sabina
>>     > > >     >     >
>>     > > >     >
>>     > > >     >
>>     > > >
>>     > > >
>>     > >
>>     >
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message