kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ankit tyagi <ankittyagi.mn...@gmail.com>
Subject too many rebalances in my consumer log and fetcher threads are getting stopped
Date Fri, 18 Apr 2014 10:58:43 GMT
Hi,

I am seeing consumer re-balances very frequently and getting socket
reconnect exception. log is given below for more insights


[2014-04-18
16:02:52.061][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][SyncProducer:67]
Disconnecting from kafka_leader_coms07.snapdeal.com:9092
[2014-04-18
16:02:52.103][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager:67]
[ConsumerFetcherManager-1397812122507] *Stopping leader finder thread*
[2014-04-18
16:02:52.104][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
[kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
Shutting down
[2014-04-18
16:02:52.105][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
[kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
Shutdown completed
[2014-04-18
16:02:52.106][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager:67]
[ConsumerFetcherManager-1397812122507]* Stopping all fetchers*
[2014-04-18
16:02:52.106][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
[ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
Shutting down
[2014-04-18
16:02:52.107][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-*timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0][INFO][SimpleConsumer:75]
Reconnect due to socket error: *
*java.nio.channels.ClosedByInterruptException*
* at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)*
* at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:386)*
* at
sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:220)*
* at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)*
at
java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
at kafka.utils.Utils$.read(Utils.scala:394)
at
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67)
at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
at
kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:73)
at
kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:110)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:109)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:108)
at
kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
[2014-04-18
16:02:52.108][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0][INFO][ConsumerFetcherThread:67]
[ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
Stopped
[2014-04-18
16:02:52.109][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-*1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
[ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
Shutdown completed*
[2014-04-18
16:02:52.171][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
[ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-2],
Shutting down
[2014-04-18
16:02:52.109][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
[kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
Stopped
[*2014-04-18
16:02:52.171][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-2][INFO][SimpleConsumer:75]
Reconnect due to socket error: *
*java.nio.channels.ClosedByInterruptException*
* at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)*
* at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:386)*
* at
sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:220)*
* at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)*
at
java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
at kafka.utils.Utils$.read(Utils.scala:394)
at
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67)
at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
at
kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:73)
at
kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:110)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:109)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:108)
at
kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)


currently we are using 3 node kafka cluster(0.8.0_beta) with 1 topic of 100
partition. According to kakfa documentation, GC maybe the reason of too may
rebalances so i monitored my app with jstat, couldn't any issue over there.

[coms@coms04 coms-timemachine]$ jstat -gcutil 7419 1000 100
  S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT
 35.09   8.88 100.00  66.89  90.73   1119  820.300   102 1093.934 1914.234
  0.00  59.06  21.02  70.93  90.89   1119  821.362   102 1093.934 1915.296
  0.00  59.06  84.61  70.93  91.85   1119  821.362   102 1093.934 1915.296
 35.08  59.06 100.00  79.29  92.00   1120  821.362   102 1093.934 1915.296
 53.67   0.00  43.22  80.63  92.55   1120  822.511   102 1093.934 1916.445
 53.67   0.00  89.50  80.63  93.55   1120  822.511   102 1093.934 1916.445
 53.67  31.42 100.00  86.79  93.97   1121  822.511   102 1093.934 1916.445
  0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934 1917.514
  0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934 1917.514
  0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934 1917.514
  0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934 1917.514
  0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934 1917.514
  0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934 1917.514
  0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934 1917.514
  0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934 1917.514
  0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934 1917.514
  0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934 1917.514
  0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934 1917.514
  0.00   0.00  65.02  41.18  90.02   1121  823.580   103 1104.702 1928.282
 36.43   0.00   4.34  41.18  90.58   1122  823.979   103 1104.702 1928.681
 36.43   0.00  69.49  41.18  91.02   1122  823.979   103 1104.702 1928.681
 36.43  26.60 100.00  42.34  91.20   1123  823.979   103 1104.702 1928.681
  0.00  35.07  30.98  46.02  91.41   1123  824.749   103 1104.702 1929.451
  0.00  35.07  86.93  46.02  91.60   1123  824.749   103 1104.702 1929.451
  1.60   0.00  14.58  51.85  91.74   1124  825.198   103 1104.702 1929.900
  1.60   0.00  62.30  51.85  91.98   1124  825.198   103 1104.702 1929.900
  0.00   9.65   6.49  52.08  92.19   1125  825.360   103 1104.702 1930.063
  0.00   9.65  58.28  52.08  92.47   1125  825.360   103 1104.702 1930.063
 19.19   9.65 100.00  53.38  92.63   1126  825.360   103 1104.702 1930.063
 25.31   0.00  51.89  53.48  92.87   1126  825.784   103 1104.702 1930.487
 25.31  11.21 100.00  55.24  93.04   1127  825.784   103 1104.702 1930.487
  0.00  99.97  39.63  57.69  93.22   1127  826.513   103 1104.702 1931.216
  0.00  99.97  86.22  57.69  93.40   1127  826.513   103 1104.702 1931.216
 72.36  99.97 100.00  57.69  93.51   1128  826.513   103 1104.702 1931.216
 85.99   0.00  44.82  57.69  93.76   1128  827.339   103 1104.702 1932.041
 85.99   0.83 100.00  66.34  94.09   1129  827.339   103 1104.702 1932.041
  0.00  36.82  12.21  73.88  94.17   1129  828.588   103 1104.702 1933.290
  0.00  36.82 100.00  73.88  94.94   1130  828.588   103 1104.702 1933.290
 47.42   0.00   5.23  79.99  95.02   1130  829.497   103 1104.702 1934.199
 47.42   0.00  69.50  79.99  96.14   1130  829.497   103 1104.702 1934.199
 47.42  19.36 100.00  80.78  97.13   1131  829.497   103 1104.702 1934.199
  0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702 1935.156
  0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702 1935.156
  0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702 1935.156
  0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702 1935.156
  0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702 1935.156
  0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702 1935.156
  0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702 1935.156
  0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702 1935.156
  0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702 1935.156
  0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702 1935.156
  0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702 1935.156
  0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702 1935.156
  0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702 1935.156
  0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702 1935.156
  0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702 1935.156
  0.00   0.00  33.49  59.22  89.53   1131  830.454   104 1119.378 1949.831
  0.00   0.00  87.06  59.22  90.67   1131  830.454   104 1119.378 1949.831
 24.49   0.00  19.01  59.22  91.22   1132  830.770   104 1119.378 1950.147
 24.49   0.00  72.05  59.22  91.46   1132  830.770   104 1119.378 1950.147
 24.49  39.18 100.00  59.22  91.64   1133  830.770   104 1119.378 1950.147
  0.00  99.99  44.95  59.22  91.82   1133  831.338   104 1119.378 1950.715
  0.00  99.99  94.30  59.22  92.17   1133  831.338   104 1119.378 1950.715
 58.40   0.00  11.63  59.22  92.22   1134  831.993   104 1119.378 1951.370
 58.40   0.00  58.77  59.22  92.46   1134  831.993   104 1119.378 1951.370
 58.40  27.41 100.00  59.22  92.55   1135  831.993   104 1119.378 1951.370
  0.00  63.03  30.83  59.22  92.71   1135  832.675   104 1119.378 1952.053
  0.00  63.03  81.31  59.22  92.94   1135  832.675   104 1119.378 1952.053
 55.01  63.03 100.00  59.22  92.98   1136  832.675   104 1119.378 1952.053
 79.90   0.00  37.84  59.22  93.14   1136  833.582   104 1119.378 1952.960
 79.90   0.00  92.75  59.22  93.24   1136  833.582   104 1119.378 1952.960
 79.90  69.59 100.00  59.22  93.30   1137  833.582   104 1119.378 1952.960
  0.00 100.00  54.86  59.77  93.65   1137  834.765   104 1119.378 1954.143
 31.63 100.00  98.64  60.31  94.04   1138  834.765   104 1119.378 1954.143
100.00 100.00  98.64  63.88  94.04   1138  834.765   104 1119.378 1954.143
100.00   0.00  50.43  65.69  94.50   1138  836.225   104 1119.378 1955.603
100.00  14.88 100.00  66.31  95.25   1139  836.225   104 1119.378 1955.603
100.00 100.00 100.00  67.98  95.25   1139  836.225   104 1119.378 1955.603
  0.00 100.00  43.37  72.29  95.82   1139  837.636   104 1119.378 1957.014
 24.55 100.00 100.00  76.41  96.40   1140  837.636   104 1119.378 1957.014
 93.50 100.00 100.00  80.50  96.40   1140  837.636   104 1119.378 1957.014
100.00   0.00  62.48  82.20  96.79   1140  839.089   104 1119.378 1958.467
100.00  22.78 100.00  88.93  97.05   1141  839.089   104 1119.378 1958.467
  0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378 1959.958
  0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378 1959.958
  0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378 1959.958
  0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378 1959.958
  0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378 1959.958
  0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378 1959.958
  0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378 1959.958
  0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378 1959.958
  0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378 1959.958
  0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378 1959.958
  0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378 1959.958
  0.00   0.00  53.78  46.66  89.50   1141  840.580   105 1129.520 1970.101
  0.27   0.00 100.00  46.66  89.75   1142  840.580   105 1129.520 1970.101
 26.45   0.00  33.61  46.66  89.77   1142  840.924   105 1129.520 1970.444
 26.45   0.00  84.87  46.66  89.81   1142  840.924   105 1129.520 1970.444
  0.00   2.68  16.33  50.87  89.83   1143  841.276   105 1129.520 1970.796
  0.00   2.68  63.77  50.87  89.87   1143  841.276   105 1129.520 1970.796


Not sure what is happening over here. any leads would be helpful..

Regards,
Ankit TYagi

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message