kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: too many rebalances in my consumer log and fetcher threads are getting stopped
Date Sat, 19 Apr 2014 04:01:13 GMT
Do you see any rebalances? The fetcher was stopped because it was shut
down, which typically happens during rebalances.

Thanks,

Jun


On Fri, Apr 18, 2014 at 11:08 AM, ankit tyagi <ankittyagi.mnnit@gmail.com>wrote:

> I have checked that. There was no full gc at that time. I have attached
> jstat ouput too in my mail.
>
> I have concerned over why consumer fetcher thread are getting stopped.
> On 18 Apr 2014 22:51, "Jun Rao" <junrao@gmail.com> wrote:
>
> > Have you looked at
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog
> > ?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Fri, Apr 18, 2014 at 3:58 AM, ankit tyagi <ankittyagi.mnnit@gmail.com
> > >wrote:
> >
> > > Hi,
> > >
> > > I am seeing consumer re-balances very frequently and getting socket
> > > reconnect exception. log is given below for more insights
> > >
> > >
> > > [2014-04-18
> > >
> > >
> >
> 16:02:52.061][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][SyncProducer:67]
> > > Disconnecting from kafka_leader_coms07.snapdeal.com:9092
> > > [2014-04-18
> > >
> > >
> >
> 16:02:52.103][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager:67]
> > > [ConsumerFetcherManager-1397812122507] *Stopping leader finder thread*
> > > [2014-04-18
> > >
> > >
> >
> 16:02:52.104][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
> > >
> > >
> >
> [kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
> > > Shutting down
> > > [2014-04-18
> > >
> > >
> >
> 16:02:52.105][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
> > >
> > >
> >
> [kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
> > > Shutdown completed
> > > [2014-04-18
> > >
> > >
> >
> 16:02:52.106][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager:67]
> > > [ConsumerFetcherManager-1397812122507]* Stopping all fetchers*
> > > [2014-04-18
> > >
> > >
> >
> 16:02:52.106][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
> > >
> > >
> >
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
> > > Shutting down
> > > [2014-04-18
> > >
> > >
> >
> 16:02:52.107][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-*timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0][INFO][SimpleConsumer:75]
> > > Reconnect due to socket error: *
> > > *java.nio.channels.ClosedByInterruptException*
> > > * at
> > >
> > >
> >
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)*
> > > * at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:386)*
> > > * at
> > >
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:220)*
> > > * at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)*
> > > at
> > >
> >
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> > > at kafka.utils.Utils$.read(Utils.scala:394)
> > > at
> > >
> > >
> >
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67)
> > > at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
> > > at
> > >
> > >
> >
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> > > at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
> > > at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:73)
> > > at
> > >
> > >
> >
> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71)
> > > at
> > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:110)
> > > at
> > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> > > at
> > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > at
> > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:109)
> > > at
> > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> > > at
> > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:108)
> > > at
> > >
> > >
> >
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
> > > at
> > >
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
> > > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > > [2014-04-18
> > >
> > >
> >
> 16:02:52.108][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0][INFO][ConsumerFetcherThread:67]
> > >
> > >
> >
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
> > > Stopped
> > > [2014-04-18
> > >
> > >
> >
> 16:02:52.109][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-*1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
> > >
> > >
> >
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
> > > Shutdown completed*
> > > [2014-04-18
> > >
> > >
> >
> 16:02:52.171][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
> > >
> > >
> >
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-2],
> > > Shutting down
> > > [2014-04-18
> > >
> > >
> >
> 16:02:52.109][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
> > >
> > >
> >
> [kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
> > > Stopped
> > > [*2014-04-18
> > >
> > >
> >
> 16:02:52.171][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-2][INFO][SimpleConsumer:75]
> > > Reconnect due to socket error: *
> > > *java.nio.channels.ClosedByInterruptException*
> > > * at
> > >
> > >
> >
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)*
> > > * at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:386)*
> > > * at
> > >
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:220)*
> > > * at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)*
> > > at
> > >
> >
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> > > at kafka.utils.Utils$.read(Utils.scala:394)
> > > at
> > >
> > >
> >
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67)
> > > at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
> > > at
> > >
> > >
> >
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> > > at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
> > > at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:73)
> > > at
> > >
> > >
> >
> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71)
> > > at
> > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:110)
> > > at
> > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> > > at
> > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > at
> > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:109)
> > > at
> > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> > > at
> > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:108)
> > > at
> > >
> > >
> >
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
> > > at
> > >
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
> > > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > >
> > >
> > > currently we are using 3 node kafka cluster(0.8.0_beta) with 1 topic of
> > 100
> > > partition. According to kakfa documentation, GC maybe the reason of too
> > may
> > > rebalances so i monitored my app with jstat, couldn't any issue over
> > there.
> > >
> > > [coms@coms04 coms-timemachine]$ jstat -gcutil 7419 1000 100
> > >   S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT
> > >  35.09   8.88 100.00  66.89  90.73   1119  820.300   102 1093.934
> > 1914.234
> > >   0.00  59.06  21.02  70.93  90.89   1119  821.362   102 1093.934
> > 1915.296
> > >   0.00  59.06  84.61  70.93  91.85   1119  821.362   102 1093.934
> > 1915.296
> > >  35.08  59.06 100.00  79.29  92.00   1120  821.362   102 1093.934
> > 1915.296
> > >  53.67   0.00  43.22  80.63  92.55   1120  822.511   102 1093.934
> > 1916.445
> > >  53.67   0.00  89.50  80.63  93.55   1120  822.511   102 1093.934
> > 1916.445
> > >  53.67  31.42 100.00  86.79  93.97   1121  822.511   102 1093.934
> > 1916.445
> > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> > 1917.514
> > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> > 1917.514
> > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> > 1917.514
> > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> > 1917.514
> > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> > 1917.514
> > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> > 1917.514
> > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> > 1917.514
> > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> > 1917.514
> > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> > 1917.514
> > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> > 1917.514
> > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> > 1917.514
> > >   0.00   0.00  65.02  41.18  90.02   1121  823.580   103 1104.702
> > 1928.282
> > >  36.43   0.00   4.34  41.18  90.58   1122  823.979   103 1104.702
> > 1928.681
> > >  36.43   0.00  69.49  41.18  91.02   1122  823.979   103 1104.702
> > 1928.681
> > >  36.43  26.60 100.00  42.34  91.20   1123  823.979   103 1104.702
> > 1928.681
> > >   0.00  35.07  30.98  46.02  91.41   1123  824.749   103 1104.702
> > 1929.451
> > >   0.00  35.07  86.93  46.02  91.60   1123  824.749   103 1104.702
> > 1929.451
> > >   1.60   0.00  14.58  51.85  91.74   1124  825.198   103 1104.702
> > 1929.900
> > >   1.60   0.00  62.30  51.85  91.98   1124  825.198   103 1104.702
> > 1929.900
> > >   0.00   9.65   6.49  52.08  92.19   1125  825.360   103 1104.702
> > 1930.063
> > >   0.00   9.65  58.28  52.08  92.47   1125  825.360   103 1104.702
> > 1930.063
> > >  19.19   9.65 100.00  53.38  92.63   1126  825.360   103 1104.702
> > 1930.063
> > >  25.31   0.00  51.89  53.48  92.87   1126  825.784   103 1104.702
> > 1930.487
> > >  25.31  11.21 100.00  55.24  93.04   1127  825.784   103 1104.702
> > 1930.487
> > >   0.00  99.97  39.63  57.69  93.22   1127  826.513   103 1104.702
> > 1931.216
> > >   0.00  99.97  86.22  57.69  93.40   1127  826.513   103 1104.702
> > 1931.216
> > >  72.36  99.97 100.00  57.69  93.51   1128  826.513   103 1104.702
> > 1931.216
> > >  85.99   0.00  44.82  57.69  93.76   1128  827.339   103 1104.702
> > 1932.041
> > >  85.99   0.83 100.00  66.34  94.09   1129  827.339   103 1104.702
> > 1932.041
> > >   0.00  36.82  12.21  73.88  94.17   1129  828.588   103 1104.702
> > 1933.290
> > >   0.00  36.82 100.00  73.88  94.94   1130  828.588   103 1104.702
> > 1933.290
> > >  47.42   0.00   5.23  79.99  95.02   1130  829.497   103 1104.702
> > 1934.199
> > >  47.42   0.00  69.50  79.99  96.14   1130  829.497   103 1104.702
> > 1934.199
> > >  47.42  19.36 100.00  80.78  97.13   1131  829.497   103 1104.702
> > 1934.199
> > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > 1935.156
> > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > 1935.156
> > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > 1935.156
> > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > 1935.156
> > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > 1935.156
> > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > 1935.156
> > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > 1935.156
> > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > 1935.156
> > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > 1935.156
> > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > 1935.156
> > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > 1935.156
> > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > 1935.156
> > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > 1935.156
> > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > 1935.156
> > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > 1935.156
> > >   0.00   0.00  33.49  59.22  89.53   1131  830.454   104 1119.378
> > 1949.831
> > >   0.00   0.00  87.06  59.22  90.67   1131  830.454   104 1119.378
> > 1949.831
> > >  24.49   0.00  19.01  59.22  91.22   1132  830.770   104 1119.378
> > 1950.147
> > >  24.49   0.00  72.05  59.22  91.46   1132  830.770   104 1119.378
> > 1950.147
> > >  24.49  39.18 100.00  59.22  91.64   1133  830.770   104 1119.378
> > 1950.147
> > >   0.00  99.99  44.95  59.22  91.82   1133  831.338   104 1119.378
> > 1950.715
> > >   0.00  99.99  94.30  59.22  92.17   1133  831.338   104 1119.378
> > 1950.715
> > >  58.40   0.00  11.63  59.22  92.22   1134  831.993   104 1119.378
> > 1951.370
> > >  58.40   0.00  58.77  59.22  92.46   1134  831.993   104 1119.378
> > 1951.370
> > >  58.40  27.41 100.00  59.22  92.55   1135  831.993   104 1119.378
> > 1951.370
> > >   0.00  63.03  30.83  59.22  92.71   1135  832.675   104 1119.378
> > 1952.053
> > >   0.00  63.03  81.31  59.22  92.94   1135  832.675   104 1119.378
> > 1952.053
> > >  55.01  63.03 100.00  59.22  92.98   1136  832.675   104 1119.378
> > 1952.053
> > >  79.90   0.00  37.84  59.22  93.14   1136  833.582   104 1119.378
> > 1952.960
> > >  79.90   0.00  92.75  59.22  93.24   1136  833.582   104 1119.378
> > 1952.960
> > >  79.90  69.59 100.00  59.22  93.30   1137  833.582   104 1119.378
> > 1952.960
> > >   0.00 100.00  54.86  59.77  93.65   1137  834.765   104 1119.378
> > 1954.143
> > >  31.63 100.00  98.64  60.31  94.04   1138  834.765   104 1119.378
> > 1954.143
> > > 100.00 100.00  98.64  63.88  94.04   1138  834.765   104 1119.378
> > 1954.143
> > > 100.00   0.00  50.43  65.69  94.50   1138  836.225   104 1119.378
> > 1955.603
> > > 100.00  14.88 100.00  66.31  95.25   1139  836.225   104 1119.378
> > 1955.603
> > > 100.00 100.00 100.00  67.98  95.25   1139  836.225   104 1119.378
> > 1955.603
> > >   0.00 100.00  43.37  72.29  95.82   1139  837.636   104 1119.378
> > 1957.014
> > >  24.55 100.00 100.00  76.41  96.40   1140  837.636   104 1119.378
> > 1957.014
> > >  93.50 100.00 100.00  80.50  96.40   1140  837.636   104 1119.378
> > 1957.014
> > > 100.00   0.00  62.48  82.20  96.79   1140  839.089   104 1119.378
> > 1958.467
> > > 100.00  22.78 100.00  88.93  97.05   1141  839.089   104 1119.378
> > 1958.467
> > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> > 1959.958
> > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> > 1959.958
> > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> > 1959.958
> > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> > 1959.958
> > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> > 1959.958
> > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> > 1959.958
> > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> > 1959.958
> > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> > 1959.958
> > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> > 1959.958
> > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> > 1959.958
> > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> > 1959.958
> > >   0.00   0.00  53.78  46.66  89.50   1141  840.580   105 1129.520
> > 1970.101
> > >   0.27   0.00 100.00  46.66  89.75   1142  840.580   105 1129.520
> > 1970.101
> > >  26.45   0.00  33.61  46.66  89.77   1142  840.924   105 1129.520
> > 1970.444
> > >  26.45   0.00  84.87  46.66  89.81   1142  840.924   105 1129.520
> > 1970.444
> > >   0.00   2.68  16.33  50.87  89.83   1143  841.276   105 1129.520
> > 1970.796
> > >   0.00   2.68  63.77  50.87  89.87   1143  841.276   105 1129.520
> > 1970.796
> > >
> > >
> > > Not sure what is happening over here. any leads would be helpful..
> > >
> > > Regards,
> > > Ankit TYagi
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message