kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: too many rebalances in my consumer log and fetcher threads are getting stopped
Date Mon, 21 Apr 2014 05:13:32 GMT
Yes, you could increase zk session timeout too. The impact is that a hard
kill of the consumer could take up to that timeout to be detected.

Thanks,

Jun


On Sun, Apr 20, 2014 at 9:50 AM, ankit tyagi <ankittyagi.mnnit@gmail.com>wrote:

> Can i reduce the consumer balances if i increase the zookeeper session
> timeout?
>
>
>
>
> On Sat, Apr 19, 2014 at 9:11 PM, ankit tyagi <ankittyagi.mnnit@gmail.com
> >wrote:
>
> > Hi Jun,
> >
> > i grabbed for 'begin rebalancing consumer' in my log and got around
> *5000*times in today's logs.
> >
> > Can there be any other reason instead full GC to initiate re-balances.
> >  Are there any broker properties which can be tune to reduce rebalances?
> >
> >
> > On Sat, Apr 19, 2014 at 9:31 AM, Jun Rao <junrao@gmail.com> wrote:
> >
> >> Do you see any rebalances? The fetcher was stopped because it was shut
> >> down, which typically happens during rebalances.
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >>
> >> On Fri, Apr 18, 2014 at 11:08 AM, ankit tyagi <
> ankittyagi.mnnit@gmail.com
> >> >wrote:
> >>
> >> > I have checked that. There was no full gc at that time. I have
> attached
> >> > jstat ouput too in my mail.
> >> >
> >> > I have concerned over why consumer fetcher thread are getting stopped.
> >> > On 18 Apr 2014 22:51, "Jun Rao" <junrao@gmail.com> wrote:
> >> >
> >> > > Have you looked at
> >> > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog
> >> > > ?
> >> > >
> >> > > Thanks,
> >> > >
> >> > > Jun
> >> > >
> >> > >
> >> > > On Fri, Apr 18, 2014 at 3:58 AM, ankit tyagi <
> >> ankittyagi.mnnit@gmail.com
> >> > > >wrote:
> >> > >
> >> > > > Hi,
> >> > > >
> >> > > > I am seeing consumer re-balances very frequently and getting
> socket
> >> > > > reconnect exception. log is given below for more insights
> >> > > >
> >> > > >
> >> > > > [2014-04-18
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> 16:02:52.061][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][SyncProducer:67]
> >> > > > Disconnecting from kafka_leader_coms07.snapdeal.com:9092
> >> > > > [2014-04-18
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> 16:02:52.103][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager:67]
> >> > > > [ConsumerFetcherManager-1397812122507] *Stopping leader finder
> >> thread*
> >> > > > [2014-04-18
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> 16:02:52.104][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> [kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
> >> > > > Shutting down
> >> > > > [2014-04-18
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> 16:02:52.105][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> [kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
> >> > > > Shutdown completed
> >> > > > [2014-04-18
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> 16:02:52.106][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager:67]
> >> > > > [ConsumerFetcherManager-1397812122507]* Stopping all fetchers*
> >> > > > [2014-04-18
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> 16:02:52.106][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
> >> > > > Shutting down
> >> > > > [2014-04-18
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> 16:02:52.107][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-*timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0][INFO][SimpleConsumer:75]
> >> > > > Reconnect due to socket error: *
> >> > > > *java.nio.channels.ClosedByInterruptException*
> >> > > > * at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)*
> >> > > > * at
> sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:386)*
> >> > > > * at
> >> > > >
> >> >
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:220)*
> >> > > > * at
> >> sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)*
> >> > > > at
> >> > > >
> >> > >
> >> >
> >>
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> >> > > > at kafka.utils.Utils$.read(Utils.scala:394)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67)
> >> > > > at
> kafka.network.Receive$class.readCompletely(Transmission.scala:56)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> >> > > > at
> kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
> >> > > > at
> >> kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:73)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:110)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> >> > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:109)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> >> > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> >> > > > at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:108)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
> >> > > > at
> >> > > >
> >> >
> >>
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
> >> > > > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> >> > > > [2014-04-18
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> 16:02:52.108][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0][INFO][ConsumerFetcherThread:67]
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
> >> > > > Stopped
> >> > > > [2014-04-18
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> 16:02:52.109][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-*1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
> >> > > > Shutdown completed*
> >> > > > [2014-04-18
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> 16:02:52.171][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-2],
> >> > > > Shutting down
> >> > > > [2014-04-18
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> 16:02:52.109][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> [kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
> >> > > > Stopped
> >> > > > [*2014-04-18
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> 16:02:52.171][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-2][INFO][SimpleConsumer:75]
> >> > > > Reconnect due to socket error: *
> >> > > > *java.nio.channels.ClosedByInterruptException*
> >> > > > * at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)*
> >> > > > * at
> sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:386)*
> >> > > > * at
> >> > > >
> >> >
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:220)*
> >> > > > * at
> >> sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)*
> >> > > > at
> >> > > >
> >> > >
> >> >
> >>
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> >> > > > at kafka.utils.Utils$.read(Utils.scala:394)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67)
> >> > > > at
> kafka.network.Receive$class.readCompletely(Transmission.scala:56)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> >> > > > at
> kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
> >> > > > at
> >> kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:73)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:110)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> >> > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:109)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> >> > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> >> > > > at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:108)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
> >> > > > at
> >> > > >
> >> >
> >>
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
> >> > > > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> >> > > >
> >> > > >
> >> > > > currently we are using 3 node kafka cluster(0.8.0_beta) with
1
> >> topic of
> >> > > 100
> >> > > > partition. According to kakfa documentation, GC maybe the reason
> of
> >> too
> >> > > may
> >> > > > rebalances so i monitored my app with jstat, couldn't any issue
> over
> >> > > there.
> >> > > >
> >> > > > [coms@coms04 coms-timemachine]$ jstat -gcutil 7419 1000 100
> >> > > >   S0     S1     E      O      P     YGC     YGCT    FGC    FGCT
> >> GCT
> >> > > >  35.09   8.88 100.00  66.89  90.73   1119  820.300   102 1093.934
> >> > > 1914.234
> >> > > >   0.00  59.06  21.02  70.93  90.89   1119  821.362   102 1093.934
> >> > > 1915.296
> >> > > >   0.00  59.06  84.61  70.93  91.85   1119  821.362   102 1093.934
> >> > > 1915.296
> >> > > >  35.08  59.06 100.00  79.29  92.00   1120  821.362   102 1093.934
> >> > > 1915.296
> >> > > >  53.67   0.00  43.22  80.63  92.55   1120  822.511   102 1093.934
> >> > > 1916.445
> >> > > >  53.67   0.00  89.50  80.63  93.55   1120  822.511   102 1093.934
> >> > > 1916.445
> >> > > >  53.67  31.42 100.00  86.79  93.97   1121  822.511   102 1093.934
> >> > > 1916.445
> >> > > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> >> > > 1917.514
> >> > > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> >> > > 1917.514
> >> > > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> >> > > 1917.514
> >> > > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> >> > > 1917.514
> >> > > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> >> > > 1917.514
> >> > > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> >> > > 1917.514
> >> > > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> >> > > 1917.514
> >> > > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> >> > > 1917.514
> >> > > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> >> > > 1917.514
> >> > > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> >> > > 1917.514
> >> > > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> >> > > 1917.514
> >> > > >   0.00   0.00  65.02  41.18  90.02   1121  823.580   103 1104.702
> >> > > 1928.282
> >> > > >  36.43   0.00   4.34  41.18  90.58   1122  823.979   103 1104.702
> >> > > 1928.681
> >> > > >  36.43   0.00  69.49  41.18  91.02   1122  823.979   103 1104.702
> >> > > 1928.681
> >> > > >  36.43  26.60 100.00  42.34  91.20   1123  823.979   103 1104.702
> >> > > 1928.681
> >> > > >   0.00  35.07  30.98  46.02  91.41   1123  824.749   103 1104.702
> >> > > 1929.451
> >> > > >   0.00  35.07  86.93  46.02  91.60   1123  824.749   103 1104.702
> >> > > 1929.451
> >> > > >   1.60   0.00  14.58  51.85  91.74   1124  825.198   103 1104.702
> >> > > 1929.900
> >> > > >   1.60   0.00  62.30  51.85  91.98   1124  825.198   103 1104.702
> >> > > 1929.900
> >> > > >   0.00   9.65   6.49  52.08  92.19   1125  825.360   103 1104.702
> >> > > 1930.063
> >> > > >   0.00   9.65  58.28  52.08  92.47   1125  825.360   103 1104.702
> >> > > 1930.063
> >> > > >  19.19   9.65 100.00  53.38  92.63   1126  825.360   103 1104.702
> >> > > 1930.063
> >> > > >  25.31   0.00  51.89  53.48  92.87   1126  825.784   103 1104.702
> >> > > 1930.487
> >> > > >  25.31  11.21 100.00  55.24  93.04   1127  825.784   103 1104.702
> >> > > 1930.487
> >> > > >   0.00  99.97  39.63  57.69  93.22   1127  826.513   103 1104.702
> >> > > 1931.216
> >> > > >   0.00  99.97  86.22  57.69  93.40   1127  826.513   103 1104.702
> >> > > 1931.216
> >> > > >  72.36  99.97 100.00  57.69  93.51   1128  826.513   103 1104.702
> >> > > 1931.216
> >> > > >  85.99   0.00  44.82  57.69  93.76   1128  827.339   103 1104.702
> >> > > 1932.041
> >> > > >  85.99   0.83 100.00  66.34  94.09   1129  827.339   103 1104.702
> >> > > 1932.041
> >> > > >   0.00  36.82  12.21  73.88  94.17   1129  828.588   103 1104.702
> >> > > 1933.290
> >> > > >   0.00  36.82 100.00  73.88  94.94   1130  828.588   103 1104.702
> >> > > 1933.290
> >> > > >  47.42   0.00   5.23  79.99  95.02   1130  829.497   103 1104.702
> >> > > 1934.199
> >> > > >  47.42   0.00  69.50  79.99  96.14   1130  829.497   103 1104.702
> >> > > 1934.199
> >> > > >  47.42  19.36 100.00  80.78  97.13   1131  829.497   103 1104.702
> >> > > 1934.199
> >> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> >> > > 1935.156
> >> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> >> > > 1935.156
> >> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> >> > > 1935.156
> >> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> >> > > 1935.156
> >> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> >> > > 1935.156
> >> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> >> > > 1935.156
> >> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> >> > > 1935.156
> >> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> >> > > 1935.156
> >> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> >> > > 1935.156
> >> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> >> > > 1935.156
> >> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> >> > > 1935.156
> >> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> >> > > 1935.156
> >> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> >> > > 1935.156
> >> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> >> > > 1935.156
> >> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> >> > > 1935.156
> >> > > >   0.00   0.00  33.49  59.22  89.53   1131  830.454   104 1119.378
> >> > > 1949.831
> >> > > >   0.00   0.00  87.06  59.22  90.67   1131  830.454   104 1119.378
> >> > > 1949.831
> >> > > >  24.49   0.00  19.01  59.22  91.22   1132  830.770   104 1119.378
> >> > > 1950.147
> >> > > >  24.49   0.00  72.05  59.22  91.46   1132  830.770   104 1119.378
> >> > > 1950.147
> >> > > >  24.49  39.18 100.00  59.22  91.64   1133  830.770   104 1119.378
> >> > > 1950.147
> >> > > >   0.00  99.99  44.95  59.22  91.82   1133  831.338   104 1119.378
> >> > > 1950.715
> >> > > >   0.00  99.99  94.30  59.22  92.17   1133  831.338   104 1119.378
> >> > > 1950.715
> >> > > >  58.40   0.00  11.63  59.22  92.22   1134  831.993   104 1119.378
> >> > > 1951.370
> >> > > >  58.40   0.00  58.77  59.22  92.46   1134  831.993   104 1119.378
> >> > > 1951.370
> >> > > >  58.40  27.41 100.00  59.22  92.55   1135  831.993   104 1119.378
> >> > > 1951.370
> >> > > >   0.00  63.03  30.83  59.22  92.71   1135  832.675   104 1119.378
> >> > > 1952.053
> >> > > >   0.00  63.03  81.31  59.22  92.94   1135  832.675   104 1119.378
> >> > > 1952.053
> >> > > >  55.01  63.03 100.00  59.22  92.98   1136  832.675   104 1119.378
> >> > > 1952.053
> >> > > >  79.90   0.00  37.84  59.22  93.14   1136  833.582   104 1119.378
> >> > > 1952.960
> >> > > >  79.90   0.00  92.75  59.22  93.24   1136  833.582   104 1119.378
> >> > > 1952.960
> >> > > >  79.90  69.59 100.00  59.22  93.30   1137  833.582   104 1119.378
> >> > > 1952.960
> >> > > >   0.00 100.00  54.86  59.77  93.65   1137  834.765   104 1119.378
> >> > > 1954.143
> >> > > >  31.63 100.00  98.64  60.31  94.04   1138  834.765   104 1119.378
> >> > > 1954.143
> >> > > > 100.00 100.00  98.64  63.88  94.04   1138  834.765   104 1119.378
> >> > > 1954.143
> >> > > > 100.00   0.00  50.43  65.69  94.50   1138  836.225   104 1119.378
> >> > > 1955.603
> >> > > > 100.00  14.88 100.00  66.31  95.25   1139  836.225   104 1119.378
> >> > > 1955.603
> >> > > > 100.00 100.00 100.00  67.98  95.25   1139  836.225   104 1119.378
> >> > > 1955.603
> >> > > >   0.00 100.00  43.37  72.29  95.82   1139  837.636   104 1119.378
> >> > > 1957.014
> >> > > >  24.55 100.00 100.00  76.41  96.40   1140  837.636   104 1119.378
> >> > > 1957.014
> >> > > >  93.50 100.00 100.00  80.50  96.40   1140  837.636   104 1119.378
> >> > > 1957.014
> >> > > > 100.00   0.00  62.48  82.20  96.79   1140  839.089   104 1119.378
> >> > > 1958.467
> >> > > > 100.00  22.78 100.00  88.93  97.05   1141  839.089   104 1119.378
> >> > > 1958.467
> >> > > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> >> > > 1959.958
> >> > > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> >> > > 1959.958
> >> > > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> >> > > 1959.958
> >> > > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> >> > > 1959.958
> >> > > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> >> > > 1959.958
> >> > > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> >> > > 1959.958
> >> > > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> >> > > 1959.958
> >> > > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> >> > > 1959.958
> >> > > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> >> > > 1959.958
> >> > > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> >> > > 1959.958
> >> > > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> >> > > 1959.958
> >> > > >   0.00   0.00  53.78  46.66  89.50   1141  840.580   105 1129.520
> >> > > 1970.101
> >> > > >   0.27   0.00 100.00  46.66  89.75   1142  840.580   105 1129.520
> >> > > 1970.101
> >> > > >  26.45   0.00  33.61  46.66  89.77   1142  840.924   105 1129.520
> >> > > 1970.444
> >> > > >  26.45   0.00  84.87  46.66  89.81   1142  840.924   105 1129.520
> >> > > 1970.444
> >> > > >   0.00   2.68  16.33  50.87  89.83   1143  841.276   105 1129.520
> >> > > 1970.796
> >> > > >   0.00   2.68  63.77  50.87  89.87   1143  841.276   105 1129.520
> >> > > 1970.796
> >> > > >
> >> > > >
> >> > > > Not sure what is happening over here. any leads would be helpful..
> >> > > >
> >> > > > Regards,
> >> > > > Ankit TYagi
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message