kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ankit tyagi <ankittyagi.mn...@gmail.com>
Subject Re: too many rebalances in my consumer log and fetcher threads are getting stopped
Date Sat, 19 Apr 2014 15:41:57 GMT
Hi Jun,

i grabbed for 'begin rebalancing consumer' in my log and got around
*5000*times in today's logs.

Can there be any other reason instead full GC to initiate re-balances.  Are
there any broker properties which can be tune to reduce rebalances?


On Sat, Apr 19, 2014 at 9:31 AM, Jun Rao <junrao@gmail.com> wrote:

> Do you see any rebalances? The fetcher was stopped because it was shut
> down, which typically happens during rebalances.
>
> Thanks,
>
> Jun
>
>
> On Fri, Apr 18, 2014 at 11:08 AM, ankit tyagi <ankittyagi.mnnit@gmail.com
> >wrote:
>
> > I have checked that. There was no full gc at that time. I have attached
> > jstat ouput too in my mail.
> >
> > I have concerned over why consumer fetcher thread are getting stopped.
> > On 18 Apr 2014 22:51, "Jun Rao" <junrao@gmail.com> wrote:
> >
> > > Have you looked at
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog
> > > ?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Fri, Apr 18, 2014 at 3:58 AM, ankit tyagi <
> ankittyagi.mnnit@gmail.com
> > > >wrote:
> > >
> > > > Hi,
> > > >
> > > > I am seeing consumer re-balances very frequently and getting socket
> > > > reconnect exception. log is given below for more insights
> > > >
> > > >
> > > > [2014-04-18
> > > >
> > > >
> > >
> >
> 16:02:52.061][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][SyncProducer:67]
> > > > Disconnecting from kafka_leader_coms07.snapdeal.com:9092
> > > > [2014-04-18
> > > >
> > > >
> > >
> >
> 16:02:52.103][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager:67]
> > > > [ConsumerFetcherManager-1397812122507] *Stopping leader finder
> thread*
> > > > [2014-04-18
> > > >
> > > >
> > >
> >
> 16:02:52.104][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
> > > >
> > > >
> > >
> >
> [kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
> > > > Shutting down
> > > > [2014-04-18
> > > >
> > > >
> > >
> >
> 16:02:52.105][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
> > > >
> > > >
> > >
> >
> [kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
> > > > Shutdown completed
> > > > [2014-04-18
> > > >
> > > >
> > >
> >
> 16:02:52.106][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherManager:67]
> > > > [ConsumerFetcherManager-1397812122507]* Stopping all fetchers*
> > > > [2014-04-18
> > > >
> > > >
> > >
> >
> 16:02:52.106][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
> > > >
> > > >
> > >
> >
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
> > > > Shutting down
> > > > [2014-04-18
> > > >
> > > >
> > >
> >
> 16:02:52.107][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-*timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0][INFO][SimpleConsumer:75]
> > > > Reconnect due to socket error: *
> > > > *java.nio.channels.ClosedByInterruptException*
> > > > * at
> > > >
> > > >
> > >
> >
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)*
> > > > * at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:386)*
> > > > * at
> > > >
> > sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:220)*
> > > > * at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)*
> > > > at
> > > >
> > >
> >
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> > > > at kafka.utils.Utils$.read(Utils.scala:394)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67)
> > > > at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> > > > at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
> > > > at
> kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:73)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:110)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:109)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:108)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
> > > > at
> > > >
> > kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
> > > > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > > > [2014-04-18
> > > >
> > > >
> > >
> >
> 16:02:52.108][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0][INFO][ConsumerFetcherThread:67]
> > > >
> > > >
> > >
> >
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
> > > > Stopped
> > > > [2014-04-18
> > > >
> > > >
> > >
> >
> 16:02:52.109][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-*1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
> > > >
> > > >
> > >
> >
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-0],
> > > > Shutdown completed*
> > > > [2014-04-18
> > > >
> > > >
> > >
> >
> 16:02:52.171][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663_watcher_executor][INFO][ConsumerFetcherThread:67]
> > > >
> > > >
> > >
> >
> [ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-2],
> > > > Shutting down
> > > > [2014-04-18
> > > >
> > > >
> > >
> >
> 16:02:52.109][kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread][INFO][ConsumerFetcherManager$LeaderFinderThread:67]
> > > >
> > > >
> > >
> >
> [kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-leader-finder-thread],
> > > > Stopped
> > > > [*2014-04-18
> > > >
> > > >
> > >
> >
> 16:02:52.171][ConsumerFetcherThread-kafka.coms.consumer.kafka_topic_coms_esb_prod_coms.coms-timemachine.coms.coms04.snapdeal.com_coms04.snapdeal.com-1397812122323-509d9663-0-2][INFO][SimpleConsumer:75]
> > > > Reconnect due to socket error: *
> > > > *java.nio.channels.ClosedByInterruptException*
> > > > * at
> > > >
> > > >
> > >
> >
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)*
> > > > * at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:386)*
> > > > * at
> > > >
> > sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:220)*
> > > > * at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)*
> > > > at
> > > >
> > >
> >
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> > > > at kafka.utils.Utils$.read(Utils.scala:394)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67)
> > > > at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> > > > at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
> > > > at
> kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:73)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:110)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:110)
> > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:109)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:109)
> > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:108)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
> > > > at
> > > >
> > kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
> > > > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> > > >
> > > >
> > > > currently we are using 3 node kafka cluster(0.8.0_beta) with 1 topic
> of
> > > 100
> > > > partition. According to kakfa documentation, GC maybe the reason of
> too
> > > may
> > > > rebalances so i monitored my app with jstat, couldn't any issue over
> > > there.
> > > >
> > > > [coms@coms04 coms-timemachine]$ jstat -gcutil 7419 1000 100
> > > >   S0     S1     E      O      P     YGC     YGCT    FGC    FGCT
> GCT
> > > >  35.09   8.88 100.00  66.89  90.73   1119  820.300   102 1093.934
> > > 1914.234
> > > >   0.00  59.06  21.02  70.93  90.89   1119  821.362   102 1093.934
> > > 1915.296
> > > >   0.00  59.06  84.61  70.93  91.85   1119  821.362   102 1093.934
> > > 1915.296
> > > >  35.08  59.06 100.00  79.29  92.00   1120  821.362   102 1093.934
> > > 1915.296
> > > >  53.67   0.00  43.22  80.63  92.55   1120  822.511   102 1093.934
> > > 1916.445
> > > >  53.67   0.00  89.50  80.63  93.55   1120  822.511   102 1093.934
> > > 1916.445
> > > >  53.67  31.42 100.00  86.79  93.97   1121  822.511   102 1093.934
> > > 1916.445
> > > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> > > 1917.514
> > > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> > > 1917.514
> > > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> > > 1917.514
> > > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> > > 1917.514
> > > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> > > 1917.514
> > > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> > > 1917.514
> > > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> > > 1917.514
> > > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> > > 1917.514
> > > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> > > 1917.514
> > > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> > > 1917.514
> > > >   0.00  40.26   0.00  89.55  93.97   1121  823.580   103 1093.934
> > > 1917.514
> > > >   0.00   0.00  65.02  41.18  90.02   1121  823.580   103 1104.702
> > > 1928.282
> > > >  36.43   0.00   4.34  41.18  90.58   1122  823.979   103 1104.702
> > > 1928.681
> > > >  36.43   0.00  69.49  41.18  91.02   1122  823.979   103 1104.702
> > > 1928.681
> > > >  36.43  26.60 100.00  42.34  91.20   1123  823.979   103 1104.702
> > > 1928.681
> > > >   0.00  35.07  30.98  46.02  91.41   1123  824.749   103 1104.702
> > > 1929.451
> > > >   0.00  35.07  86.93  46.02  91.60   1123  824.749   103 1104.702
> > > 1929.451
> > > >   1.60   0.00  14.58  51.85  91.74   1124  825.198   103 1104.702
> > > 1929.900
> > > >   1.60   0.00  62.30  51.85  91.98   1124  825.198   103 1104.702
> > > 1929.900
> > > >   0.00   9.65   6.49  52.08  92.19   1125  825.360   103 1104.702
> > > 1930.063
> > > >   0.00   9.65  58.28  52.08  92.47   1125  825.360   103 1104.702
> > > 1930.063
> > > >  19.19   9.65 100.00  53.38  92.63   1126  825.360   103 1104.702
> > > 1930.063
> > > >  25.31   0.00  51.89  53.48  92.87   1126  825.784   103 1104.702
> > > 1930.487
> > > >  25.31  11.21 100.00  55.24  93.04   1127  825.784   103 1104.702
> > > 1930.487
> > > >   0.00  99.97  39.63  57.69  93.22   1127  826.513   103 1104.702
> > > 1931.216
> > > >   0.00  99.97  86.22  57.69  93.40   1127  826.513   103 1104.702
> > > 1931.216
> > > >  72.36  99.97 100.00  57.69  93.51   1128  826.513   103 1104.702
> > > 1931.216
> > > >  85.99   0.00  44.82  57.69  93.76   1128  827.339   103 1104.702
> > > 1932.041
> > > >  85.99   0.83 100.00  66.34  94.09   1129  827.339   103 1104.702
> > > 1932.041
> > > >   0.00  36.82  12.21  73.88  94.17   1129  828.588   103 1104.702
> > > 1933.290
> > > >   0.00  36.82 100.00  73.88  94.94   1130  828.588   103 1104.702
> > > 1933.290
> > > >  47.42   0.00   5.23  79.99  95.02   1130  829.497   103 1104.702
> > > 1934.199
> > > >  47.42   0.00  69.50  79.99  96.14   1130  829.497   103 1104.702
> > > 1934.199
> > > >  47.42  19.36 100.00  80.78  97.13   1131  829.497   103 1104.702
> > > 1934.199
> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > > 1935.156
> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > > 1935.156
> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > > 1935.156
> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > > 1935.156
> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > > 1935.156
> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > > 1935.156
> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > > 1935.156
> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > > 1935.156
> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > > 1935.156
> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > > 1935.156
> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > > 1935.156
> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > > 1935.156
> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > > 1935.156
> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > > 1935.156
> > > >   0.00  99.99   0.00  87.63  97.13   1131  830.454   104 1104.702
> > > 1935.156
> > > >   0.00   0.00  33.49  59.22  89.53   1131  830.454   104 1119.378
> > > 1949.831
> > > >   0.00   0.00  87.06  59.22  90.67   1131  830.454   104 1119.378
> > > 1949.831
> > > >  24.49   0.00  19.01  59.22  91.22   1132  830.770   104 1119.378
> > > 1950.147
> > > >  24.49   0.00  72.05  59.22  91.46   1132  830.770   104 1119.378
> > > 1950.147
> > > >  24.49  39.18 100.00  59.22  91.64   1133  830.770   104 1119.378
> > > 1950.147
> > > >   0.00  99.99  44.95  59.22  91.82   1133  831.338   104 1119.378
> > > 1950.715
> > > >   0.00  99.99  94.30  59.22  92.17   1133  831.338   104 1119.378
> > > 1950.715
> > > >  58.40   0.00  11.63  59.22  92.22   1134  831.993   104 1119.378
> > > 1951.370
> > > >  58.40   0.00  58.77  59.22  92.46   1134  831.993   104 1119.378
> > > 1951.370
> > > >  58.40  27.41 100.00  59.22  92.55   1135  831.993   104 1119.378
> > > 1951.370
> > > >   0.00  63.03  30.83  59.22  92.71   1135  832.675   104 1119.378
> > > 1952.053
> > > >   0.00  63.03  81.31  59.22  92.94   1135  832.675   104 1119.378
> > > 1952.053
> > > >  55.01  63.03 100.00  59.22  92.98   1136  832.675   104 1119.378
> > > 1952.053
> > > >  79.90   0.00  37.84  59.22  93.14   1136  833.582   104 1119.378
> > > 1952.960
> > > >  79.90   0.00  92.75  59.22  93.24   1136  833.582   104 1119.378
> > > 1952.960
> > > >  79.90  69.59 100.00  59.22  93.30   1137  833.582   104 1119.378
> > > 1952.960
> > > >   0.00 100.00  54.86  59.77  93.65   1137  834.765   104 1119.378
> > > 1954.143
> > > >  31.63 100.00  98.64  60.31  94.04   1138  834.765   104 1119.378
> > > 1954.143
> > > > 100.00 100.00  98.64  63.88  94.04   1138  834.765   104 1119.378
> > > 1954.143
> > > > 100.00   0.00  50.43  65.69  94.50   1138  836.225   104 1119.378
> > > 1955.603
> > > > 100.00  14.88 100.00  66.31  95.25   1139  836.225   104 1119.378
> > > 1955.603
> > > > 100.00 100.00 100.00  67.98  95.25   1139  836.225   104 1119.378
> > > 1955.603
> > > >   0.00 100.00  43.37  72.29  95.82   1139  837.636   104 1119.378
> > > 1957.014
> > > >  24.55 100.00 100.00  76.41  96.40   1140  837.636   104 1119.378
> > > 1957.014
> > > >  93.50 100.00 100.00  80.50  96.40   1140  837.636   104 1119.378
> > > 1957.014
> > > > 100.00   0.00  62.48  82.20  96.79   1140  839.089   104 1119.378
> > > 1958.467
> > > > 100.00  22.78 100.00  88.93  97.05   1141  839.089   104 1119.378
> > > 1958.467
> > > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> > > 1959.958
> > > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> > > 1959.958
> > > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> > > 1959.958
> > > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> > > 1959.958
> > > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> > > 1959.958
> > > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> > > 1959.958
> > > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> > > 1959.958
> > > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> > > 1959.958
> > > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> > > 1959.958
> > > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> > > 1959.958
> > > >   0.00  96.13   0.00  90.89  97.05   1141  840.580   105 1119.378
> > > 1959.958
> > > >   0.00   0.00  53.78  46.66  89.50   1141  840.580   105 1129.520
> > > 1970.101
> > > >   0.27   0.00 100.00  46.66  89.75   1142  840.580   105 1129.520
> > > 1970.101
> > > >  26.45   0.00  33.61  46.66  89.77   1142  840.924   105 1129.520
> > > 1970.444
> > > >  26.45   0.00  84.87  46.66  89.81   1142  840.924   105 1129.520
> > > 1970.444
> > > >   0.00   2.68  16.33  50.87  89.83   1143  841.276   105 1129.520
> > > 1970.796
> > > >   0.00   2.68  63.77  50.87  89.87   1143  841.276   105 1129.520
> > > 1970.796
> > > >
> > > >
> > > > Not sure what is happening over here. any leads would be helpful..
> > > >
> > > > Regards,
> > > > Ankit TYagi
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message