kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Evan Chan ...@ooyala.com>
Subject kafka.common.OffsetOutOfRangeException
Date Tue, 09 Aug 2011 17:37:18 GMT
Hi guys,

I'm following along the Quick Start and ran into an issue when trying to use
a custom Encoder.    I'm basically trying to use a Thrift-encoded message.
 Here is my setup:
- Local ZK, Kafka, consumer and producer all on one machine
- One topic set up with 10 partitions
- Just one consumer which takes all 10 partitions
- Consumer has     props.put("autooffset.reset", "largest")
  in order not to read earlier messages
- Writing producer and consumer in Scala

ZK and Kafka are started.  Then I start the consumer, followed by the
producer.
Consumer shows these errors:

11/08/09 09:45:45 INFO consumer.ZookeeperConsumerConnector: Consumer
player-logs-QoS_eng-dynamic-217.v101.mtv-1312908345470 selected partitions :
player_logs:0-0,player_logs:0-1,player_logs:0-2,player_logs:0-3,player_logs:0-4,player_logs:0-5,player_logs:0-6,player_logs:0-7,player_logs:0-8,player_logs:0-9
11/08/09 09:45:45 INFO consumer.ZookeeperConsumerConnector: end rebalancing
consumer player-logs-QoS_eng-dynamic-217.v101.mtv-1312908345470 try #0
[!] Starting Stream
11/08/09 09:45:45 INFO consumer.FetcherRunnable: FetchRunnable-0 start
fetching topic: player_logs part: 1 offset: 9223372036854775807 from
172.16.100.238:9092
11/08/09 09:45:45 INFO consumer.FetcherRunnable: FetchRunnable-0 start
fetching topic: player_logs part: 7 offset: 9223372036854775807 from
172.16.100.238:9092
11/08/09 09:45:45 INFO consumer.FetcherRunnable: FetchRunnable-0 start
fetching topic: player_logs part: 0 offset: 9223372036854775807 from
172.16.100.238:9092
11/08/09 09:45:45 INFO consumer.FetcherRunnable: FetchRunnable-0 start
fetching topic: player_logs part: 8 offset: 9223372036854775807 from
172.16.100.238:9092
11/08/09 09:45:45 INFO consumer.FetcherRunnable: FetchRunnable-0 start
fetching topic: player_logs part: 5 offset: 9223372036854775807 from
172.16.100.238:9092
11/08/09 09:45:45 INFO consumer.FetcherRunnable: FetchRunnable-0 start
fetching topic: player_logs part: 6 offset: 9223372036854775807 from
172.16.100.238:9092
11/08/09 09:45:45 INFO consumer.FetcherRunnable: FetchRunnable-0 start
fetching topic: player_logs part: 4 offset: 9223372036854775807 from
172.16.100.238:9092
11/08/09 09:45:45 INFO consumer.FetcherRunnable: FetchRunnable-0 start
fetching topic: player_logs part: 3 offset: 9223372036854775807 from
172.16.100.238:9092
11/08/09 09:45:45 INFO consumer.FetcherRunnable: FetchRunnable-0 start
fetching topic: player_logs part: 9 offset: 9223372036854775807 from
172.16.100.238:9092
11/08/09 09:45:45 INFO consumer.FetcherRunnable: FetchRunnable-0 start
fetching topic: player_logs part: 2 offset: 9223372036854775807 from
172.16.100.238:9092
11/08/09 09:45:45 INFO consumer.FetcherRunnable: offset 9223372036854775807
out of range
11/08/09 09:45:45 INFO consumer.FetcherRunnable: updating partition 0-1 with
earliest offset 0
11/08/09 09:45:45 INFO consumer.FetcherRunnable: offset 9223372036854775807
out of range
11/08/09 09:45:45 INFO consumer.FetcherRunnable: updating partition 0-7 with
earliest offset 0
11/08/09 09:45:45 INFO consumer.FetcherRunnable: offset 9223372036854775807
out of range
11/08/09 09:45:45 INFO consumer.FetcherRunnable: updating partition 0-0 with
earliest offset 0
11/08/09 09:45:45 INFO consumer.FetcherRunnable: offset 9223372036854775807
out of range
11/08/09 09:45:45 INFO consumer.FetcherRunnable: updating partition 0-8 with
earliest offset 0
11/08/09 09:45:45 INFO consumer.FetcherRunnable: offset 9223372036854775807
out of range
11/08/09 09:45:45 INFO consumer.FetcherRunnable: updating partition 0-5 with
earliest offset 0
11/08/09 09:45:45 INFO consumer.FetcherRunnable: offset 9223372036854775807
out of range
11/08/09 09:45:45 INFO consumer.FetcherRunnable: updating partition 0-6 with
earliest offset 0
11/08/09 09:45:45 INFO consumer.FetcherRunnable: offset 9223372036854775807
out of range
11/08/09 09:45:45 INFO consumer.FetcherRunnable: updating partition 0-4 with
earliest offset 0
11/08/09 09:45:45 INFO consumer.FetcherRunnable: offset 9223372036854775807
out of range
11/08/09 09:45:45 INFO consumer.FetcherRunnable: updating partition 0-3 with
earliest offset 0
11/08/09 09:45:45 INFO consumer.FetcherRunnable: offset 9223372036854775807
out of range
11/08/09 09:45:45 INFO consumer.FetcherRunnable: updating partition 0-9 with
earliest offset 0
11/08/09 09:45:45 INFO consumer.FetcherRunnable: offset 9223372036854775807
out of range
11/08/09 09:45:45 INFO consumer.FetcherRunnable: updating partition 0-2 with
earliest offset 0
^


Kafka shows these errors:

[2011-08-09 09:45:45,842] ERROR error when processing request
topic:player_logs, part:3 offset:9223372036854775807 maxSize:307200
(kafka.server.KafkaRequestHandlers)
kafka.common.OffsetOutOfRangeException: offset 9223372036854775807 is out of
range
at kafka.log.Log$.findRange(Log.scala:47)
at kafka.log.Log.read(Log.scala:223)
at
kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$readMessageSet(KafkaRequestHandlers.scala:124)
at
kafka.server.KafkaRequestHandlers$$anonfun$3.apply(KafkaRequestHandlers.scala:115)
at
kafka.server.KafkaRequestHandlers$$anonfun$3.apply(KafkaRequestHandlers.scala:114)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:34)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
at scala.collection.mutable.ArrayOps.map(ArrayOps.scala:34)
at
kafka.server.KafkaRequestHandlers.handleMultiFetchRequest(KafkaRequestHandlers.scala:114)
at
kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$3.apply(KafkaRequestHandlers.scala:43)
at
kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$3.apply(KafkaRequestHandlers.scala:43)
at kafka.network.Processor.handle(SocketServer.scala:268)
at kafka.network.Processor.read(SocketServer.scala:291)
at kafka.network.Processor.run(SocketServer.scala:202)
at java.lang.Thread.run(Thread.java:680)



Any ideas?

I've occasionally seen errors like these when testing with the default
StringEncoder as well.

thanks,
Evan

-- 
--
*Evan Chan*
Senior Software Engineer |
ev@ooyala.com | (650) 996-4600
www.ooyala.com | blog <http://www.ooyala.com/blog> |
@ooyala<http://www.twitter.com/ooyala>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message