storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Filipa Moura <filipa.mendesmo...@gmail.com>
Subject KafkaConfig: what is the difference between -1 and -2 offset
Date Fri, 05 Dec 2014 01:34:08 GMT
Hi,
I'm trying to get my KafkaSpout to read the latest offset from Kafka.

With my standard configurations, I see startOffsetTime being set to -2 on
the logs:
2014-12-04 23:19:34 s.k.PartitionManager [INFO] Read last commit offset
from zookeeper: 2924325359; old topology_id:
de65d4f8-a8e6-4f72-99bd-2d66e95fd293 - new topology_id:
89cf7268-9db1-423a-978b-3fe214d64e8e
2014-12-04 23:19:34 s.k.PartitionManager [INFO] Read last commit offset
from zookeeper: 3217013339; old topology_id:
de65d4f8-a8e6-4f72-99bd-2d66e95fd293 - new topology_id:
89cf7268-9db1-423a-978b-3fe214d64e8e
2014-12-04 23:19:34 s.k.PartitionManager [INFO] Last commit offset from
zookeeper: 2924325359
2014-12-04 23:19:34 s.k.PartitionManager [INFO] Starting Kafka xx3.com:1
from offset 3217013339
2014-12-04 23:19:34 s.k.PartitionManager [INFO] Commit offset 2924497976 is
more than 100000 behind, resetting to startOffsetTime=-2
2014-12-04 23:19:34 s.k.PartitionManager [INFO] Starting Kafka xxx3.com:0
from offset 2924497976

Adding the following on the code "spoutConfig.startOffsetTime = -1;" :
2014-12-04 23:14:21 s.k.PartitionManager [INFO] Read last commit offset
from zookeeper: 3217004355; old topology_id:
a7010f51-9fea-43e4-ba16-c9ad1f0ec245 - new topology_id:
de65d4f8-a8e6-4f72-99bd-2d66e95fd293
2014-12-04 23:14:21 s.k.PartitionManager [INFO] Read last commit offset
from zookeeper: 2923885086; old topology_id:
a7010f51-9fea-43e4-ba16-c9ad1f0ec245 - new topology_id:
de65d4f8-a8e6-4f72-99bd-2d66e95fd293
2014-12-04 23:14:21 s.k.PartitionManager [INFO] Starting Kafka xxx3.com:1
from offset 3217004355
2014-12-04 23:14:21 s.k.PartitionManager [INFO] Last commit offset from
zookeeper: 2923885086
2014-12-04 23:14:21 s.k.PartitionManager [INFO] Commit offset 2924022915 is
more than 100000 behind, resetting to startOffsetTime=-1
2014-12-04 23:14:21 s.k.PartitionManager [INFO] Starting Kafka xxx3.com:0
from offset 2924022915

What is the difference? And how can I be sure it's using the most recent
offset from Kafka?

Thank you,
Filipa

Mime
View raw message