storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ziemer, Tom" <tom.zie...@wirecard.com>
Subject (storm-kafka-client) KafkaSpout issues
Date Thu, 11 Aug 2016 08:34:23 GMT
Hi everybody,

I just tried to upgrade from the "traditional" KafkaSpout to the new storm-kafka-client version.
Initially everything was working fine, yet I soon discovered that the new Spout stops emitting
tuples after some time.
The happy path (just ack, no errors) seems to work fine, yet when tuples are failed or delayed,
it seems that the (spout-internal) counters are incorrect.

After some debugging sessions, I found the reason here:

private boolean poll() {
  return !waitingToEmit() && numUncommittedOffsets < kafkaSpoutConfig.getMaxUncommittedOffsets();
}

This method always returned false since numUncommittedOffsets was much higher than kafkaSpoutConfig.getMaxUncommittedOffsets().
Apparently I am not the only one facing this problem because similar issues are reported here:
https://github.com/apache/storm/pull/1131#issuecomment-217776015

I could not find any commits related to these issues.

In the latest 1.0.2 release I saw that there are quite a few fixes for the "old" KafkaSpout,
yet almost none for the "new" one. Will both versions be supported in future? Which one is
recommended version to use?

Thanks,
Tom





Mime
View raw message