kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael J. Kitchin" <mcoyote...@gmail.com>
Subject Questions re: auto-commit and camel-kafka
Date Fri, 03 Jul 2015 04:47:53 GMT
Hi there,

These are questions re: the official camel-kafka integration. Since the
issues touch on both camel and kafka I'm sending to both users lists in
search of answers.

- - - -

I have a simple, inonly, point-to-point, synchronous camel route (a)
consuming from kafka topic (consumer), (b) running the resulting exchanges
(messages) through a processor chain and (c) forwarding the outcome on to
somewhere else.

If the runtime dies while exchanges are in this pipeline, I want things to
pick up where they left off when restarted. I'm not picky about seeing the
same data more than once (as long as it's recent), I just can't lose

In brief, everything's working great except this failure/recovery part --
in-flight exchanges are getting lost and there is no, apparent re-delivery
on restart. My reading of the JMX data suggests the kafka logs are intact.

I think this has to do with consumer auto-commit, which is the default
behavior. My reading of the kafka and camel-kafka docs suggests disabling
auto-commit will give me what I want, but when I try it I'm not seeing
re-delivery kick off when I restart.

So, first question:
(1) Is auto-commit off the key to getting what I want and/or what else
might I need to do?

- - - - -

Meanwhile, on the producer side I'm seeing the first (and only the first)
message apparently get eaten. It's possible it's being buffered, but it
never seems to timeout. There are no error messages on startup and the
camel context, routes, etc. appear to have started successfully. The second
message and everything that follows is golden.

The payloads are ~70-character byte arrays, if it makes a difference.

Second question, then:
(2) Is there a batching setting or something else I might be overlooking
behind this behavior?

- - - - -

Thanks, in advance for your time and consideration. We've been impressed
with kafka so far and are looking forward to employing it in production.

Please let me know if I may provide any additional information. Thanks.



- - - - -

*Michael J. Kitchin*

Senior Software Engineer

Operational Systems, Inc.

4450 Arapahoe Avenue, Suite 100

Boulder, CO 80303

Phone: 719-271-6476

Email: michael.kitchin@opsysinc.com

Web: www.opsysinc.com

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message