kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexey Romanchuk <alexey.romanc...@gmail.com>
Subject Re: New client commitAsync problem
Date Sat, 06 Feb 2016 15:34:46 GMT
Hi Jason!

Thanks for reply. I try to implement it and looks like it works. Few
moments about pause/resume:
- is it cost something? is it any network activity to pause/resume?
- there is no API to get information of current status of resume/pause for
client. Am I wrong? What about having such API?

Also, you told about "one option". Is it another?

Thanks!

On Sat, Feb 6, 2016 at 6:13 AM, Jason Gustafson <jason@confluent.io> wrote:

> Hey Alexey,
>
> The API of the new consumer is designed around an event loop in which all
> IO is driven by the poll() API. To make this work, you need to call poll()
> in a loop (see the javadocs for examples). So in this example, when you
> call commitAsync(), the request is basically just queued up to be sent. It
> won't actually be transmitted, however, until the next poll() is called.
> And it may not return until an even later call.
>
> If I understand correctly, the basic problem you're trying to figure out is
> how you can do an asynchronous commit without continuing to receive records
> for the associated partitions. One option would be to use the pause() API
> to suspend fetching from those partitions. So you could call commitAsync()
> with the commits you need to send, then pause the respective partitions.
> After the commit returns, you could resume() the partitions in the commit
> callback, which will reenable fetching.
>
> Hope that helps!
>
> -Jason
>
>
>
> On Fri, Feb 5, 2016 at 6:35 AM, Alexey Romanchuk <
> alexey.romanchuk@gmail.com
> > wrote:
>
> > Hi all!
> >
> > Right now I am working on reactive streams connector to kafka. I am using
> > new client and found strange behavior of commitAsync method which not
> > calling callbacks at all at some cases.
> >
> > I found, that callback calling is a part of handling of incoming
> messages.
> > These messages are not fetching in background, but fetching during other
> > activity (like fetching from topic). In the other hand there is no way to
> > perform "blank" activity to fetch commit confirmation from Consumer.
> >
> > Right now if I message processing is depended on commit confirmation it
> is
> > impossible to work in reactive way.
> >
> > Here it is very small example of problem -
> > https://gist.github.com/13h3r/496e802afe65233b184a
> >
> > My questions are:
> > - is it bug or design decision?
> > - if it is not bug how I can write reactive consumer?
> >
> > Thanks!
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message