kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Kreps <jay.kr...@gmail.com>
Subject Re: 0.8 producer -- many questions
Date Thu, 06 Dec 2012 17:20:05 GMT
Hi Ben,

If I understand your question, you are asking if it is possible to decouple
or pipeline the sending of the request and the reading of the response.
E.g. do something like
  send(req1)
  send(req2)
  recv(resp1)
  recv(resp2)
I think your question is, if I do this how do I know what response goes to
what request.

The answer is that the server guarantees in-order processing of requests on
a given connection. So correlating requests to responses can be done by
keeping a queue of as-yet unanswered requests. In 0.8 we also added an
official correlation id (an integer field you can set that the server
passes back unchanged in the response. This is meant to help do tracing
across client and server and may help to debug your implementation. We do
not currently send back the API key since there is only one correct value
that could have--whatever the api of the corresponding request was.

One caveat is that the server doesn't really fully pipeline request
processing at this time so if you do pipeline you may not get quite as much
parallelism as you expect (you can never get full parallelism because of
the ordering guarantee). But in the current code until a response is sent
for req1 the server won't start reading req2. This is not hard to fix and
we will get to it in the next major release I think.

-Jay



On Thu, Dec 6, 2012 at 7:50 AM, ben fleis <ben@monkey.org> wrote:

> I just realized something important.  The BoundedByteBufferSend comment
> above is not in sync with my original question.  I was asking about
> Responses -- meaning, can I multiplex off a single socket, and when reading
> an arbitrary response, switch on request_id to the right handler?
>
> In the blocking console sync client (which is the only trivial way to
> follow the code that I've found -- please enlighten if there's an async
> thing I can easily trace), this is a non issue, since you request, wait,
> parse expected response.
>
> Or do I need instead to just open separate sockets?  (Not a big deal, but
> nice to understand clearly.)
>
> ben
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message