kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Kreps <jay.kr...@gmail.com>
Subject Re: 0.8 producer -- many questions
Date Thu, 06 Dec 2012 17:21:48 GMT
Also we are trying to document all this stuff well with the 0.8 release.
What could I do to improve this document:
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol

Its kind of hard to write good documentation, so any feedback would be
appreciated.

-Jay


On Thu, Dec 6, 2012 at 9:20 AM, Jay Kreps <jay.kreps@gmail.com> wrote:

> Hi Ben,
>
> If I understand your question, you are asking if it is possible to
> decouple or pipeline the sending of the request and the reading of the
> response. E.g. do something like
>   send(req1)
>   send(req2)
>   recv(resp1)
>   recv(resp2)
> I think your question is, if I do this how do I know what response goes to
> what request.
>
> The answer is that the server guarantees in-order processing of requests
> on a given connection. So correlating requests to responses can be done by
> keeping a queue of as-yet unanswered requests. In 0.8 we also added an
> official correlation id (an integer field you can set that the server
> passes back unchanged in the response. This is meant to help do tracing
> across client and server and may help to debug your implementation. We do
> not currently send back the API key since there is only one correct value
> that could have--whatever the api of the corresponding request was.
>
> One caveat is that the server doesn't really fully pipeline request
> processing at this time so if you do pipeline you may not get quite as much
> parallelism as you expect (you can never get full parallelism because of
> the ordering guarantee). But in the current code until a response is sent
> for req1 the server won't start reading req2. This is not hard to fix and
> we will get to it in the next major release I think.
>
> -Jay
>
>
>
> On Thu, Dec 6, 2012 at 7:50 AM, ben fleis <ben@monkey.org> wrote:
>
>> I just realized something important.  The BoundedByteBufferSend comment
>> above is not in sync with my original question.  I was asking about
>> Responses -- meaning, can I multiplex off a single socket, and when
>> reading
>> an arbitrary response, switch on request_id to the right handler?
>>
>> In the blocking console sync client (which is the only trivial way to
>> follow the code that I've found -- please enlighten if there's an async
>> thing I can easily trace), this is a non issue, since you request, wait,
>> parse expected response.
>>
>> Or do I need instead to just open separate sockets?  (Not a big deal, but
>> nice to understand clearly.)
>>
>> ben
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message