kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dana Powers <dana.pow...@gmail.com>
Subject Re: Kafka Protocol API versions
Date Tue, 17 May 2016 13:39:25 GMT
I'd also add that there is some discussion of the mixed-broker issue
on the KIP-35 wiki page as well:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version#KIP-35-Retrievingprotocolversion-Aclientdeveloperwantstoaddsupportforanewfeature

Of course, since 0.9 (and earlier) brokers do not support ApiVersion,
you're still in a bit of a pickle. I think like most shiny new stuff,
we'll probably have to wait and see how different clients approach the
problem and see what works best over time. The 2 non-java clients that
I'm aware of that are attempting to tackle broker
backwards-compatibility are librdkafka (Magnus) and kafka-python
(myself). Asish or others can correct me if I'm wrong here, but I
believe the java client is not attempting to do this yet outside of a
single request flow (SASL authentication).

FWIW, the approach we take in kafka-python is to pick a single
version-state for the cluster and use it for all brokers. At the
present that version-state is chosen during bootstrap by checking the
first contacted broker. That works great for most clusters, where all
brokers are the same version. A more robust approach might be to check
the versions for all brokers in the cluster, not rely on the first
contacted, and pick the most conservative / lowest version-state
found. That's not likely to be perfect, but would support clients that
bootstrap into a mixed cluster. But in any event, providing a manual
override for users to specifically configure the version-state and
skip the dynamic broker version checks is always a good fallback for
managing cluster upgrades.

HTH,

-Dana

On Tue, May 17, 2016 at 6:21 AM, Ismael Juma <ismael@juma.me.uk> wrote:
> Hi Oleksiy,
>
> Ashish has just created a PR to add some information about how to use
> KIP-35:
>
> https://github.com/apache/kafka/pull/1395
>
> Maybe you can have a look and leave a comment with areas that are not
> covered by the first draft.
>
> Ismael
>
> On Tue, May 17, 2016 at 1:34 PM, Oleksiy Krivoshey <oleksiyk@gmail.com>
> wrote:
>
>> Given lots of protocol changes in 0.10 compared to 0.9, with many API
>> methods now having up to 3 possible versions, what is the suggested way for
>> the client library to coordinate with Kafka brokers? I can see there is
>> ApiVersions request which is great, but what happens if Kafka cluster
>> consists of mixed 0.9 and 0.10 brokers (for example during upgrade), should
>> the library keep and refresh the state of supported APIs for each broker
>> the same way it does for topic metadata?
>> Another example is the Metadata request (Api Key 3) which has two versions,
>> how would the client library in the first place know which version of the
>> request to send to each of the initial brokers?
>>
>> I can't see an easy way for the client library to support for example both
>> 0.9 and 0.10 without complicated state switches and error handling. I'm not
>> talking about official Java client, but about other language libraries.
>>
>> What would you suggest? Thanks!
>>

Mime
View raw message