kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Compton <d...@danielcompton.net>
Subject Re: Programmatic Kafka version detection/extraction?
Date Sat, 15 Nov 2014 08:29:09 GMT
This has been covered in passing in the preceding threads but I'd like to make it explicit,
can we have a command line option/script for getting the Kafka version (probably with Scala
version too)? 

I ran into this recently, where we wanted to verify that the right version of Kafka had been
installed on a number of machines. I ended up getting the version number from the bundled
jars but it wasn't ideal. Being able to run kafka --version would have been great. 

---
Daniel

> On 15/11/2014, at 8:33 am, Magnus Edenhill <magnus@edenhill.se> wrote:
> 
> Hi Gwen,
> 
> the protocol version in an InfoResponse message would be the maximum
> supported mainline protocol version, each change to the
> effective protocol specification bumps the version by one.
> 
> 
> The current per-request-type version ID is unfortunately useless to the
> client, it only allows the broker to act differently
> on different versions of a known request type.
> 
> Since the broker currently does not handle unknown request types gracefully
> - it simply disconnects the client, which from the client's perspective
> could mean anything and nothing - the addition of an InfoRequest/Respones
> would allow the client to know if a desired broker-feature is available or
> not and thus allows the client library to provide the application (and in
> the end the user) with usable error message (compare "Broker disconnected"
> to "offsetCommit not supported by 0.8.0 broker").
> Having the broker return a fabricated response with the header.err field
> set to UNKNOWN_REQUEST_TYPE would be very useful as well of course.
> 
> 
> Regards,
> Magnus
> 
> 
> 
> 2014-11-14 18:17 GMT+01:00 Gwen Shapira <gshapira@cloudera.com>:
> 
>> I'm not sure there's a single "protocol version". Each
>> request/response has its own version ID embedded in the request
>> itself.
>> 
>> Broker version, broker ID and (as Joel suggested) git hash are all
>> reasonable. And +1 for adding this to API to support non-JVM clients.
>> 
>> Gwen
>> 
>> On Fri, Nov 14, 2014 at 8:46 AM, Magnus Edenhill <magnus@edenhill.se>
>> wrote:
>>> Please let's not add dependencies on more third party protocols/software,
>>> the move away from Zookeeper dependence on the clients is very much
>> welcomed
>>> and it'd be a pity to see a new foreign dependency added for such a
>> trivial
>>> thing
>>> as propagating the version of a broker to its client.
>>> 
>>> My suggestion is to add a new protocol request which returns:
>>> - broker version
>>> - protocol version
>>> - broker id
>>> 
>>> A generic future-proof solution would be the use of tags (or named TLVs):
>>> InfoResponse: [InfoTag]
>>> InfoTag:
>>>   intX tag      ( KAFKA_TAG_BROKER_VERSION, KAFKA_TAG_PROTO_VERSION,
>>> KAFKA_TAG_SSL, ... )
>>>   intX type    ( KAFKA_TYPE_STR, KAFKA_TYPE_INT32, KAFKA_TYPE_INT64,
>> ...)
>>>   intX len
>>>   bytes payload
>>> 
>>> Local site/vendor tags could be defined in the configuration file,
>>> 
>>> 
>>> This would also allow clients to enable/disable features based on
>> protocol
>>> version,
>>> e.g., there is no point in trying offsetCommit on a 0.8.0 broker and the
>>> client library
>>> can inform the application about this early, rather than having its
>>> offsetCommit requests
>>> fail by connection teardown (which is not much of an error propagation).
>>> 
>>> 
>>> My two cents,
>>> Magnus
>>> 
>>> 
>>> 2014-11-12 20:11 GMT+01:00 Mark Roberts <wizzat@gmail.com>:
>>> 
>>>> I haven't worked much with JMX before, but some quick googling (10-20
>>>> minutes) is very inconclusive as to how I would go about getting the
>> server
>>>> version I'm connecting to from a Python client.  Can someone please
>>>> reassure me that it's relatively trivial for non Java clients to query
>> JMX
>>>> for the server version of every server in the cluster? Is there a reason
>>>> not to include this in the API itself?
>>>> 
>>>> -Mark
>>>> 
>>>> On Wed, Nov 12, 2014 at 9:50 AM, Joel Koshy <jjkoshy.w@gmail.com>
>> wrote:
>>>> 
>>>>> +1 on the JMX + gradle properties. Is there any (seamless) way of
>>>>> including the exact git hash? That would be extremely useful if users
>>>>> need help debugging and happen to be on an unreleased build (say, off
>>>>> trunk)
>>>>> 
>>>>>> On Wed, Nov 12, 2014 at 09:34:35AM -0800, Gwen Shapira wrote:
>>>>>> Actually, Jun suggested exposing this via JMX.
>>>>>> 
>>>>>> On Wed, Nov 12, 2014 at 9:31 AM, Gwen Shapira <
>> gshapira@cloudera.com>
>>>>> wrote:
>>>>>> 
>>>>>>> Good question.
>>>>>>> 
>>>>>>> The server will need to expose this in the protocol, so Kafka
>> clients
>>>>> will
>>>>>>> know what they are talking to.
>>>>>>> 
>>>>>>> We may also want to expose this in the producer and consumer,
so
>>>> people
>>>>>>> who use Kafka's built-in clients will know which version they
>> have in
>>>>> the
>>>>>>> environment.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Wed, Nov 12, 2014 at 9:09 AM, Mark Roberts <wizzat@gmail.com>
>>>>> wrote:
>>>>>>> 
>>>>>>>> Just to be clear: this is going to be exposed via some Api
the
>>>> clients
>>>>>>>> can call at startup?
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Nov 12, 2014, at 08:59, Guozhang Wang <wangguoz@gmail.com>
>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> Sounds great, +1 on this.
>>>>>>>>> 
>>>>>>>>>> On Tue, Nov 11, 2014 at 1:36 PM, Gwen Shapira <
>>>>> gshapira@cloudera.com>
>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> So it looks like we can use Gradle to add properties
to
>> manifest
>>>>> file
>>>>>>>> and
>>>>>>>>>> then use getResourceAsStream to read the file and
parse it.
>>>>>>>>>> 
>>>>>>>>>> The Gradle part would be something like:
>>>>>>>>>> jar.manifest {
>>>>>>>>>>           attributes('Implementation-Title': project.name,
>>>>>>>>>>           'Implementation-Version': project.version,
>>>>>>>>>>           'Built-By': System.getProperty('user.name'),
>>>>>>>>>>           'Built-JDK': System.getProperty('java.version'),
>>>>>>>>>>           'Built-Host': getHostname(),
>>>>>>>>>>           'Source-Compatibility':
>> project.sourceCompatibility,
>>>>>>>>>>           'Target-Compatibility': project.targetCompatibility
>>>>>>>>>>           )
>>>>>>>>>>       }
>>>>>>>>>> 
>>>>>>>>>> The code part would be:
>> this.getClass().getClassLoader().getResourceAsStream("/META-INF/MANIFEST.MF")
>>>>>>>>>> 
>>>>>>>>>> Does that look like the right approach?
>>>>>>>>>> 
>>>>>>>>>> Gwen
>>>>>>>>>> 
>>>>>>>>>> On Tue, Nov 11, 2014 at 10:43 AM, Bhavesh Mistry
<
>>>>>>>>>> mistry.p.bhavesh@gmail.com
>>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> If is maven artifact then you will get following
pre-build
>>>>> property
>>>>>>>> file
>>>>>>>>>>> from maven build called pom.properties under
>>>>>>>>>>> /META-INF/maven/groupid/artifactId/pom.properties
folder.
>>>>>>>>>>> 
>>>>>>>>>>> Here is sample:
>>>>>>>>>>> #Generated by Maven
>>>>>>>>>>> #Mon Oct 10 10:44:31 EDT 2011
>>>>>>>>>>> version=10.0.1
>>>>>>>>>>> groupId=com.google.guava
>>>>>>>>>>> artifactId=guava
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> 
>>>>>>>>>>> Bhavesh
>>>>>>>>>>> 
>>>>>>>>>>> On Tue, Nov 11, 2014 at 10:34 AM, Gwen Shapira
<
>>>>> gshapira@cloudera.com
>>>>>>>>> 
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> In Sqoop we do the following:
>>>>>>>>>>>> 
>>>>>>>>>>>> Maven runs a shell script, passing the version
as a
>> parameter.
>>>>>>>>>>>> The shell-script generates a small java class,
which is then
>>>>> built
>>>>>>>>>> with a
>>>>>>>>>>>> Maven plugin.
>>>>>>>>>>>> Our code references this generated class
when we expose
>>>>>>>> "getVersion()".
>>>>>>>>>>>> 
>>>>>>>>>>>> Its complex and ugly, so I'm kind of hoping
that there's a
>>>>> better way
>>>>>>>>>> to
>>>>>>>>>>> do
>>>>>>>>>>>> it :)
>>>>>>>>>>>> 
>>>>>>>>>>>> Gwen
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Tue, Nov 11, 2014 at 9:42 AM, Jun
Rao <junrao@gmail.com
>>> 
>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Currently, the version number is only
stored in our build
>>>> config
>>>>>>>>>> file,
>>>>>>>>>>>>> gradle.properties. Not sure how we can
automatically
>> extract
>>>> it
>>>>> and
>>>>>>>>>>>> expose
>>>>>>>>>>>>> it in an mbean. How do other projects
do this?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Jun
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Tue, Nov 11, 2014 at 7:05 AM, Otis
Gospodnetic <
>>>>>>>>>>>>> otis.gospodnetic@gmail.com> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Sounds good.  But is the version
number stored anywhere
>> from
>>>>> where
>>>>>>>>>> it
>>>>>>>>>>>>> could
>>>>>>>>>>>>>> be gotten?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Otis
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Monitoring * Alerting * Anomaly Detection
* Centralized
>> Log
>>>>>>>>>>> Management
>>>>>>>>>>>>>> Solr & Elasticsearch Support
* http://sematext.com/
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Tue, Nov 11, 2014 at 12:45 AM,
Jun Rao <
>> junrao@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Otis,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> We don't have an api for that
now. We can probably expose
>>>> this
>>>>>>>>>> as a
>>>>>>>>>>>> JMX
>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>> part of kafka-1481.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Mon, Nov 10, 2014 at 7:17
PM, Otis Gospodnetic <
>>>>>>>>>>>>>>> otis.gospodnetic@gmail.com>
wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Is there a way to detect
which version of Kafka one is
>>>>> running?
>>>>>>>>>>>>>>>> Is there an API for that,
or a constant with this
>> value, or
>>>>>>>>>> maybe
>>>>>>>>>>>> an
>>>>>>>>>>>>>>> MBean
>>>>>>>>>>>>>>>> or some other way to get
to this info?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Otis
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Monitoring * Alerting * Anomaly
Detection * Centralized
>> Log
>>>>>>>>>>>>> Management
>>>>>>>>>>>>>>>> Solr & Elasticsearch
Support * http://sematext.com/
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> -- Guozhang
>> 

Mime
View raw message