kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Holoman <jholo...@cloudera.com>
Subject Re: Programmatic Kafka version detection/extraction?
Date Sat, 15 Nov 2014 14:41:36 GMT
Seems like this aspect of Daniel's request could (would be) included here
pretty easily:

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements

Jeff

On Sat, Nov 15, 2014 at 3:29 AM, Daniel Compton <desk@danielcompton.net>
wrote:

> This has been covered in passing in the preceding threads but I'd like to
> make it explicit, can we have a command line option/script for getting the
> Kafka version (probably with Scala version too)?
>
> I ran into this recently, where we wanted to verify that the right version
> of Kafka had been installed on a number of machines. I ended up getting the
> version number from the bundled jars but it wasn't ideal. Being able to run
> kafka --version would have been great.
>
> ---
> Daniel
>
> > On 15/11/2014, at 8:33 am, Magnus Edenhill <magnus@edenhill.se> wrote:
> >
> > Hi Gwen,
> >
> > the protocol version in an InfoResponse message would be the maximum
> > supported mainline protocol version, each change to the
> > effective protocol specification bumps the version by one.
> >
> >
> > The current per-request-type version ID is unfortunately useless to the
> > client, it only allows the broker to act differently
> > on different versions of a known request type.
> >
> > Since the broker currently does not handle unknown request types
> gracefully
> > - it simply disconnects the client, which from the client's perspective
> > could mean anything and nothing - the addition of an InfoRequest/Respones
> > would allow the client to know if a desired broker-feature is available
> or
> > not and thus allows the client library to provide the application (and in
> > the end the user) with usable error message (compare "Broker
> disconnected"
> > to "offsetCommit not supported by 0.8.0 broker").
> > Having the broker return a fabricated response with the header.err field
> > set to UNKNOWN_REQUEST_TYPE would be very useful as well of course.
> >
> >
> > Regards,
> > Magnus
> >
> >
> >
> > 2014-11-14 18:17 GMT+01:00 Gwen Shapira <gshapira@cloudera.com>:
> >
> >> I'm not sure there's a single "protocol version". Each
> >> request/response has its own version ID embedded in the request
> >> itself.
> >>
> >> Broker version, broker ID and (as Joel suggested) git hash are all
> >> reasonable. And +1 for adding this to API to support non-JVM clients.
> >>
> >> Gwen
> >>
> >> On Fri, Nov 14, 2014 at 8:46 AM, Magnus Edenhill <magnus@edenhill.se>
> >> wrote:
> >>> Please let's not add dependencies on more third party
> protocols/software,
> >>> the move away from Zookeeper dependence on the clients is very much
> >> welcomed
> >>> and it'd be a pity to see a new foreign dependency added for such a
> >> trivial
> >>> thing
> >>> as propagating the version of a broker to its client.
> >>>
> >>> My suggestion is to add a new protocol request which returns:
> >>> - broker version
> >>> - protocol version
> >>> - broker id
> >>>
> >>> A generic future-proof solution would be the use of tags (or named
> TLVs):
> >>> InfoResponse: [InfoTag]
> >>> InfoTag:
> >>>   intX tag      ( KAFKA_TAG_BROKER_VERSION, KAFKA_TAG_PROTO_VERSION,
> >>> KAFKA_TAG_SSL, ... )
> >>>   intX type    ( KAFKA_TYPE_STR, KAFKA_TYPE_INT32, KAFKA_TYPE_INT64,
> >> ...)
> >>>   intX len
> >>>   bytes payload
> >>>
> >>> Local site/vendor tags could be defined in the configuration file,
> >>>
> >>>
> >>> This would also allow clients to enable/disable features based on
> >> protocol
> >>> version,
> >>> e.g., there is no point in trying offsetCommit on a 0.8.0 broker and
> the
> >>> client library
> >>> can inform the application about this early, rather than having its
> >>> offsetCommit requests
> >>> fail by connection teardown (which is not much of an error
> propagation).
> >>>
> >>>
> >>> My two cents,
> >>> Magnus
> >>>
> >>>
> >>> 2014-11-12 20:11 GMT+01:00 Mark Roberts <wizzat@gmail.com>:
> >>>
> >>>> I haven't worked much with JMX before, but some quick googling (10-20
> >>>> minutes) is very inconclusive as to how I would go about getting the
> >> server
> >>>> version I'm connecting to from a Python client.  Can someone please
> >>>> reassure me that it's relatively trivial for non Java clients to query
> >> JMX
> >>>> for the server version of every server in the cluster? Is there a
> reason
> >>>> not to include this in the API itself?
> >>>>
> >>>> -Mark
> >>>>
> >>>> On Wed, Nov 12, 2014 at 9:50 AM, Joel Koshy <jjkoshy.w@gmail.com>
> >> wrote:
> >>>>
> >>>>> +1 on the JMX + gradle properties. Is there any (seamless) way of
> >>>>> including the exact git hash? That would be extremely useful if
users
> >>>>> need help debugging and happen to be on an unreleased build (say,
off
> >>>>> trunk)
> >>>>>
> >>>>>> On Wed, Nov 12, 2014 at 09:34:35AM -0800, Gwen Shapira wrote:
> >>>>>> Actually, Jun suggested exposing this via JMX.
> >>>>>>
> >>>>>> On Wed, Nov 12, 2014 at 9:31 AM, Gwen Shapira <
> >> gshapira@cloudera.com>
> >>>>> wrote:
> >>>>>>
> >>>>>>> Good question.
> >>>>>>>
> >>>>>>> The server will need to expose this in the protocol, so
Kafka
> >> clients
> >>>>> will
> >>>>>>> know what they are talking to.
> >>>>>>>
> >>>>>>> We may also want to expose this in the producer and consumer,
so
> >>>> people
> >>>>>>> who use Kafka's built-in clients will know which version
they
> >> have in
> >>>>> the
> >>>>>>> environment.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, Nov 12, 2014 at 9:09 AM, Mark Roberts <wizzat@gmail.com>
> >>>>> wrote:
> >>>>>>>
> >>>>>>>> Just to be clear: this is going to be exposed via some
Api the
> >>>> clients
> >>>>>>>> can call at startup?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On Nov 12, 2014, at 08:59, Guozhang Wang <wangguoz@gmail.com>
> >>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Sounds great, +1 on this.
> >>>>>>>>>
> >>>>>>>>>> On Tue, Nov 11, 2014 at 1:36 PM, Gwen Shapira
<
> >>>>> gshapira@cloudera.com>
> >>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> So it looks like we can use Gradle to add properties
to
> >> manifest
> >>>>> file
> >>>>>>>> and
> >>>>>>>>>> then use getResourceAsStream to read the file
and parse it.
> >>>>>>>>>>
> >>>>>>>>>> The Gradle part would be something like:
> >>>>>>>>>> jar.manifest {
> >>>>>>>>>>           attributes('Implementation-Title':
project.name,
> >>>>>>>>>>           'Implementation-Version': project.version,
> >>>>>>>>>>           'Built-By': System.getProperty('user.name'),
> >>>>>>>>>>           'Built-JDK': System.getProperty('java.version'),
> >>>>>>>>>>           'Built-Host': getHostname(),
> >>>>>>>>>>           'Source-Compatibility':
> >> project.sourceCompatibility,
> >>>>>>>>>>           'Target-Compatibility': project.targetCompatibility
> >>>>>>>>>>           )
> >>>>>>>>>>       }
> >>>>>>>>>>
> >>>>>>>>>> The code part would be:
> >>
> this.getClass().getClassLoader().getResourceAsStream("/META-INF/MANIFEST.MF")
> >>>>>>>>>>
> >>>>>>>>>> Does that look like the right approach?
> >>>>>>>>>>
> >>>>>>>>>> Gwen
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Nov 11, 2014 at 10:43 AM, Bhavesh Mistry
<
> >>>>>>>>>> mistry.p.bhavesh@gmail.com
> >>>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> If is maven artifact then you will get following
pre-build
> >>>>> property
> >>>>>>>> file
> >>>>>>>>>>> from maven build called pom.properties under
> >>>>>>>>>>> /META-INF/maven/groupid/artifactId/pom.properties
folder.
> >>>>>>>>>>>
> >>>>>>>>>>> Here is sample:
> >>>>>>>>>>> #Generated by Maven
> >>>>>>>>>>> #Mon Oct 10 10:44:31 EDT 2011
> >>>>>>>>>>> version=10.0.1
> >>>>>>>>>>> groupId=com.google.guava
> >>>>>>>>>>> artifactId=guava
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>>
> >>>>>>>>>>> Bhavesh
> >>>>>>>>>>>
> >>>>>>>>>>> On Tue, Nov 11, 2014 at 10:34 AM, Gwen Shapira
<
> >>>>> gshapira@cloudera.com
> >>>>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> In Sqoop we do the following:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Maven runs a shell script, passing the
version as a
> >> parameter.
> >>>>>>>>>>>> The shell-script generates a small java
class, which is then
> >>>>> built
> >>>>>>>>>> with a
> >>>>>>>>>>>> Maven plugin.
> >>>>>>>>>>>> Our code references this generated class
when we expose
> >>>>>>>> "getVersion()".
> >>>>>>>>>>>>
> >>>>>>>>>>>> Its complex and ugly, so I'm kind of
hoping that there's a
> >>>>> better way
> >>>>>>>>>> to
> >>>>>>>>>>> do
> >>>>>>>>>>>> it :)
> >>>>>>>>>>>>
> >>>>>>>>>>>> Gwen
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On Tue, Nov 11, 2014 at 9:42 AM,
Jun Rao <junrao@gmail.com
> >>>
> >>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Currently, the version number is
only stored in our build
> >>>> config
> >>>>>>>>>> file,
> >>>>>>>>>>>>> gradle.properties. Not sure how
we can automatically
> >> extract
> >>>> it
> >>>>> and
> >>>>>>>>>>>> expose
> >>>>>>>>>>>>> it in an mbean. How do other projects
do this?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Tue, Nov 11, 2014 at 7:05 AM,
Otis Gospodnetic <
> >>>>>>>>>>>>> otis.gospodnetic@gmail.com> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi Jun,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Sounds good.  But is the version
number stored anywhere
> >> from
> >>>>> where
> >>>>>>>>>> it
> >>>>>>>>>>>>> could
> >>>>>>>>>>>>>> be gotten?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>> Otis
> >>>>>>>>>>>>>> --
> >>>>>>>>>>>>>> Monitoring * Alerting * Anomaly
Detection * Centralized
> >> Log
> >>>>>>>>>>> Management
> >>>>>>>>>>>>>> Solr & Elasticsearch Support
* http://sematext.com/
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Tue, Nov 11, 2014 at 12:45
AM, Jun Rao <
> >> junrao@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Otis,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> We don't have an api for
that now. We can probably expose
> >>>> this
> >>>>>>>>>> as a
> >>>>>>>>>>>> JMX
> >>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>> part of kafka-1481.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Mon, Nov 10, 2014 at
7:17 PM, Otis Gospodnetic <
> >>>>>>>>>>>>>>> otis.gospodnetic@gmail.com>
wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Is there a way to detect
which version of Kafka one is
> >>>>> running?
> >>>>>>>>>>>>>>>> Is there an API for
that, or a constant with this
> >> value, or
> >>>>>>>>>> maybe
> >>>>>>>>>>>> an
> >>>>>>>>>>>>>>> MBean
> >>>>>>>>>>>>>>>> or some other way to
get to this info?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>> Otis
> >>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>> Monitoring * Alerting
* Anomaly Detection * Centralized
> >> Log
> >>>>>>>>>>>>> Management
> >>>>>>>>>>>>>>>> Solr & Elasticsearch
Support * http://sematext.com/
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> -- Guozhang
> >>
>



-- 
Jeff Holoman
Systems Engineer
678-612-9519

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message