kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Stein <joe.st...@stealth.ly>
Subject Re: Programmatic Kafka version detection/extraction?
Date Sat, 15 Nov 2014 15:45:54 GMT
Agreed. We can have the response be the list of all brokers and each the
current version. Also an option to single broker just for that brokers
version. On start up the broker can register with meta data store
(zookeeper for now) their current version using one of the mentioned
gathered info.

Something like that, yeah.

I will create the JIRA when I get back to computer desk later today.


/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop
********************************************/
On Nov 15, 2014 9:42 AM, "Jeff Holoman" <jholoman@cloudera.com> wrote:

> Seems like this aspect of Daniel's request could (would be) included here
> pretty easily:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements
>
> Jeff
>
> On Sat, Nov 15, 2014 at 3:29 AM, Daniel Compton <desk@danielcompton.net>
> wrote:
>
> > This has been covered in passing in the preceding threads but I'd like to
> > make it explicit, can we have a command line option/script for getting
> the
> > Kafka version (probably with Scala version too)?
> >
> > I ran into this recently, where we wanted to verify that the right
> version
> > of Kafka had been installed on a number of machines. I ended up getting
> the
> > version number from the bundled jars but it wasn't ideal. Being able to
> run
> > kafka --version would have been great.
> >
> > ---
> > Daniel
> >
> > > On 15/11/2014, at 8:33 am, Magnus Edenhill <magnus@edenhill.se> wrote:
> > >
> > > Hi Gwen,
> > >
> > > the protocol version in an InfoResponse message would be the maximum
> > > supported mainline protocol version, each change to the
> > > effective protocol specification bumps the version by one.
> > >
> > >
> > > The current per-request-type version ID is unfortunately useless to the
> > > client, it only allows the broker to act differently
> > > on different versions of a known request type.
> > >
> > > Since the broker currently does not handle unknown request types
> > gracefully
> > > - it simply disconnects the client, which from the client's perspective
> > > could mean anything and nothing - the addition of an
> InfoRequest/Respones
> > > would allow the client to know if a desired broker-feature is available
> > or
> > > not and thus allows the client library to provide the application (and
> in
> > > the end the user) with usable error message (compare "Broker
> > disconnected"
> > > to "offsetCommit not supported by 0.8.0 broker").
> > > Having the broker return a fabricated response with the header.err
> field
> > > set to UNKNOWN_REQUEST_TYPE would be very useful as well of course.
> > >
> > >
> > > Regards,
> > > Magnus
> > >
> > >
> > >
> > > 2014-11-14 18:17 GMT+01:00 Gwen Shapira <gshapira@cloudera.com>:
> > >
> > >> I'm not sure there's a single "protocol version". Each
> > >> request/response has its own version ID embedded in the request
> > >> itself.
> > >>
> > >> Broker version, broker ID and (as Joel suggested) git hash are all
> > >> reasonable. And +1 for adding this to API to support non-JVM clients.
> > >>
> > >> Gwen
> > >>
> > >> On Fri, Nov 14, 2014 at 8:46 AM, Magnus Edenhill <magnus@edenhill.se>
> > >> wrote:
> > >>> Please let's not add dependencies on more third party
> > protocols/software,
> > >>> the move away from Zookeeper dependence on the clients is very much
> > >> welcomed
> > >>> and it'd be a pity to see a new foreign dependency added for such a
> > >> trivial
> > >>> thing
> > >>> as propagating the version of a broker to its client.
> > >>>
> > >>> My suggestion is to add a new protocol request which returns:
> > >>> - broker version
> > >>> - protocol version
> > >>> - broker id
> > >>>
> > >>> A generic future-proof solution would be the use of tags (or named
> > TLVs):
> > >>> InfoResponse: [InfoTag]
> > >>> InfoTag:
> > >>>   intX tag      ( KAFKA_TAG_BROKER_VERSION, KAFKA_TAG_PROTO_VERSION,
> > >>> KAFKA_TAG_SSL, ... )
> > >>>   intX type    ( KAFKA_TYPE_STR, KAFKA_TYPE_INT32, KAFKA_TYPE_INT64,
> > >> ...)
> > >>>   intX len
> > >>>   bytes payload
> > >>>
> > >>> Local site/vendor tags could be defined in the configuration file,
> > >>>
> > >>>
> > >>> This would also allow clients to enable/disable features based on
> > >> protocol
> > >>> version,
> > >>> e.g., there is no point in trying offsetCommit on a 0.8.0 broker and
> > the
> > >>> client library
> > >>> can inform the application about this early, rather than having its
> > >>> offsetCommit requests
> > >>> fail by connection teardown (which is not much of an error
> > propagation).
> > >>>
> > >>>
> > >>> My two cents,
> > >>> Magnus
> > >>>
> > >>>
> > >>> 2014-11-12 20:11 GMT+01:00 Mark Roberts <wizzat@gmail.com>:
> > >>>
> > >>>> I haven't worked much with JMX before, but some quick googling
> (10-20
> > >>>> minutes) is very inconclusive as to how I would go about getting
the
> > >> server
> > >>>> version I'm connecting to from a Python client.  Can someone please
> > >>>> reassure me that it's relatively trivial for non Java clients to
> query
> > >> JMX
> > >>>> for the server version of every server in the cluster? Is there
a
> > reason
> > >>>> not to include this in the API itself?
> > >>>>
> > >>>> -Mark
> > >>>>
> > >>>> On Wed, Nov 12, 2014 at 9:50 AM, Joel Koshy <jjkoshy.w@gmail.com>
> > >> wrote:
> > >>>>
> > >>>>> +1 on the JMX + gradle properties. Is there any (seamless)
way of
> > >>>>> including the exact git hash? That would be extremely useful
if
> users
> > >>>>> need help debugging and happen to be on an unreleased build
(say,
> off
> > >>>>> trunk)
> > >>>>>
> > >>>>>> On Wed, Nov 12, 2014 at 09:34:35AM -0800, Gwen Shapira
wrote:
> > >>>>>> Actually, Jun suggested exposing this via JMX.
> > >>>>>>
> > >>>>>> On Wed, Nov 12, 2014 at 9:31 AM, Gwen Shapira <
> > >> gshapira@cloudera.com>
> > >>>>> wrote:
> > >>>>>>
> > >>>>>>> Good question.
> > >>>>>>>
> > >>>>>>> The server will need to expose this in the protocol,
so Kafka
> > >> clients
> > >>>>> will
> > >>>>>>> know what they are talking to.
> > >>>>>>>
> > >>>>>>> We may also want to expose this in the producer and
consumer, so
> > >>>> people
> > >>>>>>> who use Kafka's built-in clients will know which version
they
> > >> have in
> > >>>>> the
> > >>>>>>> environment.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Wed, Nov 12, 2014 at 9:09 AM, Mark Roberts <wizzat@gmail.com>
> > >>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Just to be clear: this is going to be exposed via
some Api the
> > >>>> clients
> > >>>>>>>> can call at startup?
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> On Nov 12, 2014, at 08:59, Guozhang Wang <wangguoz@gmail.com>
> > >>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>> Sounds great, +1 on this.
> > >>>>>>>>>
> > >>>>>>>>>> On Tue, Nov 11, 2014 at 1:36 PM, Gwen Shapira
<
> > >>>>> gshapira@cloudera.com>
> > >>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> So it looks like we can use Gradle to add
properties to
> > >> manifest
> > >>>>> file
> > >>>>>>>> and
> > >>>>>>>>>> then use getResourceAsStream to read the
file and parse it.
> > >>>>>>>>>>
> > >>>>>>>>>> The Gradle part would be something like:
> > >>>>>>>>>> jar.manifest {
> > >>>>>>>>>>           attributes('Implementation-Title':
project.name,
> > >>>>>>>>>>           'Implementation-Version': project.version,
> > >>>>>>>>>>           'Built-By': System.getProperty('user.name'),
> > >>>>>>>>>>           'Built-JDK': System.getProperty('java.version'),
> > >>>>>>>>>>           'Built-Host': getHostname(),
> > >>>>>>>>>>           'Source-Compatibility':
> > >> project.sourceCompatibility,
> > >>>>>>>>>>           'Target-Compatibility': project.targetCompatibility
> > >>>>>>>>>>           )
> > >>>>>>>>>>       }
> > >>>>>>>>>>
> > >>>>>>>>>> The code part would be:
> > >>
> >
> this.getClass().getClassLoader().getResourceAsStream("/META-INF/MANIFEST.MF")
> > >>>>>>>>>>
> > >>>>>>>>>> Does that look like the right approach?
> > >>>>>>>>>>
> > >>>>>>>>>> Gwen
> > >>>>>>>>>>
> > >>>>>>>>>> On Tue, Nov 11, 2014 at 10:43 AM, Bhavesh
Mistry <
> > >>>>>>>>>> mistry.p.bhavesh@gmail.com
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> If is maven artifact then you will
get following pre-build
> > >>>>> property
> > >>>>>>>> file
> > >>>>>>>>>>> from maven build called pom.properties
under
> > >>>>>>>>>>> /META-INF/maven/groupid/artifactId/pom.properties
folder.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Here is sample:
> > >>>>>>>>>>> #Generated by Maven
> > >>>>>>>>>>> #Mon Oct 10 10:44:31 EDT 2011
> > >>>>>>>>>>> version=10.0.1
> > >>>>>>>>>>> groupId=com.google.guava
> > >>>>>>>>>>> artifactId=guava
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Bhavesh
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Tue, Nov 11, 2014 at 10:34 AM, Gwen
Shapira <
> > >>>>> gshapira@cloudera.com
> > >>>>>>>>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> In Sqoop we do the following:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Maven runs a shell script, passing
the version as a
> > >> parameter.
> > >>>>>>>>>>>> The shell-script generates a small
java class, which is then
> > >>>>> built
> > >>>>>>>>>> with a
> > >>>>>>>>>>>> Maven plugin.
> > >>>>>>>>>>>> Our code references this generated
class when we expose
> > >>>>>>>> "getVersion()".
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Its complex and ugly, so I'm kind
of hoping that there's a
> > >>>>> better way
> > >>>>>>>>>> to
> > >>>>>>>>>>> do
> > >>>>>>>>>>>> it :)
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Gwen
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> On Tue, Nov 11, 2014 at 9:42
AM, Jun Rao <junrao@gmail.com
> > >>>
> > >>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Currently, the version number
is only stored in our build
> > >>>> config
> > >>>>>>>>>> file,
> > >>>>>>>>>>>>> gradle.properties. Not sure
how we can automatically
> > >> extract
> > >>>> it
> > >>>>> and
> > >>>>>>>>>>>> expose
> > >>>>>>>>>>>>> it in an mbean. How do other
projects do this?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Tue, Nov 11, 2014 at 7:05
AM, Otis Gospodnetic <
> > >>>>>>>>>>>>> otis.gospodnetic@gmail.com>
wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hi Jun,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Sounds good.  But is the
version number stored anywhere
> > >> from
> > >>>>> where
> > >>>>>>>>>> it
> > >>>>>>>>>>>>> could
> > >>>>>>>>>>>>>> be gotten?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>> Otis
> > >>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>> Monitoring * Alerting *
Anomaly Detection * Centralized
> > >> Log
> > >>>>>>>>>>> Management
> > >>>>>>>>>>>>>> Solr & Elasticsearch
Support * http://sematext.com/
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Tue, Nov 11, 2014 at
12:45 AM, Jun Rao <
> > >> junrao@gmail.com>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Otis,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> We don't have an api
for that now. We can probably expose
> > >>>> this
> > >>>>>>>>>> as a
> > >>>>>>>>>>>> JMX
> > >>>>>>>>>>>>>> as
> > >>>>>>>>>>>>>>> part of kafka-1481.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Mon, Nov 10, 2014
at 7:17 PM, Otis Gospodnetic <
> > >>>>>>>>>>>>>>> otis.gospodnetic@gmail.com>
wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Hi,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Is there a way
to detect which version of Kafka one is
> > >>>>> running?
> > >>>>>>>>>>>>>>>> Is there an API
for that, or a constant with this
> > >> value, or
> > >>>>>>>>>> maybe
> > >>>>>>>>>>>> an
> > >>>>>>>>>>>>>>> MBean
> > >>>>>>>>>>>>>>>> or some other way
to get to this info?
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>> Otis
> > >>>>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>>>> Monitoring * Alerting
* Anomaly Detection * Centralized
> > >> Log
> > >>>>>>>>>>>>> Management
> > >>>>>>>>>>>>>>>> Solr & Elasticsearch
Support * http://sematext.com/
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> --
> > >>>>>>>>> -- Guozhang
> > >>
> >
>
>
>
> --
> Jeff Holoman
> Systems Engineer
> 678-612-9519
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message