kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stevo Slavić <ssla...@gmail.com>
Subject Re: Offset management: client vs broker side responsibility
Date Tue, 16 Jun 2015 08:31:18 GMT
Found out that there is standard API for retrieving and committing offsets
(see
https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka
)

Problem is that the server/broker side is not extensible (see
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaApis.scala#L142
) - i.e. there is no API one can implement and deploy/configure together
with Kafka binary with support for handling unsupported or overriding
handling of already supported
offsetCommitRequest.versionId/offsetFetchRequest.versionId

It does not prevent one to implement custom offset management on client
side (instead of using standard API to commit and retrieve offsets, one can
directly talk with custom offset store) but then problem arises that no
commercial or FOSS kafka monitoring solution support it out of the box.

I know I would, but the question to Apache Kafka community is would you
like to have Kafka broker commit/fetch extensible, and then also what
committers think about this?

Kind regards,
Stevo Slavic.


On Tue, Jun 2, 2015 at 7:11 PM, Otis Gospodnetic <otis.gospodnetic@gmail.com
> wrote:

> Hi,
>
> I haven't followed the changes to offset tracking closely, other than that
> storing them in ZK is not the only option any more.
> I think what Stevo is asking about/suggesting is that there there be a
> single API from which offset information can be retrieved (e.g. by
> monitoring tools), so that monitoring tools work regardless of where one
> chose to store offsets.
> I know we'd love to have this for SPM's Kafka monitoring and can tell you
> that adding support for N different APIs for N different offset storage
> systems would be hard/time-consuming/expensive.
> But maybe this single API already exists?
>
> Thanks,
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Mon, Jun 1, 2015 at 4:41 PM, Jason Rosenberg <jbr@squareup.com> wrote:
>
> > Stevo,
> >
> > Both of the main solutions used by the high-level consumer are
> standardized
> > and supported directly by the kafka client libraries (e.g. maintaining
> > offsets in zookeeper or in kafka itself).  And for the zk case, there is
> > the consumer offset checker (which is good for monitoring).  Consumer
> > offset checker still needs to be extended for offsets stored in kafka
> > _consumer_offset topics though.
> >
> > Anyway, I'm not sure I understand your question, you want something for
> > better monitoring of all possible clients (some of which might choose to
> > manage offsets in their own way)?
> >
> > It's just not part of the kafka design to directly track individual
> > consumers.
> >
> > Jason
> >
> > On Wed, May 27, 2015 at 7:42 AM, Shady Xu <shadyxu@gmail.com> wrote:
> >
> > > I guess adding a new component will increase the complexity of the
> system
> > > structure. And if the new component consists of one or a few nodes, it
> > may
> > > becomes the bottleneck of the whole system, if it consists of many
> nodes,
> > > it will make the system even more complex.
> > >
> > > Although every solution has its downsides, I think the current one is
> > > decent.
> > >
> > > 2015-05-27 17:10 GMT+08:00 Stevo Slavić <sslavic@gmail.com>:
> > >
> > > > It could be a separate server component, does not have to be
> > > > monolith/coupled with broker.
> > > > Such solution would have benefits - single API, pluggable
> > > implementations.
> > > >
> > > > On Wed, May 27, 2015 at 8:57 AM, Shady Xu <shadyxu@gmail.com> wrote:
> > > >
> > > > > Storing and managing offsets by broker will leave high pressure on
> > the
> > > > > brokers which will affect the performance of the cluster.
> > > > >
> > > > > You can use the advanced consumer APIs, then you can get the
> offsets
> > > > either
> > > > > from zookeeper or the __consumer_offsets__ topic. On the other
> hand,
> > if
> > > > you
> > > > > use the simple consumer APIs, you mean to manage offsets yourself,
> > then
> > > > you
> > > > > should monitor them yourself, simple and plain, right?
> > > > >
> > > > > 2015-04-22 14:36 GMT+08:00 Stevo Slavić <sslavic@gmail.com>:
> > > > >
> > > > > > Hello Apache Kafka community,
> > > > > >
> > > > > > Please correct me if wrong, AFAIK currently (Kafka 0.8.2.x)
> offset
> > > > > > management responsibility is mainly client/consumer side
> > > > responsibility.
> > > > > >
> > > > > > Wouldn't it be better if it was broker side only responsibility?
> > > > > >
> > > > > > E.g. now if one wants to use custom offset management, any of
the
> > > Kafka
> > > > > > monitoring tools cannot see the offsets - they would need to
use
> > same
> > > > > > custom client implementation which is practically not possible.
> > > > > >
> > > > > > Kind regards,
> > > > > > Stevo Slavic.
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message