kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis.gospodne...@gmail.com>
Subject Re: Offset management: client vs broker side responsibility
Date Tue, 02 Jun 2015 17:11:54 GMT
Hi,

I haven't followed the changes to offset tracking closely, other than that
storing them in ZK is not the only option any more.
I think what Stevo is asking about/suggesting is that there there be a
single API from which offset information can be retrieved (e.g. by
monitoring tools), so that monitoring tools work regardless of where one
chose to store offsets.
I know we'd love to have this for SPM's Kafka monitoring and can tell you
that adding support for N different APIs for N different offset storage
systems would be hard/time-consuming/expensive.
But maybe this single API already exists?

Thanks,
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/


On Mon, Jun 1, 2015 at 4:41 PM, Jason Rosenberg <jbr@squareup.com> wrote:

> Stevo,
>
> Both of the main solutions used by the high-level consumer are standardized
> and supported directly by the kafka client libraries (e.g. maintaining
> offsets in zookeeper or in kafka itself).  And for the zk case, there is
> the consumer offset checker (which is good for monitoring).  Consumer
> offset checker still needs to be extended for offsets stored in kafka
> _consumer_offset topics though.
>
> Anyway, I'm not sure I understand your question, you want something for
> better monitoring of all possible clients (some of which might choose to
> manage offsets in their own way)?
>
> It's just not part of the kafka design to directly track individual
> consumers.
>
> Jason
>
> On Wed, May 27, 2015 at 7:42 AM, Shady Xu <shadyxu@gmail.com> wrote:
>
> > I guess adding a new component will increase the complexity of the system
> > structure. And if the new component consists of one or a few nodes, it
> may
> > becomes the bottleneck of the whole system, if it consists of many nodes,
> > it will make the system even more complex.
> >
> > Although every solution has its downsides, I think the current one is
> > decent.
> >
> > 2015-05-27 17:10 GMT+08:00 Stevo Slavić <sslavic@gmail.com>:
> >
> > > It could be a separate server component, does not have to be
> > > monolith/coupled with broker.
> > > Such solution would have benefits - single API, pluggable
> > implementations.
> > >
> > > On Wed, May 27, 2015 at 8:57 AM, Shady Xu <shadyxu@gmail.com> wrote:
> > >
> > > > Storing and managing offsets by broker will leave high pressure on
> the
> > > > brokers which will affect the performance of the cluster.
> > > >
> > > > You can use the advanced consumer APIs, then you can get the offsets
> > > either
> > > > from zookeeper or the __consumer_offsets__ topic. On the other hand,
> if
> > > you
> > > > use the simple consumer APIs, you mean to manage offsets yourself,
> then
> > > you
> > > > should monitor them yourself, simple and plain, right?
> > > >
> > > > 2015-04-22 14:36 GMT+08:00 Stevo Slavić <sslavic@gmail.com>:
> > > >
> > > > > Hello Apache Kafka community,
> > > > >
> > > > > Please correct me if wrong, AFAIK currently (Kafka 0.8.2.x) offset
> > > > > management responsibility is mainly client/consumer side
> > > responsibility.
> > > > >
> > > > > Wouldn't it be better if it was broker side only responsibility?
> > > > >
> > > > > E.g. now if one wants to use custom offset management, any of the
> > Kafka
> > > > > monitoring tools cannot see the offsets - they would need to use
> same
> > > > > custom client implementation which is practically not possible.
> > > > >
> > > > > Kind regards,
> > > > > Stevo Slavic.
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message