kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From radai <radai.rosenbl...@gmail.com>
Subject Re: Broker Interceptors
Date Wed, 04 Dec 2019 05:15:33 GMT
yeah, we tried for this a while back (kip 388 -
https://cwiki.apache.org/confluence/display/KAFKA/KIP-388%3A+Add+observer+interface+to+record+request+and+response)

its implemented in our kafka repo (linked to above)

On Tue, Dec 3, 2019 at 8:59 PM Ignacio Solis <isolis@igso.net> wrote:
>
> At LinkedIn we run a style of "read-only" interceptor we call an observer.
> We use this for usage monitoring.
> https://github.com/linkedin/kafka/commit/a378c8980af16e3c6d3f6550868ac0fd5a58682e
>
> There is always a tension between exposing internals, creating stable
> interfaces and performance.  It's understandable that upstream feels uneasy
> about this type of change.
>
> For us, the observer has become an essential tool, so we're ok if we need
> to update code.
>
> Personally, I would like to argue that a general interceptor framework
> would have been valuable from the very beginning.  Since one didn't exist,
> we've had to find one-off solutions for a few different problems.  A few
> examples that may have fit in there include up/down convert formats,
> authorizer, quotas, transaction coordinator, idempotent producer, etc.  Ok,
> ok, I'm overreaching, but you get the idea.  The chain of processes that a
> message/request goes through are basically interceptors, and we have
> decided that instead of a generic framework, we prefer to do the one-offs.
>
> Nacho
>
>
> On Tue, Dec 3, 2019 at 8:03 AM Ismael Juma <ismael@juma.me.uk> wrote:
>
> > The main challenge is doing this without exposing a bunch of internal
> > classes. I haven't seen a proposal that handles that aspect well so far.
> >
> > Ismael
> >
> > On Tue, Dec 3, 2019 at 7:21 AM Sönke Liebau
> > <soenke.liebau@opencore.com.invalid> wrote:
> >
> > > Hi Thomas,
> > >
> > > I think that idea is worth looking at. As you say, if no interceptor is
> > > configured then the performance overhead should be negligible. Basically
> > it
> > > is then up to the user to decide if he wants tomtake the performance hit.
> > > We should make sure to think about monitoring capabilities like time
> > spent
> > > in the interceptor for records etc.
> > >
> > > The most obvious use case I think is server side schema validation, which
> > > Confluent are also offering as part of their commercial product, but
> > other
> > > ideas come to mind as well.
> > >
> > > Best regards,
> > > Sönke
> > >
> > > Thomas Aley <Thomas.Aley@ibm.com> schrieb am Di., 3. Dez. 2019, 10:45:
> > >
> > > > Hi M. Manna,
> > > >
> > > > Thank you for your feedback, any and all thoughts on this are
> > appreciated
> > > > from the community.
> > > >
> > > > I think it is important to distinguish that there are two parts to
> > this.
> > > > One would be a server side interceptor framework and the other would be
> > > > the interceptor implementations themselves.
> > > >
> > > > The idea would be that the Interceptor framework manifests as a plug
> > > point
> > > > in the request/response paths that by itself has negligible performance
> > > > impact as without an interceptor registered in the framework it is
> > > > essentially a no-op. This way the out-the-box behavior of the Kafka
> > > broker
> > > > remains essentially unchanged, it is only if the cluster administrator
> > > > registers an interceptor into the framework that the path of a record
> > is
> > > > intercepted. This is much like the already accepted and implemented
> > > client
> > > > interceptors - the capability exists and it is an opt-in feature.
> > > >
> > > > As with the client interceptors and indeed interception in general, the
> > > > interceptor implementations need to be thoughtfully crafted to ensure
> > > > minimal performance impact. Yes the interceptor framework could tap
> > into
> > > > nearly everything but would only be tapping into the subset of APIs
> > that
> > > > the user wishes to intercept for their use case.
> > > >
> > > > Tom Aley
> > > > thomas.aley@ibm.com
> > > >
> > > >
> > > >
> > > > From:   "M. Manna" <manmedia@gmail.com>
> > > > To:     Kafka Users <users@kafka.apache.org>
> > > > Cc:     dev@kafka.apache.org
> > > > Date:   02/12/2019 11:31
> > > > Subject:        [EXTERNAL] Re: Broker Interceptors
> > > >
> > > >
> > > >
> > > > Hi Tom,
> > > >
> > > > On Mon, 2 Dec 2019 at 09:41, Thomas Aley <Thomas.Aley@ibm.com> wrote:
> > > >
> > > > > Hi Kafka community,
> > > > >
> > > > > I am hoping to get some feedback and thoughts about broker
> > > interceptors.
> > > > >
> > > > > KIP-42 Added Producer and Consumer interceptors which have provided
> > > > Kafka
> > > > > users the ability to collect client side metrics and trace the path
> > of
> > > > > individual messages end-to-end.
> > > > >
> > > > > This KIP also mentioned "Adding message interceptor on the broker
> > makes
> > > > a
> > > > > lot of sense, and will add more detail to monitoring. However, the
> > > > > proposal is to do it later in a separate KIP".
> > > > >
> > > > > One of the motivations for leading with client interceptors was to
> > gain
> > > > > experience and see how useable they are before tackling the server
> > side
> > > > > implementation which would ultimately "allow us to have a more
> > > > > complete/detailed message monitoring".
> > > > >
> > > > > Broker interceptors could also provide more value than just more
> > > > complete
> > > > > and detailed monitoring such as server side schema validation, so
I
> > am
> > > > > curious to learn if anyone in the community has progressed this work;
> > > > has
> > > > > ideas about other potential server side interceptor uses or has
> > > actually
> > > > > implemented something similar.
> > > > >
> > > >
> > > >  I personally feel that the cost here is the impact on performance. If
> > I
> > > > am
> > > > right, this interceptor is going to tap into nearly everything. If you
> > > > have
> > > > strong guarantee (min.in.sync.replicas = N-1) then this may incur some
> > > > delay (and let's not forget inter broker comms protection by TLS
> > config).
> > > > This may not be desirable for some systems. That said, it would be good
> > > to
> > > > know what others think about this.
> > > >
> > > > Thanks,
> > > >
> > > > >
> > > > > Regards,
> > > > >
> > > > > Tom Aley
> > > > > thomas.aley@ibm.com
> > > > > Unless stated otherwise above:
> > > > > IBM United Kingdom Limited - Registered in England and Wales with
> > > number
> > > > > 741598.
> > > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire
> > PO6
> > > > 3AU
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > Unless stated otherwise above:
> > > > IBM United Kingdom Limited - Registered in England and Wales with
> > number
> > > > 741598.
> > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
> > > 3AU
> > > >
> > > >
> > >
> >
>
>
> --
> Nacho - Ignacio Solis - isolis@igso.net

Mime
View raw message