kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Aley <Thomas.A...@ibm.com>
Subject RE: Broker Interceptors
Date Tue, 03 Dec 2019 10:45:11 GMT
Hi M. Manna,

Thank you for your feedback, any and all thoughts on this are appreciated 
from the community.

I think it is important to distinguish that there are two parts to this. 
One would be a server side interceptor framework and the other would be 
the interceptor implementations themselves.

The idea would be that the Interceptor framework manifests as a plug point 
in the request/response paths that by itself has negligible performance 
impact as without an interceptor registered in the framework it is 
essentially a no-op. This way the out-the-box behavior of the Kafka broker 
remains essentially unchanged, it is only if the cluster administrator 
registers an interceptor into the framework that the path of a record is 
intercepted. This is much like the already accepted and implemented client 
interceptors - the capability exists and it is an opt-in feature.

As with the client interceptors and indeed interception in general, the 
interceptor implementations need to be thoughtfully crafted to ensure 
minimal performance impact. Yes the interceptor framework could tap into 
nearly everything but would only be tapping into the subset of APIs that 
the user wishes to intercept for their use case. 

Tom Aley
thomas.aley@ibm.com



From:   "M. Manna" <manmedia@gmail.com>
To:     Kafka Users <users@kafka.apache.org>
Cc:     dev@kafka.apache.org
Date:   02/12/2019 11:31
Subject:        [EXTERNAL] Re: Broker Interceptors



Hi Tom,

On Mon, 2 Dec 2019 at 09:41, Thomas Aley <Thomas.Aley@ibm.com> wrote:

> Hi Kafka community,
>
> I am hoping to get some feedback and thoughts about broker interceptors.
>
> KIP-42 Added Producer and Consumer interceptors which have provided 
Kafka
> users the ability to collect client side metrics and trace the path of
> individual messages end-to-end.
>
> This KIP also mentioned "Adding message interceptor on the broker makes 
a
> lot of sense, and will add more detail to monitoring. However, the
> proposal is to do it later in a separate KIP".
>
> One of the motivations for leading with client interceptors was to gain
> experience and see how useable they are before tackling the server side
> implementation which would ultimately "allow us to have a more
> complete/detailed message monitoring".
>
> Broker interceptors could also provide more value than just more 
complete
> and detailed monitoring such as server side schema validation, so I am
> curious to learn if anyone in the community has progressed this work; 
has
> ideas about other potential server side interceptor uses or has actually
> implemented something similar.
>

 I personally feel that the cost here is the impact on performance. If I 
am
right, this interceptor is going to tap into nearly everything. If you 
have
strong guarantee (min.in.sync.replicas = N-1) then this may incur some
delay (and let's not forget inter broker comms protection by TLS config).
This may not be desirable for some systems. That said, it would be good to
know what others think about this.

Thanks,

>
> Regards,
>
> Tom Aley
> thomas.aley@ibm.com
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 
3AU
>
>



Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message