kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Compton <d...@danielcompton.net>
Subject Aeron, a high throughput, low latency messaging system
Date Mon, 17 Nov 2014 19:12:29 GMT
Recently there has been some interesting discussion on the Mechanical Sympathy mailing list
about Martin Thompson’s new extremely high performance, extremely low latency, open source
messaging system called Aeron. For those of you that don’t know, Martin Thompson is an expert
on wringing as much throughput and as little latency as possible out of hardware for financial
firms so his background made me very interested in this.

The high level gist is that Aeron has publishers that write messages to a partitioned, replicated,
immutable log, and subscribers that are delivered messages from the immutable log using message
offsets. Sound familiar? There are a few differences (https://groups.google.com/d/msg/mechanical-sympathy/fr7moLcsuiI/ZfUMk4QUbN8J),
the key ones being that there are no brokers in this system, and that Aeron is a pure OSI
layer 4 implementation.

At first it seems like Aeron is a competing product to Kafka, and indeed for some cases it
could be competing. However because it is implemented as a layer 4 transport, after thinking
about it for a while, I think it could be implemented into Kafka as an alternative to the
TCP transport that Kafka uses. The semantics would need to be thought through, but I think
you could implement the brokered model on top of this, as well as supporting point to point
subscriptions bypassing the brokers.  

I wanted to bring this to the groups attention and see what others thought. Aeron is still
pretty new, and there’s a lot to digest in this so I wouldn’t recommend trying to land
it in 0.8.2 :), but I thought it’s worth considering it for future development.  

The Youtube video where he talks about it is at https://www.youtube.com/watch?v=tM4YskS94b0
and the GitHub repo is at https://github.com/real-logic/Aeron. Discussion is spread across
a few threads but starts here: https://groups.google.com/d/msg/mechanical-sympathy/fr7moLcsuiI/04unxUfIJ8sJ
and https://groups.google.com/forum/#!topic/mechanical-sympathy/GrEce0gP7RU.  

Daniel.


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message