kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Piotr HusiatyƄski <...@optiopay.com>
Subject Re: New kafka client for Go (golang)
Date Tue, 07 Apr 2015 15:57:18 GMT
Sorry if this mail is not send properly, but I have no idea how to
send in gmail response to mailing list without having original message
in mailbox.

>  How does it compare to Sarama?

There are several important differences. It's been about two months
since I looked at sarama API and I know that things changed there, but
here are the main reasons that made us decide to write our own
library:

Sarama is providing API build on channels. If you want to read or
write message from kafka, at some point you will get two or more
channels that you will have to read from. This was very problematic
for us, because it's non trivial to make sure your writes are
successful or not. You cannot also choose to make write of certain
group of messages atomic.
Kafka on the other hand provides blocking API, which is very easy to
work with and understand what is the logic flow.

Sarama's code is split into many files, approximately 150 lines
(comments and whitespaces included) of code per file. This made it
really hard for me to read the code. I believe that kafka library,
having only few files that you can read top-down requires less grep
and is a bit easier to read and understand.

Testing is another topic that in my opinion makes great difference.
This library was written from the very beginning with unit tests in
mind. It is easy to write code that is using kafka connection and test
it without real server running. This is possible because of public API
defined as set of tiny interfaces and kafkatest package that mocks all
common interfaces.

As side effect of having API build from small interfaces, it is
possible to wrap producer and consumer with custom functionality
without need to adjust any code using it. We are using this pattern to
for example transparently gather statistics about kafka usage across
clients or filter messages.

I have no idea if sarama is supporting this already, but kafka can
write consumed messages offset into kafka which is functionality
introduced in latest release. We don't really care about this
functionality though, so it's not tested with real applications.

Last worth mentioning thing is that if for some reason you don't like
connection abstraction layer, you can use only the wire protocol
implementation over any reader/writer.

Hope you will find this fair and helpful comparison.

Mime
View raw message