kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Taylor <gary.tay...@hismessages.com>
Subject Re: Developing with kafka and other non jvm languages
Date Tue, 27 Mar 2018 06:20:11 GMT
Hi Robin,
   Maybe this is where I have been getting confused - when I read the docs about the regex
support for topics - I probably made a big assumption it was a server side thing.  Is this
not the case ? would explain why Ive not found much support for it lol



> On 26 Mar 2018, at 16:18, Dmitriy Vsekhvalnov <dvsekhvalnov@gmail.com> wrote:
> Hi Gary,
> don't have experience with other go libs (they seems to be way younger),
> but Sarama is quite low level, which is both at same time powerful and to
> some extent more complicated to work with.
> With pure Sarama client you have to implement wildcard (or pattern based)
> topic subscription yourself. In all clients in all languages it is usually
> as simple as:
> - every X seconds (minutes, e.tc.) refresh metadata (see
> Client.RefreshMetadata(..))
> - diff new metadata with old metadata = removed topics/added topics
> - stop consuming for removed topics
> - start consuming for new topics
> On Mon, Mar 26, 2018 at 11:58 AM, Gary Taylor <gary.taylor@hismessages.com>
> wrote:
>> Hi Robin - thanks for that.  I started off with sarama, but found it did
>> not (seem to) support regex matching of the topics - I tried all sorts of
>> ‘regex as strings’ as the topic - but it has to be a string according to
>> the interface.
>> I will dig deeper though and look at those other clients too
>> Cheers
>> Gary
>>> On 26 Mar 2018, at 09:54, Robin Bartholdson <
>> robin.bartholdson@formulate.se> wrote:
>>> For go I would seriously consider using a pure go client, for the
>> reasons listed in https://dave.cheney.net/2016/01/18/cgo-is-not-go <
>> https://dave.cheney.net/2016/01/18/cgo-is-not-go>.
>>> There is now a wealth of pure go Kafka clients to choose from:
>>> - https://github.com/Shopify/sarama <https://github.com/Shopify/sarama>
>>> - https://github.com/segmentio/kafka-go/ <https://github.com/segmentio/
>> kafka-go/>
>>> - https://github.com/optiopay/kafka <https://github.com/optiopay/kafka>
>>> The functionality you are looking for is built into sarama-cluster, the
>> package that provides consumer group functionality on top of sarama:
>>> https://github.com/bsm/sarama-cluster/blob/master/config.go#L69-L75 <
>> https://github.com/bsm/sarama-cluster/blob/master/config.go#L69-L75>
>>> If you for some reason don’t want to use sarama-cluster, you can
>> implement it yourself quite easily: Every now and then, as metadata is
>> being refreshed, just cycle through the list of topics and determine if any
>> of the topics are worth subscribing to:
>>> https://github.com/bsm/sarama-cluster/blob/master/consumer.go#L421-L448
>> <https://github.com/bsm/sarama-cluster/blob/master/consumer.go#L421-L448>
>>> For each of the clients these are the relevant methods to use to grab
>> the list of topics:
>>> - sarama: Topics(): https://godoc.org/github.com/Shopify/sarama#Client <
>> https://godoc.org/github.com/Shopify/sarama#Client>
>>> - kafka-go: ReadPartitions(): https://godoc.org/github.com/
>> segmentio/kafka-go#Conn.ReadPartitions <https://godoc.org/github.com/
>> segmentio/kafka-go#Conn.ReadPartitions>
>>> - optiopay/kafka: Metadata(): https://godoc.org/github.com/
>> optiopay/kafka#Broker.Metadata <https://godoc.org/github.com/
>> optiopay/kafka#Broker.Metadata>
>>> -Robin
>>>> On 25 Mar 2018, at 09:46, Gary Taylor <garydavidtaylor@gmail.com>
>> wrote:
>>>> Hi,
>>>> This is a fairly generic question but has some specifics too
>>>> Ill ask the specific first - I am trying to use golang to talk to kafka
>> and it works, but a fairly important part of my application is to subscribe
>> to information in many topics where the topic is matched server side and
>> will include new topics added since the subscribe.  I have read that kafka
>> can do this, but I cannot see any way of asking it to do it via golang
>> apart from with the library https://github.com/
>> confluentinc/confluent-kafka-go <https://github.com/
>> confluentinc/confluent-kafka-go> - which uses a library called librdkafka
>> which I would have to compile and I am struggling as I am using an arm
>> processor (long story behind that one - but I don’t think this solution
>> will ever work with an arm processor).  All other libraries seem to not
>> offer this.  Is this because of some underlying restriction in kafka or do
>> you think it is going to be a case of keep hunting until I find one that
>> does ?
>>>> Then the more generic question - I get the feeling that Scala and Java
>> are first class citizens when it comes to using kafka.  Whilst I love Scala
>> - I am learning golang and am more generally a ruby developer.  Am I likely
>> go get a ‘second class’ service with these other languages - with
>> restrictions around zookeeper for example (I think I read something about
>> having to have knowledge of which partition to subscribe to - where
>> zookeeper would normally track this but there is no API for it) ?
>>>> Many Thanks
>>>> Gary Taylor

View raw message