kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thunder Stumpges <tstump...@ntent.com>
Subject RE: Kafka in C#
Date Fri, 09 Jan 2015 19:00:13 GMT
Hi Matti,

We are actively testing in staging, under load and with extended reliability tests. It has
been running well from a producer side for many days. We are pushing hard to get to production
in the next couple of weeks however much of the testing is not directly related to the kafka
client, but other changes in our core product.

As such, we have not had time to update documentation. Still, we would be pleased if you had
a look and evaluated it; I think it is more fully featured than any of the other .net clients.
If you do, please let me know how you get on. We'd be glad to help you out however we can.


-----Original Message-----
From: Matti Waarna [mailto:mwaarna@sapient.com] 
Sent: Friday, January 09, 2015 9:43 AM
To: users@kafka.apache.org; Thunder Stumpges
Subject: RE: Kafka in C#


Have you had a chance to go to production with the C# producer and any updates on documentation?

I will be checking out the C# library sometime next week.



-----Original Message-----
From: Thunder Stumpges [mailto:tstumpges@ntent.com]
Sent: Monday, December 22, 2014 1:10 PM
To: users@kafka.apache.org
Subject: RE: Kafka in C#

We definitely will. Wanted to wait until we got more confident the API was solid, ran in production
for a little, and added a bit more documentation.


-----Original Message-----
From: Joe Stein [joe.stein@stealth.ly]
Received: Monday, 22 Dec 2014, 10:05AM
To: users@kafka.apache.org [users@kafka.apache.org]
Subject: Re: Kafka in C#

Thunder, can you add that to
didn't know it existed but cool that it uses Rx

On Mon, Dec 22, 2014 at 12:52 PM, Thunder Stumpges <tstumpges@ntent.com>

> Hi there,
> We looked at both of these a while back and ended up writing our own ( 
> https://github.com/ntent-ad/kafka4net).
> The first one in the list was completely synchronous, and had no 
> concept of batching. We initially attempted to use the second one 
> (kafka-net) but had some issues with detecting leader changes, 
> temporary errors, and general topic metadata changes. Also had some 
> problems with the use of async and the tracking of correlation IDs on tcp messages.
> I know our client does not have a lot of documentation (yet) and says 
> "Work in progress, not ready yet!" but we have been testing this a 
> lot, and have been running in staging under load with good results. We 
> will be entering production in the next few weeks (waiting until after 
> the holidays). I wouldn't be comfortable with it going straight to 
> production without some testing in your environment, but especially 
> the Producer is very robust, and is resilient to all sorts of changes 
> in the environment (the integration tests use Vagrant and a set of VMs 
> and test Producing to non-existent partition and waiting for 
> auto-creation of topic, partition rebalancing, broker down or other 
> partition re-assignment, etc.) The client is fully Async and leverages 
> Rx (https://rx.codeplex.com/) and an event-loop-scheduler to do all 
> processing. Changes in partition state are broadcast Rx style to 
> listening components (Producer, Consumer, PartitionRecoveryMonitor).
> One of the reasons we have not finalized the documentation and notice 
> on the github page is we weren't sure if the API might change based on usage.
> To this end, we'd like to know what you think and if you have any 
> use-cases not handled by the API.
> I understand if you're not comfortable with the beta state of the 
> client, but we'd love to have you check it out. We are actively 
> developing on this and can help with any issues.
> Thanks,
> Thunder
> -----Original Message-----
> From: Matti Waarna [mailto:mwaarna@sapient.com]
> Sent: Monday, December 22, 2014 7:55 AM
> To: users@kafka.apache.org
> Subject: Kafka in C#
> We are using kafka version and trying to produce from a C# app.
> I realize that there is no official C# library release and want to get 
> your experience with the existing solutions that are currently available.
> I am looking for a solution that is a) stable enough for production 
> environment and b) performs well.
> 1) A couple of active github projects are available along with a few 
> forks each.
> Has Anybody worked on either of the following two options to 
> contribute their findings?
> https://github.com/Jroland/kafka-net
> https://github.com/miknil/Kafka4n
> 2) Also there is the option of IKVM to import kafka JARS into a .net DLL.
> Maybe even another solution?
> Thanks

View raw message