kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Julien Moumné <julien.mou...@quicksign.com>
Subject Re: How to use RPC mechanism in Kafka?
Date Wed, 10 Feb 2016 13:08:42 GMT
Several answers in this thread seem to indicate Kafka can not be used to 
simulate RPC.

In Joe Stein's Real-Time Distributed and Reactive Systems presentation, 
RPC style calls look like they can be part of a system built using Kafka.

Reference : https://youtu.be/wRzL-0VSpTY?t=5m19s

The video does not go into details on how to achieve the 
request/response paradigm using Kafka.

Here is a solution we came-up with, please comment if there is a better 
approach or if you see any shortcomings.

In the diagram from the video, the "Producer box" needs to :
  - accept incoming client requests
  - produce messages into a Kafka topic
  - keep a registry of pending client requests
  - consume "Response" messages on a different Kafka topic
  - respond to client requests by correlating messages issued from the 
"Response" topic

If the "Producer box" is replicated, response messages could end-up in 
the wrong "Producer box". This is due to consumer partitioning.

One way to solve this issue is to attribute a unique consumer group to 
each "Producer box". This way, each consumer will receive every 
"Response" messages.

This is clearly not optimized and we would welcome ideas to improve the 
solution.



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message