kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Smith <secs...@gmail.com>
Subject Limits of REST interface to kafka
Date Thu, 24 Sep 2015 22:04:44 GMT
At the outset, this isn't about challenging the work that has been done,
primarily by folks @ Confluence for wrapping kafka in a REST API. Clearly,
there is a use case for a REST Service and they rose up to the challenge.

That said, I am trying to evaluate the limitations of a REST service around
kafka, if any, specifically for a Spark Streaming app as a consumer?

The reason I ask because I have been asked to move my Spark Streaming app,
that's a native kafka consumer, to a REST service that provides security
(auth/encryption). And, I am trying to figure out what requirements should
I place on the REST service if I am give up my direct access to the kafka

With Spark Streaming, what I have learned is that you need to maintain a
certain ratio between Spark executors and kafka partitions to balance the
processing load evenly on the app. The consumer group re-balancing in kafka
works really well to transition a load off a dead Spark executor to an idle
one. Not sure, how these features and other features that I use almost
unknowingly in kafka translate to a REST service like the kafka-rest

More broadly, are there features in kafka that inherently run into
limitations of the stateless/REST model?



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message