I have released the first version of a new Kafka integration with Spark
that we use in the company I work for: open sourced and named Maelstrom.
It is unique compared to other solutions out there as it reuses the
Kafka Consumer connection to achieve sub-milliseconds latency.
This library has been running stable in production environment and has
been proven to be resilient to numerous production issues.
Please check out the project's page in github:
P.S. I am also looking for a job opportunity, please look me up at Linked In