spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <>
Subject Re: Reliable Streaming Receiver
Date Thu, 06 Aug 2015 01:13:24 GMT
 You could very easily strip out the BlockGenerator code from the Spark
source code and use it directly in the same way the Reliable Kafka Receiver
uses it. BTW, you should know that we will be deprecating the receiver
based approach for the Direct Kafka approach. That is quite flexible, can
give exactly-once guarantee without WAL, and is more robust and performant.
Consider using it.

On Wed, Aug 5, 2015 at 5:48 PM, Sourabh Chandak <>

> Hi,
> I am trying to replicate the Kafka Streaming Receiver for a custom version
> of Kafka and want to create a Reliable receiver. The current implementation
> uses BlockGenerator which is a private class inside Spark streaming hence I
> can't use that in my code. Can someone help me with some resources to
> tackle this issue?
> Thanks,
> Sourabh

View raw message