spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sourav Mazumder <sourav.mazumde...@gmail.com>
Subject Re: Reliability of JMS Custom Receiver in Spark Streaming JMS
Date Thu, 12 May 2016 16:49:47 GMT
Any inputs on this issue ?

Regards,
Sourav

On Tue, May 10, 2016 at 6:17 PM, Sourav Mazumder <
sourav.mazumder00@gmail.com> wrote:

> Hi,
>
> Need to get bit more understanding of reliability aspects of the Custom
> Receivers in the context of the code in spark-streaming-jms
> https://github.com/mattf/spark-streaming-jms.
>
> Based on the documentation in
> http://spark.apache.org/docs/latest/streaming-custom-receivers.html#receiver-reliability,
> I understand that if the store api is called with multiple records the
> message is reliably stored as it is a blocking call. On the other hand if
> the store api is called with a single record then it is not reliable as the
> call is returned back to the calling program before the message is stored
> appropriately.
>
> Given that I have few questions
>
> 1. Which are the store APIs that relate to multiple records ? Are they the
> ones which use scala.collection.mutable.ArrayBuffer<T
> <http://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/receiver/Receiver.html>>,
>
> scala.collection.Iterator<T
> <http://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/receiver/Receiver.html>>
> and
> java.util.Iterator<T
> <http://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/receiver/Receiver.html>>
> in the parameter signature?
>
> 2. Is there a sample code which can show how to create multiple records
> like that and send the same to appropriate store API ?
>
> 3. If I take the example of spark-streaming-jms, the onMessage method of
> JMSReceiver class calls store API with one JMSEvent. Does that mean that
> this code does not guarantee the reliability of storage of the message
> received even if storage level specified to MEMORY_AND_DISK_SER_2 ?
>
> Regards,
> Sourav
>

Mime
View raw message