spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From manjuldixit <>
Subject Multiple Spark Streaming receiver model
Date Tue, 06 Jan 2015 19:11:44 GMT

We have a requirement of receiving live input messages from RabbitMQ and
process them into micro batches. For this we have selected SparkStreaming
and we have written a connector for RabbitMQ receiver and Spark streaming,
it is working fine. 

Now the main requirement is to receive different category of events from
different channels/queues in a spark streaming context.

Q1 : So how can I create different streaming context to receive message from
different source (may or may not be in same frequency) within a Spark

Q2: Is is advisable to use single StreamingContext to create different input
streams from different sources?

Q3: What all design consideration I need to take care in terms of specifying
no of Cores for Spark master.

Q4: Is there a was to distribute different Streaming receiving task to
different workers ?

You suggestions will be very valuable in my application design and if you
can have some example for complex application which you can share may some
URL reference or whatever ; it is very appreciated.



View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message