spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adrian Tanase <atan...@adobe.com>
Subject Re: Spark Streaming - use the data in different jobs
Date Mon, 19 Oct 2015 13:43:36 GMT
+1 for re-publishing to pubsub if there is only transient value in the data. If you need to
query the intermediate representation then you will need to use a database.

Sharing RDDs in memory should be possible with projects like spark job server but I think
that’s overkill in this scenario.

Lastly, if there is no strong requirement to have different jobs, you might consider collapsing
the 2 jobs into one.. And simply have multiple stages that execute in the same job.

-adrian

From: Ewan Leith
Date: Monday, October 19, 2015 at 12:34 PM
To: Oded Maimon, user
Subject: RE: Spark Streaming - use the data in different jobs

Storing the data in HBase, Cassandra, or similar is possibly the right answer, the other option
that can work well is re-publishing the data back into second queue on RabbitMQ, to be read
again by the next job.

Thanks,
Ewan

From: Oded Maimon [mailto:oded@scene53.com]
Sent: 18 October 2015 12:49
To: user <user@spark.apache.org<mailto:user@spark.apache.org>>
Subject: Spark Streaming - use the data in different jobs

Hi,
we've build a spark streaming process that get data from a pub/sub (rabbitmq in our case).

now we want the streamed data to be used in different spark jobs (also in realtime if possible)

what options do we have for doing that ?


  *   can the streaming process and different spark jobs share/access the same RDD's?
  *   can the streaming process create a sparkSQL table and other jobs read/use it?
  *   can a spark streaming process trigger other spark jobs and send the the data (in memory)?
  *   can a spark streaming process cache the data in memory and other scheduled jobs access
same rdd's?
  *   should we keep the data to hbase and read it from other jobs?
  *   other ways?

I believe that the answer will be using external db/storage..  hoping to have a different
solution :)

Thanks.


Regards,
Oded Maimon
Scene53.


This email and any files transmitted with it are confidential and intended solely for the
use of the individual or entity to whom they are addressed. Please note that any disclosure,
copying or distribution of the content of this information is strictly forbidden. If you have
received this email message in error, please destroy it immediately and notify its sender.
Mime
View raw message