spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tnegi <tnegi2...@gmail.com>
Subject Spark streaming with Redis? Working with large number of model objects at spark compute nodes.
Date Mon, 16 Jun 2014 08:47:57 GMT
We are creating a real-time stream processing system with spark streaming
which uses large number (millions)  of analytic models applied to RDDs in
the many different type of streams. Since we do not know which spark node 
will process specific RDDs , we need to make these models available at each
Spark compute node. We are planning to use Redis as in-memory cache over
Spark cluster to feed these models to the Spark compute nodes. Is it the
right approach? We can not cache all models locally at all Spark compute
nodes.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-streaming-with-Redis-Working-with-large-number-of-model-objects-at-spark-compute-nodes-tp7663.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Mime
View raw message