spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Soumya Simanta <>
Subject Re: Spark as Relational Database
Date Sun, 26 Oct 2014 15:13:45 GMT
@Peter - as Rick said - Spark's main usage is data analysis and not

Spark allows you to plugin different storage layers based on your use cases
and quality attribute requirements. So in essence if your relational
database is meeting your storage requirements you should think about how to
use that "with" Spark. Because even if you decide not to use your
relational database you will have to select some storage layer, most likely
a distributed storage layer.

Another option to think about is - "can you possible restructure your data
schema so that you don't have to do that a large number of joins?". If this
is an option then you can potentially think about using stores such as
Cassandra, HBase, HDFS etc.

Spark really excels at processing large volumes of data really fast (given
enough memory) on horizontally scalable commodity hardware. As Rick pointed
out - "It will probably outperform a relational star schema if all of your
*working* data set can fit into RAM on your cluster." However, if you data
size is much larger than your cluster memory you don't have a choice but to
select a datastore.



On Sun, Oct 26, 2014 at 10:05 AM, Rick Richardson <
> wrote:

> Spark's API definitely covers all of the things that a relational database
> can do. It will probably outperform a relational star schema if all of your
> *working* data set can fit into RAM on your cluster. It will still perform
> quite well if most of the data fits and some has to spill over to disk.
> What are your requirements exactly?
> What is massive amounts of data exactly?
> How big is your cluster?
> Note that Spark is not for data storage, only data analysis. It pulls data
> into working data sets called RDD's.
> As a migration path, you could probably pull the data out of a relational
> database to analyze. But in the long run, I would recommend using a more
> purpose built, huge storage database such as Cassandra. If your data is
> very static, you could also just store it in files.
>  On Oct 26, 2014 9:19 AM, "Peter Wolf" <> wrote:
>> My understanding is the SparkSQL allows one to access Spark data as if it
>> were stored in a relational database.  It compiles SQL queries into a
>> series of calls to the Spark API.
>> I need the performance of a SQL database, but I don't care about doing
>> queries with SQL.
>> I create the input to MLib by doing a massive JOIN query.  So, I am
>> creating a single collection by combining many collections.  This sort of
>> operation is very inefficient in Mongo, Cassandra or HDFS.
>> I could store my data in a relational database, and copy the query
>> results to Spark for processing.  However, I was hoping I could keep
>> everything in Spark.
>> On Sat, Oct 25, 2014 at 11:34 PM, Soumya Simanta <
>>> wrote:
>>> 1. What data store do you want to store your data in ? HDFS, HBase,
>>> Cassandra, S3 or something else?
>>> 2. Have you looked at SparkSQL (
>>> One option is to process the data in Spark and then store it in the
>>> relational database of your choice.
>>> On Sat, Oct 25, 2014 at 11:18 PM, Peter Wolf <> wrote:
>>>> Hello all,
>>>> We are considering Spark for our organization.  It is obviously a
>>>> superb platform for processing massive amounts of data... how about
>>>> retrieving it?
>>>> We are currently storing our data in a relational database in a star
>>>> schema.  Retrieving our data requires doing many complicated joins across
>>>> many tables.
>>>> Can we use Spark as a relational database?  Or, if not, can we put
>>>> Spark on top of a relational database?
>>>> Note that we don't care about SQL.  Accessing our data via standard
>>>> queries is nice, but we are equally happy (or more happy) to write Scala
>>>> code.
>>>> What is important to us is doing relational queries on huge amounts of
>>>> data.  Is Spark good at this?
>>>> Thank you very much in advance
>>>> Peter

View raw message