spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paolo Platter <>
Subject R: Is SPARK is the right choice for traditional OLAP query processing?
Date Wed, 29 Jul 2015 07:54:53 GMT
Try to give a look at zoomdata. They are spark based and they offer BI features with good performance.


Inviata dal mio Windows Phone
Da: Ruslan Dautkhanov<>
Inviato: ‎29/‎07/‎2015 06:18
A: renga.kannan<>
Cc: user<>
Oggetto: Re: Is SPARK is the right choice for traditional OLAP query processing?

>> We want these use actions respond within 2 to 5 seconds.

I think this goal is a stretch for Spark. Some queries may run faster than that on a large
but in general you can't put an SLA like this. For example if you have to join some huge datasets,
you'll likely will be much over that. Spark is great for huge jobs and it'll be much faster
than MR.
I don't think Spark was designed with interactive queries in mind. For example, although Spark
"in-memory", its in-memory is only for a job. It's not like in traditional RDBMS systems where
have a persistent "buffer cache" or "in-memory columnar storage" (both are Oracle terms)
If you have multiple users running interatactive BI queries, results that were cached for
first user
wouldn't be used by second user. Unless you invent something that would keep a persistent
Spark context and serve users' requests and decided which RDDs to cache, when and how.
At least that's my understanding how Spark works. If I'm wrong, I will be glad to hear that
we ran into the same questions.

As we use Cloudera's CDH, I'm not sure where Hortonworks are with their Tez project,
but Tez has components that resemble closer to "buffer cache" or "in-memory columnar storage"
from traditional RDBMS systems, and may get better and/or more predictable performance on
BI queries.

Ruslan Dautkhanov

On Mon, Jul 20, 2015 at 6:04 PM, renga.kannan <<>>
I really appreciate anyone's input on this. We are having a very simple
traditional OLAP query processing use case. Our use case is as follows.

1. We have a customer sales order table data coming from RDBMs table.
2. There are many dimension columns in the sales order table. For each of
those dimensions, we have individual dimension tables that stores the
dimension record sets.
3. We also have some BI like hierarchies that is defined for dimension data

What we want for business users is as follows.?

1. We wanted to show some aggregated values from sales Order transaction
table columns.
2. User would like to filter these with specific dimension values from
dimension table.
3. User should be able to drill down from higher level to lower level by
traversing hierarchy on dimension

We want these use actions respond within 2 to 5 seconds.

We are thinking about using SPARK as our backend enginee to sever data to
these front end application.

Has anyone tried using SPARK for these kind of use cases. These are all
traditional use cases in BI space. If so, can SPARK respond to these queries
with in 2 to 5 seconds for large data sets.


View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:<>
For additional commands, e-mail:<>

View raw message