spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrés Ivaldi <iaiva...@gmail.com>
Subject Re: Low Latency SQL query
Date Tue, 01 Dec 2015 20:58:00 GMT
Yes,
The use case would be,
Have spark in a service (I didnt invertigate this yet), through api calls
of this service we perform some aggregations over data in SQL, We are
already doing this with an internal development

Nothing complicated, for instance, a table with Product, Product Family,
cost, price, etc. Columns like Dimension and Measures,

I want to Spark for query that table to perform a kind of rollup, with cost
as Measure and Prodcut, Product Family as Dimension

Only 3 columns, it takes like 20s to perform that query and the
aggregation, the  query directly to the database with a grouping at the
columns takes like 1s

regards



On Tue, Dec 1, 2015 at 5:38 PM, Jörn Franke <jornfranke@gmail.com> wrote:

> can you elaborate more on the use case?
>
> > On 01 Dec 2015, at 20:51, Andrés Ivaldi <iaivaldi@gmail.com> wrote:
> >
> > Hi,
> >
> > I'd like to use spark to perform some transformations over data stored
> inSQL, but I need low Latency, I'm doing some test and I run into spark
> context creation and data query over SQL takes too long time.
> >
> > Any idea for speed up the process?
> >
> > regards.
> >
> > --
> > Ing. Ivaldi Andres
>



-- 
Ing. Ivaldi Andres

Mime
View raw message