spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Udbhav Agarwal <udbhav.agar...@syncoms.com>
Subject RE: spark sql performance
Date Fri, 13 Mar 2015 07:40:46 GMT
Okay Akhil.
I am having 4 cores cpu.(2.4 ghz)

Thanks,
Udbhav Agarwal

From: Akhil Das [mailto:akhil@sigmoidanalytics.com]
Sent: 13 March, 2015 1:07 PM
To: Udbhav Agarwal
Cc: user@spark.apache.org
Subject: Re: spark sql performance

You can see where it is spending time, whether there is any GC Time etc from the webUI (running
on 4040),Also how many cores are you having?

Thanks
Best Regards

On Fri, Mar 13, 2015 at 1:05 PM, Udbhav Agarwal <udbhav.agarwal@syncoms.com<mailto:udbhav.agarwal@syncoms.com>>
wrote:
Additionally I wanted to tell that presently I was running the query on one machine with 3gm
ram and the join query was taking around 6 seconds.

Thanks,
Udbhav Agarwal

From: Udbhav Agarwal
Sent: 13 March, 2015 12:45 PM
To: 'Akhil Das'
Cc: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: RE: spark sql performance

Okay Akhil! Thanks for the information.

Thanks,
Udbhav Agarwal

From: Akhil Das [mailto:akhil@sigmoidanalytics.com]
Sent: 13 March, 2015 12:34 PM

To: Udbhav Agarwal
Cc: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: spark sql performance

Can't say that unless you try it.

Thanks
Best Regards

On Fri, Mar 13, 2015 at 12:32 PM, Udbhav Agarwal <udbhav.agarwal@syncoms.com<mailto:udbhav.agarwal@syncoms.com>>
wrote:
Sounds great!
So can I expect response time in milliseconds from the join query over this much data ( 0.5
million in each table) ?


Thanks,
Udbhav Agarwal

From: Akhil Das [mailto:akhil@sigmoidanalytics.com<mailto:akhil@sigmoidanalytics.com>]
Sent: 13 March, 2015 12:27 PM

To: Udbhav Agarwal
Cc: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: spark sql performance

So you can cache upto 8GB of data in memory (hope your data size of one table is < 2GB),
then it should be pretty fast with SparkSQL. Also i'm assuming you have around 12-16 cores
total.

Thanks
Best Regards

On Fri, Mar 13, 2015 at 12:22 PM, Udbhav Agarwal <udbhav.agarwal@syncoms.com<mailto:udbhav.agarwal@syncoms.com>>
wrote:
Lets say am using 4 machines with 3gb ram. My data is customers records with 5 columns each
in two tables with 0.5 million records. I want to perform join query on these two tables.


Thanks,
Udbhav Agarwal

From: Akhil Das [mailto:akhil@sigmoidanalytics.com<mailto:akhil@sigmoidanalytics.com>]
Sent: 13 March, 2015 12:16 PM
To: Udbhav Agarwal
Cc: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: spark sql performance

The size/type of your data, and your cluster configuration would be fine i think.

Thanks
Best Regards

On Fri, Mar 13, 2015 at 12:07 PM, Udbhav Agarwal <udbhav.agarwal@syncoms.com<mailto:udbhav.agarwal@syncoms.com>>
wrote:
Thanks Akhil,
What more info should I give so we can estimate query time in my scenario?

Thanks,
Udbhav Agarwal

From: Akhil Das [mailto:akhil@sigmoidanalytics.com<mailto:akhil@sigmoidanalytics.com>]
Sent: 13 March, 2015 12:01 PM
To: Udbhav Agarwal
Cc: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: spark sql performance

That totally depends on your data size and your cluster setup.

Thanks
Best Regards

On Thu, Mar 12, 2015 at 7:32 PM, Udbhav Agarwal <udbhav.agarwal@syncoms.com<mailto:udbhav.agarwal@syncoms.com>>
wrote:
Hi,
What is query time for join query on hbase with spark sql. Say tables in hbase have 0.5 million
records each. I am expecting a query time (latency) in milliseconds with spark sql. Can this
be possible ?




Thanks,
Udbhav Agarwal






Mime
View raw message