spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: spark sql performance
Date Fri, 13 Mar 2015 07:37:24 GMT
You can see where it is spending time, whether there is any GC Time etc
from the webUI (running on 4040),Also how many cores are you having?

Thanks
Best Regards

On Fri, Mar 13, 2015 at 1:05 PM, Udbhav Agarwal <udbhav.agarwal@syncoms.com>
wrote:

>  Additionally I wanted to tell that presently I was running the query on
> one machine with 3gm ram and the join query was taking around 6 seconds.
>
>
>
> *Thanks,*
>
> *Udbhav Agarwal*
>
>
>
> *From:* Udbhav Agarwal
> *Sent:* 13 March, 2015 12:45 PM
> *To:* 'Akhil Das'
> *Cc:* user@spark.apache.org
> *Subject:* RE: spark sql performance
>
>
>
> Okay Akhil! Thanks for the information.
>
>
>
> *Thanks,*
>
> *Udbhav Agarwal*
>
>
>
> *From:* Akhil Das [mailto:akhil@sigmoidanalytics.com
> <akhil@sigmoidanalytics.com>]
> *Sent:* 13 March, 2015 12:34 PM
>
> *To:* Udbhav Agarwal
> *Cc:* user@spark.apache.org
> *Subject:* Re: spark sql performance
>
>
>
> Can't say that unless you try it.
>
>
>   Thanks
>
> Best Regards
>
>
>
> On Fri, Mar 13, 2015 at 12:32 PM, Udbhav Agarwal <
> udbhav.agarwal@syncoms.com> wrote:
>
>  Sounds great!
>
> So can I expect response time in milliseconds from the join query over
> this much data ( 0.5 million in each table) ?
>
>
>
>
>
> *Thanks,*
>
> *Udbhav Agarwal*
>
>
>
> *From:* Akhil Das [mailto:akhil@sigmoidanalytics.com]
> *Sent:* 13 March, 2015 12:27 PM
>
>
> *To:* Udbhav Agarwal
> *Cc:* user@spark.apache.org
> *Subject:* Re: spark sql performance
>
>
>
> So you can cache upto 8GB of data in memory (hope your data size of one
> table is < 2GB), then it should be pretty fast with SparkSQL. Also i'm
> assuming you have around 12-16 cores total.
>
>
>   Thanks
>
> Best Regards
>
>
>
> On Fri, Mar 13, 2015 at 12:22 PM, Udbhav Agarwal <
> udbhav.agarwal@syncoms.com> wrote:
>
>  Lets say am using 4 machines with 3gb ram. My data is customers records
> with 5 columns each in two tables with 0.5 million records. I want to
> perform join query on these two tables.
>
>
>
>
>
> *Thanks,*
>
> *Udbhav Agarwal*
>
>
>
> *From:* Akhil Das [mailto:akhil@sigmoidanalytics.com]
> *Sent:* 13 March, 2015 12:16 PM
> *To:* Udbhav Agarwal
> *Cc:* user@spark.apache.org
> *Subject:* Re: spark sql performance
>
>
>
> The size/type of your data, and your cluster configuration would be fine i
> think.
>
>
>   Thanks
>
> Best Regards
>
>
>
> On Fri, Mar 13, 2015 at 12:07 PM, Udbhav Agarwal <
> udbhav.agarwal@syncoms.com> wrote:
>
>  Thanks Akhil,
>
> What more info should I give so we can estimate query time in my scenario?
>
>
>
> *Thanks,*
>
> *Udbhav Agarwal*
>
>
>
> *From:* Akhil Das [mailto:akhil@sigmoidanalytics.com]
> *Sent:* 13 March, 2015 12:01 PM
> *To:* Udbhav Agarwal
> *Cc:* user@spark.apache.org
> *Subject:* Re: spark sql performance
>
>
>
> That totally depends on your data size and your cluster setup.
>
>
>   Thanks
>
> Best Regards
>
>
>
> On Thu, Mar 12, 2015 at 7:32 PM, Udbhav Agarwal <
> udbhav.agarwal@syncoms.com> wrote:
>
>  Hi,
>
> What is query time for join query on hbase with spark sql. Say tables in
> hbase have 0.5 million records each. I am expecting a query time (latency)
> in milliseconds with spark sql. Can this be possible ?
>
>
>
>
>
>
>
>
>
> *Thanks,*
>
> *Udbhav Agarwal*
>
>
>
>
>
>
>
>
>
>
>

Mime
View raw message