spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sourav Mazumder <sourav.mazumde...@gmail.com>
Subject Spark SQL with Thrift Server is very very slow and finally failing
Date Tue, 09 Jun 2015 03:52:01 GMT
Hi,

I am trying to run a SQL form a JDBC driver using Spark's Thrift Server.

I'm doing a join between a Hive Table of size around 100 GB and another
Hive Table with 10 KB, with a filter on a particular column

The query takes more than 45 minutes and then I get ExecutorLostFailure.
That is because of memory as once I increase the memory the failure happens
but after a long time.

I'm having executor memory 20 GB, Spark DRiver Memory 2 GB, Executor
Instances 2 and Executor Core 2.

Running the job using Yarn with master as 'yarn-client'.

Any idea if I'm missing any other configuration ?

Regards,
Sourav

Mime
View raw message