Hi All,

   On submitting 20 parallel same SQL query to Spark Thrift Server, the query execution time for some queries are less than a second and some are more than 2seconds. The Spark Thrift Server logs shows all 20 queries are submitted at same time 16/06/23 12:12:01 but the result schema are at different times.

16/06/23 12:12:01 INFO SparkExecuteStatementOperation: Running query 'select distinct val2 from philips1 where key>=1000 and key<=1500

16/06/23 12:12:02 INFO SparkExecuteStatementOperation: Result Schema: ArrayBuffer(val2#2110)
16/06/23 12:12:03 INFO SparkExecuteStatementOperation: Result Schema: ArrayBuffer(val2#2182)
16/06/23 12:12:04 INFO SparkExecuteStatementOperation: Result Schema: ArrayBuffer(val2#2344)
16/06/23 12:12:05 INFO SparkExecuteStatementOperation: Result Schema: ArrayBuffer(val2#2362)

There are sufficient executors running on YARN. The concurrency is affected by Single Driver. How to improve the concurrency and what are the best practices.

Thanks,
Prabhu Joseph