Hi Mich,

I think these comparisons are useful. One interesting aspect could be hardware scalability in this context. Additionally different type of computations. Furthermore, one could compare Spark and Tez+llap as execution engines. I have the gut feeling that  each one can be justified by different use cases.
Nevertheless, there should be always a disclaimer for such comparisons, because Spark and Hive are not good for a lot of concurrent lookups of single rows. They are not good for frequently write small amounts of data (eg sensor data). Here hbase could be more interesting. Other use cases can justify graph databases, such as Titan, or text analytics/ data matching using Solr on Hadoop.
Finally, even if you have a lot of data you need to think if you always have to process everything. For instance, I have found valid use cases in practice where we decided to evaluate 10 machine learning models in parallel on only a sample of data and only evaluate the "winning" model of the total of data.

As always it depends :) 

Best regards

P.s.: at least Hortonworks has in their distribution spark 1.5 with hive 1.2 and spark 1.6 with hive 1.2. Maybe they have somewhere described how to manage bringing both together. You may check also Apache Bigtop (vendor neutral distribution) on how they managed to bring both together.

On 23 May 2016, at 01:42, Mich Talebzadeh <mich.talebzadeh@gmail.com> wrote:

Hi,

 

I have done a number of extensive tests using Spark-shell with Hive DB and ORC tables.

 

Now one issue that we typically face is and I quote:

 

Spark is fast as it uses Memory and DAG. Great but when we save data it is not fast enough


OK but there is a solution now. If you use Spark with Hive and you are on a descent version of Hive >= 0.14, then you can also deploy Spark as execution engine for Hive. That will make your application run pretty fast as you no longer rely on the old Map-Reduce for Hive engine. In a nutshell what you are gaining speed in both querying and storage.

 

I have made some comparisons on this set-up and I am sure some of you will find it useful.

 

The version of Spark I use for Spark queries (Spark as query tool) is 1.6.

The version of Hive I use in Hive 2

The version of Spark I use as Hive execution engine is 1.3.1 It works and frankly Spark 1.3.1 as an execution engine is adequate (until we sort out the Hadoop libraries mismatch).

 

An example I am using Hive on Spark engine to find the min and max of IDs for a table with 1 billion rows:

 

0: jdbc:hive2://rhes564:10010/default>  select min(id), max(id),avg(id), stddev(id) from oraclehadoop.dummy;

Query ID = hduser_20160523002031_3e22e26e-4293-4e90-ae8b-72fe9683c006

 

 

Starting Spark Job = 5e092ef9-d798-4952-b156-74df49da9151

 

INFO  : Completed compiling command(queryId=hduser_20160523002031_3e22e26e-4293-4e90-ae8b-72fe9683c006); Time taken: 1.911 seconds

INFO  : Executing command(queryId=hduser_20160523002031_3e22e26e-4293-4e90-ae8b-72fe9683c006): select min(id), max(id),avg(id), stddev(id) from oraclehadoop.dummy

INFO  : Query ID = hduser_20160523002031_3e22e26e-4293-4e90-ae8b-72fe9683c006

INFO  : Total jobs = 1

INFO  : Launching Job 1 out of 1

INFO  : Starting task [Stage-1:MAPRED] in serial mode

 

Query Hive on Spark job[0] stages:

0

1

Status: Running (Hive on Spark job[0])

Job Progress Format

CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount [StageCost]

2016-05-23 00:21:19,062 Stage-0_0: 0/22 Stage-1_0: 0/1

2016-05-23 00:21:20,070 Stage-0_0: 0(+12)/22    Stage-1_0: 0/1

2016-05-23 00:21:23,119 Stage-0_0: 0(+12)/22    Stage-1_0: 0/1

2016-05-23 00:21:26,156 Stage-0_0: 13(+9)/22    Stage-1_0: 0/1

INFO  :

Query Hive on Spark job[0] stages:

INFO  : 0

INFO  : 1

INFO  :

Status: Running (Hive on Spark job[0])

INFO  : Job Progress Format

CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount [StageCost]

INFO  : 2016-05-23 00:21:19,062 Stage-0_0: 0/22 Stage-1_0: 0/1

INFO  : 2016-05-23 00:21:20,070 Stage-0_0: 0(+12)/22    Stage-1_0: 0/1

INFO  : 2016-05-23 00:21:23,119 Stage-0_0: 0(+12)/22    Stage-1_0: 0/1

INFO  : 2016-05-23 00:21:26,156 Stage-0_0: 13(+9)/22    Stage-1_0: 0/1

2016-05-23 00:21:29,181 Stage-0_0: 22/22 Finished       Stage-1_0: 0(+1)/1

2016-05-23 00:21:30,189 Stage-0_0: 22/22 Finished       Stage-1_0: 1/1 Finished

Status: Finished successfully in 53.25 seconds

OK

INFO  : 2016-05-23 00:21:29,181 Stage-0_0: 22/22 Finished       Stage-1_0: 0(+1)/1

INFO  : 2016-05-23 00:21:30,189 Stage-0_0: 22/22 Finished       Stage-1_0: 1/1 Finished

INFO  : Status: Finished successfully in 53.25 seconds

INFO  : Completed executing command(queryId=hduser_20160523002031_3e22e26e-4293-4e90-ae8b-72fe9683c006); Time taken: 56.337 seconds

INFO  : OK

+-----+------------+---------------+-----------------------+--+

| c0  |     c1     |      c2       |          c3           |

+-----+------------+---------------+-----------------------+--+

| 1   | 100000000  | 5.00000005E7  | 2.8867513459481288E7  |

+-----+------------+---------------+-----------------------+--+

1 row selected (58.529 seconds)

 

58 seconds first run with cold cache is pretty good

 

And let us compare it with running the same query on map-reduce engine

 

: jdbc:hive2://rhes564:10010/default> set hive.execution.engine=mr;

Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.

No rows affected (0.007 seconds)

0: jdbc:hive2://rhes564:10010/default>  select min(id), max(id),avg(id), stddev(id) from oraclehadoop.dummy;

WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.

Query ID = hduser_20160523002632_9f91d42a-ea46-4a66-a589-7d39c23b41dc

Total jobs = 1

Launching Job 1 out of 1

Number of reduce tasks determined at compile time: 1

In order to change the average load for a reducer (in bytes):

  set hive.exec.reducers.bytes.per.reducer=<number>

In order to limit the maximum number of reducers:

  set hive.exec.reducers.max=<number>

In order to set a constant number of reducers:

  set mapreduce.job.reduces=<number>

Starting Job = job_1463956731753_0005, Tracking URL = http://localhost.localdomain:8088/proxy/application_1463956731753_0005/

Kill Command = /home/hduser/hadoop-2.6.0/bin/hadoop job  -kill job_1463956731753_0005

Hadoop job information for Stage-1: number of mappers: 22; number of reducers: 1

2016-05-23 00:26:38,127 Stage-1 map = 0%,  reduce = 0%

INFO  : Compiling command(queryId=hduser_20160523002632_9f91d42a-ea46-4a66-a589-7d39c23b41dc): select min(id), max(id),avg(id), stddev(id) from oraclehadoop.dummy

INFO  : Semantic Analysis Completed

INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:c0, type:int, comment:null), FieldSchema(name:c1, type:int, comment:null), FieldSchema(name:c2, type:double, comment:null), FieldSchema(name:c3, type:double, comment:null)], properties:null)

INFO  : Completed compiling command(queryId=hduser_20160523002632_9f91d42a-ea46-4a66-a589-7d39c23b41dc); Time taken: 0.144 seconds

INFO  : Executing command(queryId=hduser_20160523002632_9f91d42a-ea46-4a66-a589-7d39c23b41dc): select min(id), max(id),avg(id), stddev(id) from oraclehadoop.dummy

WARN  : Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.

INFO  : WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.

WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.

INFO  : Query ID = hduser_20160523002632_9f91d42a-ea46-4a66-a589-7d39c23b41dc

INFO  : Total jobs = 1

INFO  : Launching Job 1 out of 1

INFO  : Starting task [Stage-1:MAPRED] in serial mode

INFO  : Number of reduce tasks determined at compile time: 1

INFO  : In order to change the average load for a reducer (in bytes):

INFO  :   set hive.exec.reducers.bytes.per.reducer=<number>

INFO  : In order to limit the maximum number of reducers:

INFO  :   set hive.exec.reducers.max=<number>

INFO  : In order to set a constant number of reducers:

INFO  :   set mapreduce.job.reduces=<number>

WARN  : Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.

INFO  : number of splits:22

INFO  : Submitting tokens for job: job_1463956731753_0005

INFO  : The url to track the job: http://localhost.localdomain:8088/proxy/application_1463956731753_0005/

INFO  : Starting Job = job_1463956731753_0005, Tracking URL = http://localhost.localdomain:8088/proxy/application_1463956731753_0005/

INFO  : Kill Command = /home/hduser/hadoop-2.6.0/bin/hadoop job  -kill job_1463956731753_0005

INFO  : Hadoop job information for Stage-1: number of mappers: 22; number of reducers: 1

INFO  : 2016-05-23 00:26:38,127 Stage-1 map = 0%,  reduce = 0%

2016-05-23 00:26:44,367 Stage-1 map = 5%,  reduce = 0%, Cumulative CPU 4.56 sec

INFO  : 2016-05-23 00:26:44,367 Stage-1 map = 5%,  reduce = 0%, Cumulative CPU 4.56 sec

2016-05-23 00:26:50,558 Stage-1 map = 9%,  reduce = 0%, Cumulative CPU 9.17 sec

INFO  : 2016-05-23 00:26:50,558 Stage-1 map = 9%,  reduce = 0%, Cumulative CPU 9.17 sec

2016-05-23 00:26:56,747 Stage-1 map = 14%,  reduce = 0%, Cumulative CPU 14.04 sec

INFO  : 2016-05-23 00:26:56,747 Stage-1 map = 14%,  reduce = 0%, Cumulative CPU 14.04 sec

2016-05-23 00:27:02,944 Stage-1 map = 18%,  reduce = 0%, Cumulative CPU 18.64 sec

INFO  : 2016-05-23 00:27:02,944 Stage-1 map = 18%,  reduce = 0%, Cumulative CPU 18.64 sec

2016-05-23 00:27:08,105 Stage-1 map = 23%,  reduce = 0%, Cumulative CPU 23.25 sec

INFO  : 2016-05-23 00:27:08,105 Stage-1 map = 23%,  reduce = 0%, Cumulative CPU 23.25 sec

2016-05-23 00:27:14,298 Stage-1 map = 27%,  reduce = 0%, Cumulative CPU 27.84 sec

INFO  : 2016-05-23 00:27:14,298 Stage-1 map = 27%,  reduce = 0%, Cumulative CPU 27.84 sec

2016-05-23 00:27:20,484 Stage-1 map = 32%,  reduce = 0%, Cumulative CPU 32.56 sec

INFO  : 2016-05-23 00:27:20,484 Stage-1 map = 32%,  reduce = 0%, Cumulative CPU 32.56 sec

2016-05-23 00:27:26,659 Stage-1 map = 36%,  reduce = 0%, Cumulative CPU 37.1 sec

INFO  : 2016-05-23 00:27:26,659 Stage-1 map = 36%,  reduce = 0%, Cumulative CPU 37.1 sec

2016-05-23 00:27:32,839 Stage-1 map = 41%,  reduce = 0%, Cumulative CPU 41.74 sec

INFO  : 2016-05-23 00:27:32,839 Stage-1 map = 41%,  reduce = 0%, Cumulative CPU 41.74 sec

2016-05-23 00:27:39,003 Stage-1 map = 45%,  reduce = 0%, Cumulative CPU 46.32 sec

INFO  : 2016-05-23 00:27:39,003 Stage-1 map = 45%,  reduce = 0%, Cumulative CPU 46.32 sec

2016-05-23 00:27:45,173 Stage-1 map = 50%,  reduce = 0%, Cumulative CPU 50.93 sec

2016-05-23 00:27:50,316 Stage-1 map = 55%,  reduce = 0%, Cumulative CPU 55.55 sec

INFO  : 2016-05-23 00:27:45,173 Stage-1 map = 50%,  reduce = 0%, Cumulative CPU 50.93 sec

INFO  : 2016-05-23 00:27:50,316 Stage-1 map = 55%,  reduce = 0%, Cumulative CPU 55.55 sec

2016-05-23 00:27:56,482 Stage-1 map = 59%,  reduce = 0%, Cumulative CPU 60.25 sec

INFO  : 2016-05-23 00:27:56,482 Stage-1 map = 59%,  reduce = 0%, Cumulative CPU 60.25 sec

2016-05-23 00:28:02,642 Stage-1 map = 64%,  reduce = 0%, Cumulative CPU 64.86 sec

INFO  : 2016-05-23 00:28:02,642 Stage-1 map = 64%,  reduce = 0%, Cumulative CPU 64.86 sec

2016-05-23 00:28:08,814 Stage-1 map = 68%,  reduce = 0%, Cumulative CPU 69.41 sec

INFO  : 2016-05-23 00:28:08,814 Stage-1 map = 68%,  reduce = 0%, Cumulative CPU 69.41 sec

2016-05-23 00:28:14,977 Stage-1 map = 73%,  reduce = 0%, Cumulative CPU 74.06 sec

INFO  : 2016-05-23 00:28:14,977 Stage-1 map = 73%,  reduce = 0%, Cumulative CPU 74.06 sec

2016-05-23 00:28:21,134 Stage-1 map = 77%,  reduce = 0%, Cumulative CPU 78.72 sec

INFO  : 2016-05-23 00:28:21,134 Stage-1 map = 77%,  reduce = 0%, Cumulative CPU 78.72 sec

2016-05-23 00:28:27,282 Stage-1 map = 82%,  reduce = 0%, Cumulative CPU 83.32 sec

INFO  : 2016-05-23 00:28:27,282 Stage-1 map = 82%,  reduce = 0%, Cumulative CPU 83.32 sec

2016-05-23 00:28:33,437 Stage-1 map = 86%,  reduce = 0%, Cumulative CPU 87.9 sec

INFO  : 2016-05-23 00:28:33,437 Stage-1 map = 86%,  reduce = 0%, Cumulative CPU 87.9 sec

2016-05-23 00:28:38,579 Stage-1 map = 91%,  reduce = 0%, Cumulative CPU 92.52 sec

INFO  : 2016-05-23 00:28:38,579 Stage-1 map = 91%,  reduce = 0%, Cumulative CPU 92.52 sec

2016-05-23 00:28:44,759 Stage-1 map = 95%,  reduce = 0%, Cumulative CPU 97.35 sec

INFO  : 2016-05-23 00:28:44,759 Stage-1 map = 95%,  reduce = 0%, Cumulative CPU 97.35 sec

2016-05-23 00:28:49,915 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 99.6 sec

INFO  : 2016-05-23 00:28:49,915 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 99.6 sec

2016-05-23 00:28:54,043 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 101.4 sec

MapReduce Total cumulative CPU time: 1 minutes 41 seconds 400 msec

Ended Job = job_1463956731753_0005

MapReduce Jobs Launched:

Stage-Stage-1: Map: 22  Reduce: 1   Cumulative CPU: 101.4 sec   HDFS Read: 5318569 HDFS Write: 46 SUCCESS

Total MapReduce CPU Time Spent: 1 minutes 41 seconds 400 msec

OK

INFO  : 2016-05-23 00:28:54,043 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 101.4 sec

INFO  : MapReduce Total cumulative CPU time: 1 minutes 41 seconds 400 msec

INFO  : Ended Job = job_1463956731753_0005

INFO  : MapReduce Jobs Launched:

INFO  : Stage-Stage-1: Map: 22  Reduce: 1   Cumulative CPU: 101.4 sec   HDFS Read: 5318569 HDFS Write: 46 SUCCESS

INFO  : Total MapReduce CPU Time Spent: 1 minutes 41 seconds 400 msec

INFO  : Completed executing command(queryId=hduser_20160523002632_9f91d42a-ea46-4a66-a589-7d39c23b41dc); Time taken: 142.525 seconds

INFO  : OK

+-----+------------+---------------+-----------------------+--+

| c0  |     c1     |      c2       |          c3           |

+-----+------------+---------------+-----------------------+--+

| 1   | 100000000  | 5.00000005E7  | 2.8867513459481288E7  |

+-----+------------+---------------+-----------------------+--+

1 row selected (142.744 seconds)

 

OK Hive on map-reduce engine took 142 seconds compared to 58 seconds with Hive on Spark. So you can obviously gain pretty well by using Hive on Spark.

 

Please also note that I did not use any vendor's build for this purpose. I compiled Spark 1.3.1 myself.

 

HTH

 

 

Dr Mich Talebzadeh

 

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com/