Just to be clear, I'm referring to having Spark reading from a file, not from a Hive table.  And it will have tungsten engine off heap serialization after 2.1, so if it was a test with like 1.63 it won't be as helpful.  


On Mon, Aug 28, 2017 at 10:50 AM, Michael Artz <michaeleartz@gmail.com> wrote:
Hi,
  There isn't any good source to answer the question if Hive as an SQL-On-Hadoop engine just as fast as Spark SQL now? I just want to know if there has been a comparison done lately for HiveQL vs Spark SQL on Spark versions 2.1 or later.  I have a large ETL process, with many table joins and some string manipulation. I don't think anyone has done this kind of testing in a while.  With Hive LLAP being so performant, I am trying to make the case for using Spark and some of the architects are light on experience so they are scared of Scala. 

Thanks