spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jeff saremi <jeffsar...@hotmail.com>
Subject Re: Is there "EXCEPT ALL" in Spark SQL?
Date Thu, 06 Jul 2017 20:19:23 GMT
EXCEPT is not the same as EXCEPT ALL

Had they implemented EXCEPT ALL in SparkSQL one could have easily obtained EXCEPT by adding
a disctint() to the results


________________________________
From: hareesh makam <makamhareesh@gmail.com>
Sent: Thursday, July 6, 2017 12:48:18 PM
To: jeff saremi
Cc: user@spark.apache.org
Subject: Re: Is there "EXCEPT ALL" in Spark SQL?

There is Except in DataFrame API.

df1.except(df2)

Same can be used in SQL as well.


public DataFrame<https://spark.apache.org/docs/1.6.1/api/java/org/apache/spark/sql/DataFrame.html>
except(DataFrame<https://spark.apache.org/docs/1.6.1/api/java/org/apache/spark/sql/DataFrame.html>
other)

Returns a new DataFrame<https://spark.apache.org/docs/1.6.1/api/java/org/apache/spark/sql/DataFrame.html>
containing rows in this frame but not in another frame. This is equivalent to EXCEPT in SQL.


-Hareesh


On 6 July 2017 at 12:22, jeff saremi <jeffsaremi@hotmail.com<mailto:jeffsaremi@hotmail.com>>
wrote:

I tried this query in 1.6 and it failed:


SELECT * FROM Table1 EXCEPT ALL SELECT * FROM Table2



Exception in thread "main" java.lang.RuntimeException: [1.32] failure: ``('' expected but
`all' found


thanks

Jeff


Mime
View raw message