spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From MrPowers <>
Subject [Spark SQL]: SQL, Python, Scala and R API Consistency
Date Thu, 28 Jan 2021 20:23:08 GMT
Thank you all for your amazing work on this project.  Spark has a great
public interface and the source code is clean.  The core team has done a
great job building and maintaining this project.  My emails / GitHub
comments focus on the 1% that we might be able to improve.

Pull requests / suggestions for improvements can come across as negative,
but I'm nothing but happy & positive about this project.  The source code is
delightful to read and the internal abstractions are beautiful.

*API consistency*

The SQL, Scala, and Python APIs are generally consistent.  They all have a
reverse function for example.

Some of the new PRs have arguments against consistent rollout of functions
across the APIs.  This seems like a break in the traditional Spark
development process when functions were implemented in all APIs (except for
functions that only make sense for certain APIs like createDataset and

The default has shifted from consistent application of function across APIs
to "case by case determination".


* The regexp_extract_all function was recently added to the SQL API.  It was
then added to the Scala API,  but then removed from the Scala API
<>  .

* There is an ongoing discussion on  if CalendarType will be added to the
Python API <>  

*Arguments against adding functions like regexp_extract_all to the Scala

* Some of these functions are SQL specific and don't make sense for the
other languages

* Scala users can access the SQL functions via expr

*Argument rebuttal*

I don't understand the "some of the functions are SQL specific argument". 
regexp_extract_all fills a gap in the API.  Users have been forced to use
UDF workarounds for this in the past.  Users from all APIs need this

Using expr isn't developer friendly.  Scala / Python users don't want to
manipulate SQL strings.  Nesting functions in SQL strings is complicated. 
The quoting and escaping is all different.  Figuring out how to invoke
regexp_replace(col("word1"), "//", "\\,") via expr would be a real pain -
would need to figure out SQL quoting, SQL escaping, and how to access column
names instead of a column object.

Any of the org.apache.spark.sql.functions can be invoked via expr.  The core
reason the Scala/Python APIs exist is so that developers don't need to
manipulate strings for expr.

regexp_extract_all should be added to the Scala API for the same reasons
that regexp_extract was added to the Scala API.  

*Next steps*

* I'd like to better understand why we've broken from the traditional Spark
development process of "consistently implementing functions across all APIs"
to "selectively implementing functions in certain APIs"

* Hopefully shift the burden of proof to those in favor of inconsistent
application.  Consistent application should be the default.  

Thank you all for your excellent work on this project.

- Matthew Powers (GitHub: MrPowers)

Sent from:

To unsubscribe e-mail:

View raw message