spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Holden Karau <hol...@pigscanfly.ca>
Subject Re: Why Spark generates Java code and not Scala?
Date Sat, 09 Nov 2019 17:52:01 GMT
Switching this from user to dev

On Sat, Nov 9, 2019 at 9:47 AM Bartosz Konieczny <bartkonieczny@gmail.com>
wrote:

> Hi there,
>
> Few days ago I got an intriguing but hard to answer question:
> "Why Spark generates Java code and not Scala code?"
> (https://github.com/bartosz25/spark-scala-playground/issues/18)
>
> Since I'm not sure about the exact answer, I'd like to ask you to confirm
> or not my thinking. I was looking for the reasons in the JIRA and the
> research paper "Spark SQL: Relational Data Processing in Spark" (
> http://people.csail.mit.edu/matei/papers/2015/sigmod_spark_sql.pdf) but
> found nothing explaining why Java over Scala. The single task I found was
> about why Scala and not Java but concerning data types (
> https://issues.apache.org/jira/browse/SPARK-5193) That's why I'm writing
> here.
>
> My guesses about choosing Java code are:
> - Java runtime compiler libs are more mature and prod-ready than the
> Scala's - or at least, they were at the implementation time
> - Scala compiler tends to be slower than the Java's
> https://stackoverflow.com/questions/3490383/java-compile-speed-vs-scala-compile-speed
>
>From the discussions when I was doing some code gen (in MLlib not SQL) I
think this is the primary reason why.

>
> <https://stackoverflow.com/questions/3490383/java-compile-speed-vs-scala-compile-speed>
> - Scala compiler seems to be more complex, so debugging & maintaining it
> would be harder
>
this was also given as a secondary reason

> - it was easier to represent a pure Java OO design than mixed FP/OO in
> Scala
>
no one brought up this point. Maybe it was a consideration and it just
wasn’t raised.

> ?
>
> Thank you for your help.
>
>
> --
> Bartosz Konieczny
> data engineer
> https://www.waitingforcode.com
> https://github.com/bartosz25/
> https://twitter.com/waitingforcode
>
> --
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Mime
View raw message