spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Lyubimov <>
Subject Re: Why Scala?
Date Thu, 29 May 2014 22:22:43 GMT
There were few known concerns about Scala, and some still are, but having
been doing Scala professionally over two years now, i learned to master and
appreciate the advanatages.

Major concern IMO is Scala in a less-than-scrupulous corporate environment.

First, Scala requires significantly more discipline in commenting and style
to still stay painlessly readable, than java. People with less than stellar
code hygiene can easily turn a project into an unmaintainable mess.

Second, from corporate management prospective, it is (still?) much harder
to staff with Scala coders as opposed to Java ones.

All these things are a headache for corporate bosses, but for public and
academic projects with thorough peer review and increased desire for
contributors to look clean in public it works out quite well, and strong
sides really shine.

Spark specifically builds around FP patterns -- such as monads and functors
-- which were absent in java prior to  8 (i am not sure that they are as
well worked out in java 8 collections even now, as opposed to Scala
collections). So java 8 simply comes a little late to the show in that

Also FP is not the only thing that is used by Spark. Spark also uses stuff
like implicits, akka/agent framework for IPC. Let's not forget that FP is
albeit important but only one out of many  stories in Scala in the grand
scale of things.

On Thu, May 29, 2014 at 1:55 PM, Nick Chammas <>wrote:

> I recently discovered Hacker News and started reading through older posts
> about Scala <!/story/forever/0/scala>. It
> looks like the language is fairly controversial on there, and it got me
> thinking.
> Scala appears to be the preferred language to work with in Spark, and
> Spark itself is written in Scala, right?
> I know that often times a successful project evolves gradually out of
> something small, and that the choice of programming language may not always
> have been made consciously at the outset.
> But pretending that it was, why is Scala the preferred language of Spark?
> Nick
> ------------------------------
> View this message in context: Why Scala?<>
> Sent from the Apache Spark User List mailing list archive<>at

View raw message