spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexy Khrabrov <al...@scalable.pro>
Subject Upcoming Scala DevFlow Training By the Bay, November 6-7
Date Mon, 27 Oct 2014 20:42:13 GMT
Spark is written in Scala.  As a strongly typed functional programming
language, Scala makes Spark possible.  The API of Spark mirrors Scala
collections API, driving adoption through beautifully intuitive data
transformations which work both locally and at scale.  And where's
more inspiration where it came from -- we've seen how Scala and Spark
influence each other at the Scala By the Bay developer conference and
its own professional training courses for Scala and Spark.

When you learn Scala for Spark, you quickly find out that there’s much
more to it than just the language.  The REPL is great and you can
start playing right away, but when the time comes to run in
production, you need to make sure that the code is correct, deployed
properly, scalable as needed.  But most importantly for the
developers, the path from experimenting in the REPL to writing code in
an IDE (or vim) to testing to deployment should be as easy but
rigorous as coding in Scala itself.  It is only natural that Scala
developers, running some of the biggest APIs and Big Data setups out
there, would think carefully about the right ways to build a
Scala-centric development flow.

Scala runs on the JVM, which comes with its own ecosystem, toolsets,
deployment, monitoring, and so on, which should be adapted to Scala
ways selectively.  Scala espouses the culture of correctness through
strong typing and rigorous test-driven development.  In addition to
local tools,  there are now many cloud-based systemsfor rapid
prototyping and deployment, such as Heroku.  If you’re building an
API, for example, or another kind of service, you can spin VMs or
Heroku versions to achieve a much higher fidelity in integration
testing with easier setup and teardown.

Git has also fundamentally changed the way we go about development in
a team.  It is an established best practice by now to develop on
branches, code-review, and use TDD automation to verify that branches
can be merged properly.  There are tools to make sure a pull request
builds properly before merging.  When it comes to deployment, a
process must be set up which will package the code and run it through
staging to production with an ability to roll back if needed.
Overall, a pragmatic programmer needs a tool chain which amplifies
his/her language in a real-life flow of repeatable cycle of
development, testing, and deployment.

The Scala DevFlow workshop is the first professional training
structured specifically around the best practices for developers who
work in Scala and use agile practices, leading to continuous
deployment.

http://bythebay.ticketleap.com/scala-devops/

It is taught by Alexy Khrabrov and James Earl Douglas, the first two
coding cofounders of Versal and meetup organizers, who drove the
all-Scala backend and its development flow and devops processes.  The
experience led to new ways to develop APIs using TDD in Scala, where
API tests are one-liners building the state of the API, with an
example at https://github.com/Versal/fireotter.  The devflow process
was presented at ScalaDays 2013 and Scala By the Bay 2014 — links to
the videos are at the workshop site.  The workshop will be a
collaborative two-day course where developers will program together to
master the flow for themselves and their teams.  If you’re running a
startup on Scala, this is the most valuable focus, as we’ve learned
through several years of systems building with it.

We’ve structured the training in two days which can be attended
together or separately.  The first day covers the fundamental
development flow, which will quickly align the team.  It uses hosted
tools.  The second day goes into continuous integration/deployment,
release management, and in-house setup.  The early bird pricing runs
through the end of the week, and we're offering a limited number of
$100 discounts, please use code DEVFLOW100 (while available).  Email
any questions to alexy@bythebay.io.

Just like deeply thought-through functional programming practices make
Scala great, properly engineered development flow makes a huge
difference.  Both sets of knowledge are community-centric -- it's what
makes Scala great.  Take advantage of this experience, and build more
& better companies with it, faster!  Hope to code & deploy with you
soon.

Cheers,
A+

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message