spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Devl Devel <devl.developm...@gmail.com>
Subject Some praise and comments on Spark
Date Wed, 25 Feb 2015 22:13:09 GMT
Hi Spark Developers,

First, apologies if this doesn't belong on this list but the
comments/praise are relevant to all developers. This is just a small note
about what we really like about Spark, I/we don't mean to start a whole
long discussion thread in this forum but just share our positive
experiences with Spark thus far.

To start, as you can tell, we think that the Spark project is amazing and
we love it! Having put in nearly half a decade worth of sweat and tears
into production Hadoop, MapReduce clusters and application development it's
so refreshing to see something arguably simpler and more elegant to
supersede it.

These are the things we love about Spark and hope these principles continue:

-the one command build; make-distribution.sh. Simple, clean  and ideal for
deployment and devops and rebuilding on different environments and nodes.
-not having too much runtime and deploy config; as admins and developers we
are sick of setting props like io.sort and mapred.job.shuffle.merge.percent
and dfs file locations and temp directories and so on and on again and
again every time we deploy a job, new cluster, environment or even change
company.
-a fully built-in stack, one global project for SQL, dataframes, MLlib etc,
so there is no need to add on projects to it on as per Hive, Hue, Hbase
etc. This helps life and keeps everything in one place.
-single (global) user based operation - no creation of a hdfs mapred unix
user, makes life much simpler
-single quick-start daemons; master and slaves. Not having to worry about
JT, NN, DN , TT, RM, Hbase master ... and doing netstat and jps on hundreds
of clusters makes life much easier.
-proper code versioning, feature releases and release management.
- good & well organised documentation with good examples.

In addition to the comments above this is where we hope Spark never ends
up:

-tonnes of configuration properties and "go faster" type flags. For example
Hadoop and Hbase users will know that there are a whole catalogue of
properties for regions, caches, network properties, block sizes, etc etc.
Please don't end up here for example:
https://hadoop.apache.org/docs/r1.0.4/mapred-default.html, it is painful
having to configure all of this and then create a set of properties for
each environment and then tie this into CI and deployment tools.
-no more daemons and processes to have to monitor and manipulate and
restart and crash.
-a project that penalises developers (that will ultimately help promote
Spark to their managers and budget holders) with expensive training,
certification, books and accreditation. Ideally this open source should be
free, free training= more users = more commercial uptake.

Anyway, those are our thoughts for what they are worth, keep up the good
work, we just had to mention it. Again sorry if this is not the right place
or if there is another forum for this stuff.

Cheers

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message