hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@hortonworks.com>
Subject Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Date Tue, 16 Aug 2016 09:25:54 GMT
+1 binding

1. built and tested apache slider (incubating) against the Hadoop 2.7.3 artifacts

2. did a build & test of Apache Spark master branch iwth 2.7.3 JARs, 

For that I had to tweak spark's build to support the staging repo; hopefully that will get
into Spark 


3. did a test run of my WiP SPARK-7481 spark-cloud module; after fixing a couple of things
on the test setup side related to HADOOP-13058, 

	mvn test --pl cloud -Pyarn,hadoop-2.7,snapshots-and-staging -Dhadoop.version=2.7.3 -Dcloud.test.configuration.file=../conf/cloud-tests.xml

all was well —albeit measurably slower than Hadoop 2.8. That's proof that the 2.8 version
of s3a really does deliver measurable speedup for those tests (currently just file input/seek;
more to come). I had originally thought things were broken as s3 init was failing -but that's
because the s3 bucket was in frankfurt, and the AWS library used can't talk to that endpoint
(v4 auth protocol, see).

4. did a full spark distribution build of that SPARK-7481 branch

	dev/make-distribution.sh  -Pyarn,hadoop-2.7,snapshots-and-staging -Dhadoop.version=2.7.3

ran command line test to do read of s3a data:

	bin/spark-submit --class org.apache.spark.cloud.s3.examples.S3LineCount \
                                              --conf spark.hadoop.fs.s3a.access.key=$AWS_KEY
                                              --conf spark.hadoop.fs.s3a.secret.key=$AWS_SECRET

5. Pulled out the microsoft Azure JAR azure-storage-2.0.0.jar and repeated step 4

-this showed that the 2.7.x branch does handle the failure to load a filesystem due to dependency
or other classloading problems —this was proving a big problem in adding the aws & azure
stuff to the spark build, as it'd stop spark from starting up if the dependencies were absent.

I've not done any of the .tar.gz diligence; I've just looked at the staged JARs and how they
worked with downstream apps —that being a key way that Hadoop artifacts are adopted.

> On 12 Aug 2016, at 17:45, Vinod Kumar Vavilapalli <vinodkv@apache.org> wrote:
> Hi all,
> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
> As discussed before, this is the next maintenance release to follow up 2.7.2.
> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/
> The RC tag in git is: release-2.7.3-RC1
> The maven artifacts are available via repository.apache.org <http://repository.apache.org/>
at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html.
I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>
for your quick perusal.
> As you may have noted,
> - few issues with RC0 forced a RC1 [1]
> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 2.7.3
(along with every other Hadoop release) to slip by quite a bit. This release's related discussion
thread is linked below: [2].
> Please try the release and vote; the vote will run for the usual 5 days.
> Thanks,
> Vinod
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106
> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html

To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
View raw message