hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Payne <erichadoo...@yahoo.com.INVALID>
Subject Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0
Date Thu, 29 Nov 2018 22:44:46 GMT
The problem is not with preemption. The yarn-site.xml that I use for my pseudo-cluster includes
a second xml:
xi:include href=".../yarn-scheduler.xml"

The property for yarn.resourcemanager.scheduler.monitor.enable = true is in this yarn-scheduler.xml.

This value IS READ when then RM starts.

However, when the refreshQueues command is run, this value IS NOT READ.

So, it looks like xml include files are not read on refresh. This will affect any property.
I just happened to notice it on the preemption properties.

I would like input from all of you to determine if this is a blocker for release. I'm on the
fence.

Thanks,
-Eric






On Wednesday, November 28, 2018, 4:58:50 PM CST, Eric Payne <erichadoop-1@yahoo.com.INVALID>
wrote: 





Sunil,

So, the basic symptoms are that if preemption is enabled on any queue, the preemption is disabled
after a 'yarn rm -refreshQueues'. In addition, all of the preemption-specific properties are
set back to the default values.

This was introduced in branch-3.1, so it is NOT new behavior for release 3.2.0. I am still
tracking down the cause. I will open a JIRA once I do further investigation if there is not
one already.

This will be a problem for installations which use preemption and which use the refreshQueues
feature.

Thanks,
-Eric


On Wednesday, November 28, 2018, 11:47:06 AM CST, Eric Payne <eric.payne1000@yahoo.com>
wrote: 





Sunil, thanks for all of the hard work on this release.

I have discovered that queue refresh doesn't work in some cases. For example, when I change
yarn.scheduler.capacity.root.default.disable_preemption, it doesn't take effect unless I restart
the RM.

I am still investigating, but I thought I should bring this up asap.

Thanks,
-Eric




On Friday, November 23, 2018, 6:07:04 AM CST, Sunil G <sunilg@apache.org> wrote: 





Hi folks,



Thanks to all contributors who helped in this release [1]. I have created

first release candidate (RC0) for Apache Hadoop 3.2.0.


Artifacts for this RC are available here:

http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/



RC tag in git is release-3.2.0-RC0.



The maven artifacts are available via repository.apache.org at

https://repository.apache.org/content/repositories/orgapachehadoop-1174/


This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST.



3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
additions

are the highlights of this release.

1. Node Attributes Support in YARN

2. Hadoop Submarine project for running Deep Learning workloads on YARN

3. Support service upgrade via YARN Service API and CLI

4. HDFS Storage Policy Satisfier

5. Support Windows Azure Storage - Blob file system in Hadoop

6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a

7. Improvements in Router-based HDFS federation



Thanks to Wangda, Vinod, Marton for helping me in preparing the release.

I have done few testing with my pseudo cluster. My +1 to start.



Regards,

Sunil



[1]

https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E

[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
ORDER BY fixVersion ASC

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org


Mime
View raw message