ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Fernandez" <afernan...@hortonworks.com>
Subject Re: Review Request 39111: [PREVIEW] Stop-and-Start Upgrade: Handle Core Services
Date Thu, 08 Oct 2015 00:06:40 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39111/#review101836
-----------------------------------------------------------



ambari-common/src/main/python/resource_management/libraries/script/script.py (line 399)
<https://reviews.apache.org/r/39111/#comment159348>

    Started passing upgrade_type to all of these functions.



ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
(line 937)
<https://reviews.apache.org/r/39111/#comment159349>

    A stage object now has parameters that can be applied.



ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java (line 398)
<https://reviews.apache.org/r/39111/#comment159350>

    This was the trick to generate 2 tasks for NameNodes, each with different params.



ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java
(line 48)
<https://reviews.apache.org/r/39111/#comment159351>

    I made params default to null by overloading the constructor.



ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py
(line 37)
<https://reviews.apache.org/r/39111/#comment159352>

    I had to write this function because "hdfs" command was coming from /usr/bin/hdfs (HDP
2.1) even after upgrading to HDP 2.3.
    
    I may need to remove HDP 2.1.



ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py
(line 87)
<https://reviews.apache.org/r/39111/#comment159353>

    Please review this, it's more complex now.



ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py
(line 185)
<https://reviews.apache.org/r/39111/#comment159354>

    I have to figure out where to back these up to, or create a manual task to let the user
do it on their own.



ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py
(line 57)
<https://reviews.apache.org/r/39111/#comment159355>

    Here's the command used to distinguish the active from the standby during Express Upgrade
start.



ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
(line 100)
<https://reviews.apache.org/r/39111/#comment159356>

    I'll likely have to set a proper default.


- Alejandro Fernandez


On Oct. 8, 2015, midnight, Alejandro Fernandez wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39111/
> -----------------------------------------------------------
> 
> (Updated Oct. 8, 2015, midnight)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jonathan Hurley, and
Nate Cole.
> 
> 
> Bugs: AMBARI-12701
>     https://issues.apache.org/jira/browse/AMBARI-12701
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Stop-and-Start Upgrade of the Stack to handle core services: HDFS, YARN, MR, ZK
> 
> * Renamed the functions pre_rolling_upgrade and post_rolling_upgrade to pre_upgrade_restart
and post_upgrade_restart that accept the upgrade_type (rolling|nonrolling) as a param.
> 
> * When NameNode HA is present, the command used to start NameNode is different between
the active and the standby, so changed MasterHostResolver.java to pick one of the NameNodes
to become the active and the other the standby, and pass it in a parameter called "desired_namenode_role"
> 
> * Wrote a function to take a backup of HDFS
> 
> * Changed the "hdfs" command to actually search which one to use depending on the stack
and version to account for Express Upgrade from HDP 2.1 -> 2.3
> 
> * Tested Express Upgrade with and without NameNode HA
> 
> 
> Diffs
> -----
> 
>   ambari-common/src/main/python/resource_management/libraries/script/script.py 26fd158

>   ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
a2f642c 
>   ambari-server/src/main/java/org/apache/ambari/server/stack/MasterHostResolver.java
ef75d38 
>   ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java ed9b3a7

>   ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java
648e7a4 
>   ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ColocatedGrouping.java
a8e9c43 
>   ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ConfigureTask.java
1164335 
>   ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java
4116101 
>   ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ServiceCheckGrouping.java
68fec03 
>   ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java
92df3b5 
>   ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapperBuilder.java
6c0f3c7 
>   ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/TaskWrapper.java
f7cc930 
>   ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/TaskWrapperBuilder.java
a5813e3 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py
fa68435 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode_upgrade.py
2e5ac19 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_client.py
dd0dca4 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py
b11d7ea 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/journalnode.py
46c7272 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/journalnode_upgrade.py
e2ebbcb 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py
07be496 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode_ha_state.py
e8c142c 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode_upgrade.py
fb39878 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/nfsgateway.py
be6f0d5 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py
5f6b8fe 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/setup_ranger_hdfs.py
f5df86f 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/snamenode.py
108ef01 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py
7bc0b6a 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/zkfc_slave.py
14de094 
>   ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py
f3b3d11 
>   ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/historyserver.py
16e34d4 
>   ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/mapreduce2_client.py
5263d9f 
>   ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/nodemanager.py
fd25651 
>   ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
f3914f3 
>   ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py
e67f1ce 
>   ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn_client.py
e58ea3c 
>   ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py
973fa0f 
>   ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_client.py
02c1006 
>   ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py
b7fb578 
>   ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_service.py
685eb6d 
> 
> Diff: https://reviews.apache.org/r/39111/diff/
> 
> 
> Testing
> -------
> 
> * Tested Express Upgrade with and without NameNode HA
> 
> There are still a handful of bugs.
> * YARN history server start fails because NameNode is still in safemode and waiting for
block reports.
> * YARN Service Check fails due to some bad properties in the config pack from 2.1->2.3
> * May need to remove HDP 2.1
> * Taking HDFS backup and report to /tmp folder
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message