kudu-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From a...@apache.org
Subject [1/4] kudu git commit: [docs] Improvements to multi-master migration doc
Date Tue, 06 Mar 2018 23:10:18 GMT
Repository: kudu
Updated Branches:
  refs/heads/master 9f93e97a6 -> c8724c615


[docs] Improvements to multi-master migration doc

- Add extra reminder to run as the Kudu user.

- Note that the copy_from_remote command requires authenticating
  to the remote service as the Kudu user.

- Note that the workflow can be used to migrate 2->3 masters by
  making straightforward adjustments to the procedure.

- Move steps for verifying the migration was successful to a new
  section so they are more noticeable.

Change-Id: I77ef796f8b35729871ef8ddf2b635989278c2ebc
Reviewed-on: http://gerrit.cloudera.org:8080/9466
Reviewed-by: Adar Dembo <adar@cloudera.com>
Tested-by: Will Berkeley <wdberkeley@gmail.com>


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/e7c7d4a1
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/e7c7d4a1
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/e7c7d4a1

Branch: refs/heads/master
Commit: e7c7d4a1203d4869713ec2d912dadb3f43c60bdb
Parents: 9f93e97
Author: Will Berkeley <wdberkeley@apache.org>
Authored: Thu Mar 1 11:31:37 2018 -0800
Committer: Will Berkeley <wdberkeley@gmail.com>
Committed: Tue Mar 6 22:58:15 2018 +0000

----------------------------------------------------------------------
 docs/administration.adoc | 84 +++++++++++++++++++++++--------------------
 1 file changed, 46 insertions(+), 38 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/kudu/blob/e7c7d4a1/docs/administration.adoc
----------------------------------------------------------------------
diff --git a/docs/administration.adoc b/docs/administration.adoc
index ab78aa1..6f64d7a 100644
--- a/docs/administration.adoc
+++ b/docs/administration.adoc
@@ -210,13 +210,14 @@ The frequency with which metrics are dumped to the diagnostics log is
configured
 For high availability and to avoid a single point of failure, Kudu clusters should be created
with
 multiple masters. Many Kudu clusters were created with just a single master, either for simplicity
 or because Kudu multi-master support was still experimental at the time. This workflow demonstrates
-how to migrate to a multi-master configuration.
+how to migrate to a multi-master configuration. It can also be used to migrate from two masters
to
+three, with straightforward modifications.
 
-WARNING: The workflow is unsafe for adding new masters to an existing multi-master configuration.
-Do not use it for that purpose.
+WARNING: The workflow is unsafe for adding new masters to an existing configuration that
already has
+three or more masters. Do not use it for that purpose.
 
-WARNING: All of the command line steps below should be executed as the Kudu UNIX user, typically
-`kudu`.
+WARNING: All of the command line steps below should be executed as the Kudu UNIX user. The
example
+commands assume the Kudu Unix user is `kudu`, which is typical.
 
 WARNING: The workflow presupposes at least basic familiarity with Kudu configuration management.
If
 using Cloudera Manager (CM), the workflow also presupposes familiarity with it.
@@ -230,11 +231,11 @@ using Cloudera Manager (CM), the workflow also presupposes familiarity
with it.
   configurations are recommended; they can tolerate one or two failures respectively.
 
 . Perform the following preparatory steps for the existing master:
-* Identify and record the directory where the master's data lives. If using Kudu system packages,
-  the default value is /var/lib/kudu/master, but it may be customized via the `fs_wal_dir`
and
-  `fs_data_dirs` configuration parameters. Please note if you've set `fs_data_dirs` to some
directories
-  other than the value of `fs_wal_dir`, it should be explicitly included in every command
below where
-  `fs_wal_dir` is also included. For more information on configuring these directories, see
the
+* Identify and record the directories where the master's write-ahead log (WAL) and data live.
If
+  using Kudu system packages, their default locations are /var/lib/kudu/master, but they
may be
+  customized via the `fs_wal_dir` and `fs_data_dirs` configuration parameters. The commands
below
+  assume that `fs_wal_dir` is /data/kudu/master/wal and `fs_data_dirs` is /data/kudu/master/data.
+  Your configuration may differ. For more information on configuring these directories, see
the
   link:configuration.html#directory_configuration[Kudu Configuration docs].
 * Identify and record the port the master is using for RPCs. The default port value is 7051,
but it
   may have been customized using the `rpc_bind_addresses` configuration parameter.
@@ -242,7 +243,7 @@ using Cloudera Manager (CM), the workflow also presupposes familiarity
with it.
 +
 [source,bash]
 ----
-$ kudu fs dump uuid --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>]
2>/dev/null
+$ sudo -u kudu kudu fs dump uuid --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>]
2>/dev/null
 ----
 master_data_dir:: existing master's previously recorded data directory
 +
@@ -250,7 +251,7 @@ master_data_dir:: existing master's previously recorded data directory
 Example::
 +
 ----
-$ kudu fs dump uuid --fs_wal_dir=/var/lib/kudu/master 2>/dev/null
+$ sudo -u kudu kudu fs dump uuid --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data
2>/dev/null
 4aab798a69e94fab8d77069edff28ce0
 ----
 +
@@ -297,8 +298,8 @@ the migration section for updating HMS.
 +
 [source,bash]
 ----
-$ kudu fs format --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>]
-$ kudu fs dump uuid --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>]
2>/dev/null
+$ sudo -u kudu kudu fs format --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>]
+$ sudo -u kudu kudu fs dump uuid --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>]
2>/dev/null
 ----
 +
 master_data_dir:: new master's previously recorded data directory
@@ -307,8 +308,8 @@ master_data_dir:: new master's previously recorded data directory
 Example::
 +
 ----
-$ kudu fs format --fs_wal_dir=/var/lib/kudu/master
-$ kudu fs dump uuid --fs_wal_dir=/var/lib/kudu/master 2>/dev/null
+$ sudo -u kudu kudu fs format --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data
+$ sudo -u kudu kudu fs dump uuid --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data
2>/dev/null
 f5624e05f40649b79a757629a69d061e
 ----
 
@@ -322,7 +323,7 @@ f5624e05f40649b79a757629a69d061e
 +
 [source,bash]
 ----
-$ kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>]
<tablet_id> <all_masters>
+$ sudo -u kudu kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=<master_wal_dir>
[--fs_data_dirs=<master_data_dirs>] <tablet_id> <all_masters>
 ----
 +
 master_data_dir:: existing master's previously recorded data directory
@@ -337,7 +338,7 @@ port::: master's previously recorded RPC port number
 Example::
 +
 ----
-$ kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=/var/lib/kudu/master 00000000000000000000000000000000
4aab798a69e94fab8d77069edff28ce0:master-1:7051 f5624e05f40649b79a757629a69d061e:master-2:7051
988d8ac6530f426cbe180be5ba52033d:master-3:7051
+$ sudo -u kudu kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=/data/kudu/master/wal
--fs_data_dirs=/data/kudu/master/data 00000000000000000000000000000000 4aab798a69e94fab8d77069edff28ce0:master-1:7051
f5624e05f40649b79a757629a69d061e:master-2:7051 988d8ac6530f426cbe180be5ba52033d:master-3:7051
 ----
 
 . Modify the value of the `master_addresses` configuration parameter for both existing master
and new masters.
@@ -348,11 +349,14 @@ port:: master's previously recorded RPC port number
 . Start the existing master.
 
 . Copy the master data to each new master with the following command, executed on each new
master
-  machine:
+  machine.
++
+WARNING: If your Kudu cluster is secure, in addition to running as the Kudu UNIX user, you
must
+  authenticate as the Kudu service user prior to running this command.
 +
 [source,bash]
 ----
-$ kudu local_replica copy_from_remote --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>]
<tablet_id> <existing_master>
+$ sudo -u kudu kudu local_replica copy_from_remote --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>]
<tablet_id> <existing_master>
 ----
 +
 master_data_dir:: new master's previously recorded data directory
@@ -366,7 +370,7 @@ port::: existing master's previously recorded RPC port number
 Example::
 +
 ----
-$ kudu local_replica copy_from_remote --fs_wal_dir=/var/lib/kudu/master 00000000000000000000000000000000
master-1:7051
+$ sudo -u kudu kudu local_replica copy_from_remote --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data
00000000000000000000000000000000 master-1:7051
 ----
 
 . Start all of the new masters.
@@ -401,9 +405,9 @@ INVALIDATE METADATA;
 ----
 +
 
+==== Verify the migration was successful
 
-Congratulations, the cluster has now been migrated to multiple masters! To verify that all
masters
-are working properly, consider performing the following sanity checks:
+To verify that all masters are working properly, perform the following sanity checks:
 
 * Using a browser, visit each master's web UI. Look at the /masters page. All of the masters
should
   be listed there with one master in the LEADER role and the others in the FOLLOWER role.
The
@@ -468,7 +472,7 @@ WARNING: All of the command line steps below should be executed as the
Kudu UNIX
 +
 [source,bash]
 ----
-$ kudu fs dump uuid --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>]
2>/dev/null
+$ sudo -u kudu kudu fs dump uuid --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>]
2>/dev/null
 ----
 master_data_dir:: live master's previously recorded data directory
 +
@@ -476,7 +480,7 @@ master_data_dir:: live master's previously recorded data directory
 Example::
 +
 ----
-$ kudu fs dump uuid --fs_wal_dir=/var/lib/kudu/master 2>/dev/null
+$ sudo -u kudu kudu fs dump uuid --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data
2>/dev/null
 80a82c4b8a9f4c819bab744927ad765c
 ----
 +
@@ -491,7 +495,7 @@ $ kudu fs dump uuid --fs_wal_dir=/var/lib/kudu/master 2>/dev/null
 +
 [source,bash]
 ----
-$ kudu local_replica cmeta print_replica_uuids --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>]
<tablet_id> 2>/dev/null
+$ sudo -u kudu kudu local_replica cmeta print_replica_uuids --fs_wal_dir=<master_wal_dir>
[--fs_data_dirs=<master_data_dirs>] <tablet_id> 2>/dev/null
 ----
 master_data_dir:: reference master's previously recorded data directory
 tablet_id:: must be the string `00000000000000000000000000000000`
@@ -500,7 +504,7 @@ tablet_id:: must be the string `00000000000000000000000000000000`
 Example::
 +
 ----
-$ kudu local_replica cmeta print_replica_uuids --fs_wal_dir=/var/lib/kudu/master 00000000000000000000000000000000
2>/dev/null
+$ sudo -u kudu kudu local_replica cmeta print_replica_uuids --fs_wal_dir=/data/kudu/master/wal
--fs_data_dirs=/data/kudu/master/data 00000000000000000000000000000000 2>/dev/null
 80a82c4b8a9f4c819bab744927ad765c 2a73eeee5d47413981d9a1c637cce170 1c3f3094256347528d02ec107466aef3
 ----
 +
@@ -514,7 +518,7 @@ $ kudu local_replica cmeta print_replica_uuids --fs_wal_dir=/var/lib/kudu/master
 +
 [source,bash]
 ----
-$ kudu fs format --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>]
--uuid=<uuid>
+$ sudo -u kudu kudu fs format --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>]
--uuid=<uuid>
 ----
 +
 master_data_dir:: replacement master's previously recorded data directory
@@ -524,14 +528,17 @@ uuid:: dead master's previously recorded UUID
 Example::
 +
 ----
-$ kudu fs format --fs_wal_dir=/var/lib/kudu/master --uuid=80a82c4b8a9f4c819bab744927ad765c
+$ sudo -u kudu kudu fs format --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data
--uuid=80a82c4b8a9f4c819bab744927ad765c
 ----
 +
 . Copy the master data to the replacement master with the following command:
 +
+WARNING: If your Kudu cluster is secure, in addition to running as the Kudu UNIX user, you
must
+  authenticate as the Kudu service user prior to running this command.
++
 [source,bash]
 ----
-$ kudu local_replica copy_from_remote --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>]
<tablet_id> <reference_master>
+$ sudo -u kudu kudu local_replica copy_from_remote --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>]
<tablet_id> <reference_master>
 ----
 +
 master_data_dir:: replacement master's previously recorded data directory
@@ -545,7 +552,7 @@ port::: reference master's previously recorded RPC port number
 Example::
 +
 ----
-$ kudu local_replica copy_from_remote --fs_wal_dir=/var/lib/kudu/master 00000000000000000000000000000000
master-2:7051
+$ sudo -u kudu kudu local_replica copy_from_remote --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data
00000000000000000000000000000000 master-2:7051
 ----
 +
 . If using CM, add the replacement Kudu master role now, but do not start it.
@@ -621,8 +628,9 @@ remove any unwanted masters.
 
 . Start all of the tablet servers.
 
-Congratulations, the masters have now been removed! To verify that all masters are working
properly,
-consider performing the following sanity checks:
+==== Verify the migration was successful
+
+To verify that all masters are working properly, perform the following sanity checks:
 
 * Using a browser, visit each master's web UI. Look at the /masters page. All of the masters
should
   be listed there with one master in the LEADER role and the others in the FOLLOWER role.
The
@@ -644,7 +652,7 @@ addresses to be specified:
 
 [source,bash]
 ----
-$ kudu cluster ksck master-01.example.com,master-02.example.com,master-03.example.com
+$ sudo -u kudu kudu cluster ksck master-01.example.com,master-02.example.com,master-03.example.com
 ----
 
 To see a full list of the options available with `ksck`, use the `--help` flag.
@@ -694,7 +702,7 @@ be done with the following command:
 
 [source,bash]
 ----
-$ kudu cluster ksck --checksum_scan --tables IntegrationTestBigLinkedList master-01.example.com,master-02.example.com,master-03.example.com
+$ sudo -u kudu kudu cluster ksck --checksum_scan --tables IntegrationTestBigLinkedList master-01.example.com,master-02.example.com,master-03.example.com
 ----
 
 [[change_dir_config]]
@@ -744,7 +752,7 @@ UNIX user, typically `kudu`.
 +
 [source,bash]
 ----
-$ kudu fs update_dirs --fs_wal_dir=/wals --fs_data_dirs=/data/1,/data/2,/data/3
+$ sudo -u kudu kudu fs update_dirs --fs_wal_dir=/wals --fs_data_dirs=/data/1,/data/2,/data/3
 ----
 +
 
@@ -864,7 +872,7 @@ diagnosing and fixing the problem is to examine the tablet's state using
ksck:
 
 [source,bash]
 ----
-$ kudu cluster ksck --tablets=e822cab6c0584bc0858219d1539a17e6 master-00,master-01,master-02
+$ sudo -u kudu kudu cluster ksck --tablets=e822cab6c0584bc0858219d1539a17e6 master-00,master-01,master-02
 Connected to the Master
 Fetched info from all 5 Tablet Servers
 Tablet e822cab6c0584bc0858219d1539a17e6 of table 'my_table' is unavailable: 2 replica(s)
not RUNNING
@@ -898,7 +906,7 @@ will be rewritten to include only the healthy replica.
 
 [source,bash]
 ----
-$ kudu remote_replica unsafe_change_config tserver-00:7150 <tablet-id> <tserver-00-uuid>
+$ sudo -u kudu kudu remote_replica unsafe_change_config tserver-00:7150 <tablet-id>
<tserver-00-uuid>
 ----
 
 where `<tablet-id>` is `e822cab6c0584bc0858219d1539a17e6` and


Mime
View raw message