helix-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jiajunw...@apache.org
Subject [helix] branch wagedRebalancer updated (6aefab7 -> dedfdfe)
Date Tue, 04 Feb 2020 19:05:44 GMT
This is an automated email from the ASF dual-hosted git repository.

jiajunwang pushed a change to branch wagedRebalancer
in repository https://gitbox.apache.org/repos/asf/helix.git.


 discard 6aefab7  Reset the WAGED rebalancer once the controller newly acquires leadership.
(#690)
 discard d31066d  Add WAGED rebalancer reset method to clean up cached status. (#696)
 discard 21519d0  Add resource partition weight gauge (#686)
 discard fdb6605  Add instance capacity gauge (#557)
 discard 70a99d1  Fix the Helix rest tests by cleaning up the environment before testing.
(#679)
 discard 8b584f2  Reorgnize the test case so the new WAGED expand cluster tests are not skipped.
(#670)
 discard 247bf01  Asynchronously calculating the Baseline (#632)
 discard 6aaeeb0  Refine methods name and comments. (#664)
 discard 7d0a833  Refine the WAGED rebalancer to minimize the partial rebalance workload.
(#639)
 discard 8541304  Improve the algorithm so it prioritizes the assignment to the idle nodes
when the constraint evaluation results are the same (#651)
 discard 4e43125  Decouple the event type and the scheduled rebalance cache refresh option.
(#638)
 discard 977e145  Add new WAGED rebalancer config item "GLOBAL_REBALANCE_ASYNC_MODE". (#637)
 discard 5bd77f4  Simply and enhance the RebalanceLatencyGauge so it can be used in multi-threads.
(#636)
 discard 8530fa6  Fix a potential issue in the ResourceChangeSnapshot. (#635)
 discard f82a59e  Add REST API endpoints for WAGED Rebalancer (#611)
 discard 5da3a19  Fix the unstable test TestZeroReplicaAvoidance. (#603)
 discard dcd333c  Improve the WAGED rebalancer performance. (#586)
 discard bcec1f5  Change calculation for baseline divergence. (#598)
 discard 1c5476f  Add Java API for adding and validating resources for WAGED rebalancer (#570)
 discard 26ae8f9  Change ClusterConfig.setDefaultCapacityMap to be private. (#590)
 discard 857d3f8  Introduce Dry-run Waged Rebalancer for the verifiers and tests. (#573)
 discard bfc8c5d  Filter resource map with ideal states for instance capacity metrics. (#574)
 discard d822a9d  Avoid redundant writes in AssignmentMetadataStore (#564)
 discard a74b419  Fix formula incorrection in the comment for measuring baseline divergence.
(#559)
 discard c2c7f3f  Add max capacity usage metric for instance monitor. (#548)
 discard 1c6d285  Use a deep copy of the new best possible assignment for measuring baseline
divergence. (#542)
 discard fd82290  Refine ivy file config. The org were not configured correctly. (#551)
 discard 0c4d19a  Add the new Rebalancer monitor domain to the active domain list. (#550)
 discard 4277edc  Fix a missing parameter when the WAGED rebalancer init the change detector.
(#547)
 discard df495c5  Modify the ivy file to add the new math3 lib dependency. (#546)
 discard da11516  Implement increment() method in CountMetric class. (#537)
 discard 4f0047c  Adjust the replica rebalance calculating ordering to avoid static order.
(#535)
 discard ed5b041  Minor fix for the constraints related tests. (#545)
 discard 9e27b4c  Refactor soft constraints to simply the algorithm and fix potential issues.
(#520)
 discard 49c9281  Change change detector to a regular field in the WAGED rebalancer instead
of static threadlocal. (#543)
 discard 2e6c47e  Make WagedRebalancer static by creating a ThreadLocal (#540)
 discard 0727dff  Refine the rebalance scope calculating logic in the WAGED rebalancer. (#519)
 discard 029b049  Implement monitoring mbeans for the WAGED rebalancer. (#525)
 discard 7894776  Make log clearer after finishing calculateAssignment. (#531)
 discard bf0d0a7  The WAGED rebalancer returns the previously calculated assignment on calculation
failure (#514)
 discard 5fc9fed  Increase parallelism for ZkBucketDataAccessor (#506)
 discard 90ed9ae  More strict partition weight validation while creating the cluster model.
(#511)
 discard 336af27  Fixing rebalance cache issue and stablize the tests. (#510)
 discard 856813f  Add latency metric components for WAGED rebalancer (#490)
 discard 2330675  Load soft constraint weight from resources/properties file (#492)
 discard 5c6a7e8  Adjust the topology processing logic for instance to ensure backward compatibility.
 discard eb1133f  Enable maintenance mode for the WAGED rebalancer.
 discard 55ce7e2  Add delayed rebalance and user-defined preference list features to the WAGED
rebalancer. (#456)
 discard 0f7e3cc  Separate AssignableNode properties by Immutable and Mutable (#485)
 discard 3c9c084  Integrate the WAGED rebalancer with all the related components. (#466)
 discard f3cc6a6  Add the remaining implementation of ConstraintBasedAlgorithmFactory (#478)
 discard 3a92212  PartitionMovementSoftConstraint Implementation (#474)
 discard cc6d73f  Adjust the expected replica count according to fault zone count. (#476)
 discard bbc00dd  Improve ResourceTopStateAntiAffinityConstraint (#475)
 discard b70c282  Add soft constraint: ResourcePartitionAntiAffinityConstraint (#464)
 discard ce36e9d  Implement MaxCapacityUsageInstanceConstraint soft constraint (#463)
 discard 0120d0d  Add soft constraint: ResourcetopStateAntiAffinityConstraint (#465)
 discard 9df51c0  Implement one of the soft constraints (#450)
 discard bdb5d4d  Fix TestWagedRebalancer and add constructor in AssignmentMetadataStore
 discard 5b89fe2  Implement AssignmentMetadataStore (#453)
 discard c27371a  HardConstraints Implementation and unit tests (#433)
 discard 5170c10  Implement the WAGED rebalancer with the limited functionality. (#443)
 discard 71df138  Validate the instance capacity/partition weight configuration while constructing
the assignable instances (#451)
 discard 7707484  Implement the basic constraint based algorithm (#381)
 discard 2faab3d  Add BucketDataAccessor for large writes
 discard 8f24b2e  Record the replica objects in the AssignableNode in addition to the partition
name (#440)
 discard 904896d  Add special treatment for ClusterConfig
 discard d65dfc7  Modify the expected change type from CONFIG to CLUSTER_CONFIG in the WAGED
rebalancer. (#438)
 discard 280ae6a  Resubmit the change: Refine the WAGED rebalancer related interfaces for
integration (#431)
 discard 5376fda  Revert "Refine the WAGED rebalancer related interfaces for integration (#431)"
(#437)
 discard 00354a9  Refine the WAGED rebalancer related interfaces for integration (#431)
 discard 7c77b2f  Redefine the hard/soft constraints (#422)
 discard 90dd30a  Add ChangeDetector interface and ResourceChangeDetector implementation (#388)
 discard b0e25ba  Add cluster level default instance config. (#413)
 discard eacc9b8  Implement Cluster Model Provider. (#392)
 discard b212359  Convert all the internal assignment state objects to be ResourceAssignment.
(#399)
 discard db58943  Change the rebalancer assignment record to be ResourceAssignment instead
of IdealState. (#398)
 discard 57eae90  Implement the WAGED rebalancer cluster model (#362)
 discard 8b7b809  Adding the configuration items of the WAGED rebalancer. (#348)
 discard 1d0c9b0  Define the WAGED rebalancer interfaces.
     add 48cabc5  Bump jackson-databind from 2.9.5 to 2.9.10.1 in /helix-rest (#597)
     add db2e373  Integration test for controller connect and disconnect (#681)
     add ecf6bb8  Add workflow garbage collector  (#705)
     new a1f6b7e  Define the WAGED rebalancer interfaces.
     new a68ecf8  Adding the configuration items of the WAGED rebalancer. (#348)
     new 2cda87d  Implement the WAGED rebalancer cluster model (#362)
     new 3d7f5c5  Change the rebalancer assignment record to be ResourceAssignment instead
of IdealState. (#398)
     new 7045bde  Convert all the internal assignment state objects to be ResourceAssignment.
(#399)
     new 841b90d  Implement Cluster Model Provider. (#392)
     new d17bfaf  Add cluster level default instance config. (#413)
     new 9f7cf51  Add ChangeDetector interface and ResourceChangeDetector implementation (#388)
     new ca67efc  Redefine the hard/soft constraints (#422)
     new b40d16c  Refine the WAGED rebalancer related interfaces for integration (#431)
     new 7f4bc92  Revert "Refine the WAGED rebalancer related interfaces for integration (#431)"
(#437)
     new 290a0bb  Resubmit the change: Refine the WAGED rebalancer related interfaces for
integration (#431)
     new 740d6ff  Modify the expected change type from CONFIG to CLUSTER_CONFIG in the WAGED
rebalancer. (#438)
     new 6e78f16  Add special treatment for ClusterConfig
     new e3c7376  Record the replica objects in the AssignableNode in addition to the partition
name (#440)
     new 2b33c83  Add BucketDataAccessor for large writes
     new 27cf45a  Implement the basic constraint based algorithm (#381)
     new a5072b5  Validate the instance capacity/partition weight configuration while constructing
the assignable instances (#451)
     new 40adfc0  Implement the WAGED rebalancer with the limited functionality. (#443)
     new 85714a2  HardConstraints Implementation and unit tests (#433)
     new e938014  Implement AssignmentMetadataStore (#453)
     new b9626ff  Fix TestWagedRebalancer and add constructor in AssignmentMetadataStore
     new 52575d2  Implement one of the soft constraints (#450)
     new 79394c9  Add soft constraint: ResourcetopStateAntiAffinityConstraint (#465)
     new bde3449  Implement MaxCapacityUsageInstanceConstraint soft constraint (#463)
     new 2915df1  Add soft constraint: ResourcePartitionAntiAffinityConstraint (#464)
     new 3d7d00f  Improve ResourceTopStateAntiAffinityConstraint (#475)
     new 3b0ecfe  Adjust the expected replica count according to fault zone count. (#476)
     new 5f7bcaa  PartitionMovementSoftConstraint Implementation (#474)
     new 23fc562  Add the remaining implementation of ConstraintBasedAlgorithmFactory (#478)
     new 97c7206  Integrate the WAGED rebalancer with all the related components. (#466)
     new 1ea1e60  Separate AssignableNode properties by Immutable and Mutable (#485)
     new 35a9f8a  Add delayed rebalance and user-defined preference list features to the WAGED
rebalancer. (#456)
     new f33b448  Enable maintenance mode for the WAGED rebalancer.
     new 499847b  Adjust the topology processing logic for instance to ensure backward compatibility.
     new a0aeccc  Load soft constraint weight from resources/properties file (#492)
     new 6bdf45f  Add latency metric components for WAGED rebalancer (#490)
     new c1c910c  Fixing rebalance cache issue and stablize the tests. (#510)
     new f30b9e6  More strict partition weight validation while creating the cluster model.
(#511)
     new 39a1f06  Increase parallelism for ZkBucketDataAccessor (#506)
     new 4fdae2b  The WAGED rebalancer returns the previously calculated assignment on calculation
failure (#514)
     new f63b3e4  Make log clearer after finishing calculateAssignment. (#531)
     new a6bd959  Implement monitoring mbeans for the WAGED rebalancer. (#525)
     new 423c60a  Refine the rebalance scope calculating logic in the WAGED rebalancer. (#519)
     new 9249f04  Make WagedRebalancer static by creating a ThreadLocal (#540)
     new 7af2a4a  Change change detector to a regular field in the WAGED rebalancer instead
of static threadlocal. (#543)
     new e68ffb9  Refactor soft constraints to simply the algorithm and fix potential issues.
(#520)
     new dc29bb4  Minor fix for the constraints related tests. (#545)
     new 17199fe  Adjust the replica rebalance calculating ordering to avoid static order.
(#535)
     new e252907  Implement increment() method in CountMetric class. (#537)
     new 5accc59  Modify the ivy file to add the new math3 lib dependency. (#546)
     new a0721ee  Fix a missing parameter when the WAGED rebalancer init the change detector.
(#547)
     new 9660eb2  Add the new Rebalancer monitor domain to the active domain list. (#550)
     new 4c803f6  Refine ivy file config. The org were not configured correctly. (#551)
     new b19954a  Use a deep copy of the new best possible assignment for measuring baseline
divergence. (#542)
     new 9391cfa  Add max capacity usage metric for instance monitor. (#548)
     new 6d879d1  Fix formula incorrection in the comment for measuring baseline divergence.
(#559)
     new fa71a09  Avoid redundant writes in AssignmentMetadataStore (#564)
     new a9d77b2  Filter resource map with ideal states for instance capacity metrics. (#574)
     new da68dce  Introduce Dry-run Waged Rebalancer for the verifiers and tests. (#573)
     new 3e94337  Change ClusterConfig.setDefaultCapacityMap to be private. (#590)
     new 26755b2  Add Java API for adding and validating resources for WAGED rebalancer (#570)
     new 3aea744  Change calculation for baseline divergence. (#598)
     new 83f1197  Improve the WAGED rebalancer performance. (#586)
     new 0f8d39f  Fix the unstable test TestZeroReplicaAvoidance. (#603)
     new 6694dbc  Add REST API endpoints for WAGED Rebalancer (#611)
     new b67a841  Fix a potential issue in the ResourceChangeSnapshot. (#635)
     new 5389538  Simply and enhance the RebalanceLatencyGauge so it can be used in multi-threads.
(#636)
     new ac19dc4  Add new WAGED rebalancer config item "GLOBAL_REBALANCE_ASYNC_MODE". (#637)
     new 2d2201c  Decouple the event type and the scheduled rebalance cache refresh option.
(#638)
     new 170a599  Improve the algorithm so it prioritizes the assignment to the idle nodes
when the constraint evaluation results are the same (#651)
     new ec97c83  Refine the WAGED rebalancer to minimize the partial rebalance workload.
(#639)
     new 5944324  Refine methods name and comments. (#664)
     new cea1f36  Asynchronously calculating the Baseline (#632)
     new 2e34aaa  Reorgnize the test case so the new WAGED expand cluster tests are not skipped.
(#670)
     new de119d2  Fix the Helix rest tests by cleaning up the environment before testing.
(#679)
     new 9cef5e4  Add instance capacity gauge (#557)
     new e0b2cb9  Add resource partition weight gauge (#686)
     new 8facd94  Add WAGED rebalancer reset method to clean up cached status. (#696)
     new dedfdfe  Reset the WAGED rebalancer once the controller newly acquires leadership.
(#690)

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (6aefab7)
            \
             N -- N -- N   refs/heads/wagedRebalancer (dedfdfe)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 80 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../main/java/org/apache/helix/HelixTimerTask.java |   5 +
 .../stages/TaskGarbageCollectionStage.java         |   2 +
 .../healthcheck/ParticipantHealthReportTask.java   |   2 -
 .../apache/helix/manager/zk/ZKHelixManager.java    |   4 +-
 .../main/java/org/apache/helix/task/TaskUtil.java  |  44 ++++++
 .../helix/integration/TestStandAloneCMMain.java    |  83 -----------
 .../controller/TestControllerLeadershipChange.java | 165 +++++++++++++++++++--
 .../manager/ClusterControllerManager.java          |   7 +-
 .../task/TestWorkflowContextWithoutConfig.java     | 138 ++++++++++++-----
 helix-rest/pom.xml                                 |   2 +-
 10 files changed, 318 insertions(+), 134 deletions(-)
 delete mode 100644 helix-core/src/test/java/org/apache/helix/integration/TestStandAloneCMMain.java


Mime
View raw message