[RIP-70-3]Extract adaptive lock mechanism by 3424672656 · Pull Request #8663 · apache/rocketmq

3424672656 · Sep 7, 2024

Which Issue(s) This PR Fixes

Brief Description

How Did You Test This Change?

Test document ：https://shimo.im/docs/ZzkLMQ4RwwUa87AQ/

codecov-commenter · Sep 7, 2024

Codecov Report

❌ Patch coverage is 70.45455% with 39 lines in your changes missing coverage. Please review.
✅ Project coverage is 47.37%. Comparing base (2956f6d) to head (39180a3).
⚠️ Report is 354 commits behind head on develop.

Files with missing lines	Patch %	Lines
...cketmq/store/lock/AdaptiveBackOffSpinLockImpl.java	71.79%	18 Missing and 4 partials ⚠️
...rg/apache/rocketmq/store/lock/BackOffSpinLock.java	77.14%	6 Missing and 2 partials ⚠️
...ache/rocketmq/store/config/MessageStoreConfig.java	37.50%	5 Missing ⚠️
...main/java/org/apache/rocketmq/store/CommitLog.java	33.33%	0 Missing and 2 partials ⚠️
...e/rocketmq/store/lock/AdaptiveBackOffSpinLock.java	0.00%	2 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             develop    #8663      +/-   ##
=============================================
- Coverage      47.39%   47.37%   -0.03%     
- Complexity     11628    11645      +17     
=============================================
  Files           1290     1294       +4     
  Lines          90293    90424     +131     
  Branches       11609    11627      +18     
=============================================
+ Hits           42795    42834      +39     
- Misses         42231    42306      +75     
- Partials        5267     5284      +17

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

fuyou001

CR need

# Conflicts: # store/src/main/java/org/apache/rocketmq/store/lock/AdaptiveLockImpl.java # store/src/main/java/org/apache/rocketmq/store/lock/CollisionRetreatLock.java

GenerousMan · Sep 26, 2024

+        if (this.adaptiveLock instanceof CollisionRetreatLock) {
+            CollisionRetreatLock lock = (CollisionRetreatLock) this.adaptiveLock;
+            int base = Math.min(200 + tps / 200, 500);
+            if (lock.getNumberOfRetreat(slot) * base >= tps) {
+                if (lock.isAdapt()) {
+                    lock.adapt(true);
+                } else {
+                    this.tpsSwapCriticalPoint = tps;
+                    needSwap = true;
+                }
+            } else if (lock.getNumberOfRetreat(slot) * base * 3 / 2 <= tps) {
+                lock.adapt(false);
+            }
+            lock.setNumberOfRetreat(slot, 0);
+        } else {
+            if (tps <= this.tpsSwapCriticalPoint * 4 / 5) {
+                needSwap = true;
+            }


Too many magic numbers, it should be optimized.

GenerousMan · Sep 26, 2024

+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+
+public class CollisionRetreatLock implements AdaptiveBackOffLock {


What is RetreatLock? Is it a misspelling of reentrant lock?

GenerousMan · Sep 26, 2024

+    private final List<AtomicInteger> numberOfRetreat;
+
+    public CollisionRetreatLock() {
+        this.initOptimalDegree = 1000;


You'd better make these numbers as constants, like OPTIMAL_DEGREE = 1000, MAX_OPTIMAL_DEGREE = 10000

fuyou001 · Oct 17, 2024

+
+public interface AdaptiveBackOffSpinLock extends PutMessageLock {
+
+    void lock();


The PutMessageLock class includes methods for locking and unlocking.

fuyou001 · Oct 17, 2024

        };
-        this.putMessageLock = messageStore.getMessageStoreConfig().isUseReentrantLockWhenPutMessage() ? new PutMessageReentrantLock() : new PutMessageSpinLock();
+
+        AdaptiveBackOffSpinLock adaptiveBackOffSpinLock = new AdaptiveBackOffSpinLockImpl();


PutMessageLock adaptiveBackOffSpinLock = new AdaptiveBackOffSpinLockImpl() may be better

fuyou001 · Oct 17, 2024

+            }
+        }
+
+        if (needSwap) {


Add some comments to explain the principles and the rationale behind these numbers.

fuyou001 · Oct 17, 2024

+
+    public AdaptiveBackOffSpinLockImpl() {
+        this.locks = new HashMap<>();
+        this.locks.put("Reentrant", new BackOffReentrantLock());


These strings use class constants.

… adaptive_lock # Conflicts: # store/src/main/java/org/apache/rocketmq/store/config/MessageStoreConfig.java

GenerousMan

LGTM

# Conflicts: # store/src/main/java/org/apache/rocketmq/store/config/MessageStoreConfig.java

* extract the adaptive lock * extract the adaptive lock * feat(): perfect the adaptive lock * feat(): perfect the adaptive lock * Optimized code type * Optimized code type * Optimized code type * fix fail test * Optimize the adaptive locking mechanism logic * Optimize the adaptive locking mechanism logic * feat:Adaptive locking mechanism adjustment * feat:Adaptive locking mechanism adjustment * feat:Adaptive locking mechanism adjustment * Optimize the adaptive locking mechanism logic * Optimize the adaptive locking mechanism logic * Optimize the adaptive locking mechanism logic * feat:Supports the hot activation of ABS locks * feat:Supports the hot activation of ABS locks * feat:Supports the hot activation of ABS locks * feat:Supports the hot activation of ABS locks * Optimize code style * Optimize code style * Optimize code style * Optimize code style * Optimize code style * Optimize code style * Updated the locking mechanism name * Optimize the logic of switching to spin locks * Optimize the logic of switching to spin locks * Optimize the logic of switching to spin locks * Optimize the logic of switching to spin locks * Optimize the logic of switching to spin locks * Optimize the logic of switching to spin locks * Optimize the logic of switching to spin locks * Optimize the logic of switching to spin locks * delete unused import * Optimize the logic of switching to spin locks * Revert "Optimize the logic of switching to spin locks" This reverts commit 1d7bac5. * Optimize the logic of switching to spin locks * Optimize the logic of switching to spin locks * Optimize the logic of switching to spin locks * Optimize the logic of switching to spin locks * Optimize the logic of switching to spin locks * Optimize the logic of switching to spin locks * Optimize the logic of switching to spin locks * Optimized locking logic * Optimized locking logic * Optimized locking logic * fix test * fix test * fix test * fix test * Optimize code style * Optimize code style * fix test * fix test * optimize client rebalancing logic --------- Co-authored-by: wanghuaiyuan <wanghuaiyuan@xiaomi.com>

3424672656 added 8 commits September 7, 2024 13:50

extract the adaptive lock

ac6b13d

extract the adaptive lock

c4bdf51

feat(): perfect the adaptive lock

12094b4

feat(): perfect the adaptive lock

2b3c647

Optimized code type

6d02e7e

Optimized code type

2a5e23e

Optimized code type

9714ea0

fix fail test

ff9f994

fuyou001 requested changes Sep 20, 2024

View reviewed changes

wanghuaiyuan and others added 14 commits September 20, 2024 15:59

Optimize the adaptive locking mechanism logic

439e2b8

Optimize the adaptive locking mechanism logic

1d00404

feat:Adaptive locking mechanism adjustment

0904c50

feat:Adaptive locking mechanism adjustment

99e07d6

feat:Adaptive locking mechanism adjustment

ef10b0a

Merge remote-tracking branch 'origin/adaptive_lock' into adaptive_lock

8cfcf83

# Conflicts: # store/src/main/java/org/apache/rocketmq/store/lock/AdaptiveLockImpl.java # store/src/main/java/org/apache/rocketmq/store/lock/CollisionRetreatLock.java

Optimize the adaptive locking mechanism logic

a49deda

Optimize the adaptive locking mechanism logic

429f9e3

Optimize the adaptive locking mechanism logic

8b98e66

feat:Supports the hot activation of ABS locks

27a0a05

feat:Supports the hot activation of ABS locks

c8b4339

feat:Supports the hot activation of ABS locks

154fe75

Merge branch 'apache:develop' into adaptive_lock

8b9722d

feat:Supports the hot activation of ABS locks

568f412

GenerousMan reviewed Sep 26, 2024

View reviewed changes

wanghuaiyuan added 2 commits September 26, 2024 19:49

Optimize code style

f674f1b

Optimize code style

61a9354

3424672656 requested a review from GenerousMan September 26, 2024 11:59

wanghuaiyuan and others added 10 commits October 11, 2024 17:51

Optimize the logic of switching to spin locks

882f7ed

Optimized locking logic

cb96e72

Merge branch 'apache:develop' into adaptive_lock

c0751c7

Optimized locking logic

7b4401e

Merge remote-tracking branch 'origin/adaptive_lock' into adaptive_lock

52c359d

Optimized locking logic

3ed1e35

fix test

c401dca

fix test

3c6bcef

fix test

0302a49

fix test

530cb62

fuyou001 reviewed Oct 17, 2024

View reviewed changes

wanghuaiyuan added 4 commits October 17, 2024 15:41

Optimize code style

cfc0240

Optimize code style

f526056

fix test

447ad62

fix test

e886447

fuyou001 previously approved these changes Oct 19, 2024

View reviewed changes

3424672656 requested a review from RongtongJin October 19, 2024 06:24

Merge branch 'develop' of https://github.com/3424672656/rocketmq into…

4b7fa65

… adaptive_lock # Conflicts: # store/src/main/java/org/apache/rocketmq/store/config/MessageStoreConfig.java

3424672656 dismissed fuyou001’s stale review via 4b7fa65 October 19, 2024 06:27

3424672656 requested a review from fuyou001 October 19, 2024 07:11

GenerousMan approved these changes Oct 22, 2024

View reviewed changes

RongtongJin previously approved these changes Oct 23, 2024

View reviewed changes

Merge branch 'develop' into adaptive_lock

d056c40

# Conflicts: # store/src/main/java/org/apache/rocketmq/store/config/MessageStoreConfig.java

3424672656 dismissed RongtongJin’s stale review via d056c40 October 23, 2024 09:12

optimize client rebalancing logic

39180a3

fuyou001 approved these changes Oct 23, 2024

View reviewed changes

RongtongJin approved these changes Oct 23, 2024

View reviewed changes

RongtongJin merged commit 95b88ff into apache:develop Oct 23, 2024


		public interface AdaptiveBackOffSpinLock extends PutMessageLock {

		void lock();

Search code, repositories, users, issues, pull requests...

Conversation

3424672656 commented Sep 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which Issue(s) This PR Fixes

Brief Description

How Did You Test This Change?

Uh oh!

codecov-commenter commented Sep 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

fuyou001 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GenerousMan left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

3424672656 commented Sep 7, 2024 •

edited

Loading

codecov-commenter commented Sep 7, 2024 •

edited

Loading