Skip to content

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

[Lock] Add MysqlStore #45982

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 11 commits into
base: 7.3
Choose a base branch
Loading
from
Open

[Lock] Add MysqlStore #45982

wants to merge 11 commits into from

Conversation

rtek
Copy link
Contributor

@rtek rtek commented Apr 9, 2022

Q A
Branch? 6.1 6.2
Bug fix? no
New feature? yes
Deprecations? no
Tickets Fix #45251
License MIT
Doc PR TODO

Add MysqlStore based on MySQL GET_LOCK() functionality

Based on #25578

Welcoming feedback before adding docs.

@GromNaN GromNaN self-requested a review April 10, 2022 06:20
Copy link
Member

@GromNaN GromNaN left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking over this feature. It's too late for 6.1, but will be a good addition for 6.2.

Since the first PR, @jderusse created PostgreSqlStore, which is very close to this MysqlStore.

Also, we split stores for PDO Vs DBAL connection. Once the PDO version is complete, you can create the DoctrineDbalMysqlStore

src/Symfony/Component/Lock/CHANGELOG.md Outdated Show resolved Hide resolved
src/Symfony/Component/Lock/Store/MysqlStore.php Outdated Show resolved Hide resolved
src/Symfony/Component/Lock/Store/MysqlStore.php Outdated Show resolved Hide resolved
$name = (string) $key;
$name = \strlen($name) > 64 ? hash('sha256', $name) : $name;

$stmt = $this->conn->prepare('SELECT IF(IS_USED_LOCK(:name) = CONNECTION_ID(), -1, GET_LOCK(:name, 0))');
Copy link
Member

@GromNaN GromNaN Apr 10, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostgreSqlStore uses the InMemoryStore to avoid duplicate lock on the same connection.

return self::$storeRegistry[$namespace] ?? self::$storeRegistry[$namespace] = new InMemoryStore();

Copy link
Contributor Author

@rtek rtek Apr 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you elaborate on this requirement?

Im concerned that using the InMemoryStore can cause unexpected behavior in situations where either:

  1. A combined store is used with with Mysql and Postgres
  2. A the same key is passed to two LockFactory::createLockFromKey, one using Mysql and the other using Postgres

I moved the state into the store since the tests assert that exists() is bound to the Key instance, while save() is bound to the Key::$resource. Does this work?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. They should not share the same instance of InMemoryStore

$this->connectionId = $this->getConnection()->query('SELECT CONNECTION_ID()')->fetchColumn();
}

return $this->connectionId.'_'.spl_object_id($key);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

spl_object_id is not reliable.

The above code should work

$key = new Key('foo');
$store->save($key);

$key = clone($key);
$this->assertTrue($store->exists($key));

private function getLockId(Key $key): string
{
if (!isset($this->connectionId)) {
$this->connectionId = $this->getConnection()->query('SELECT CONNECTION_ID()')->fetchColumn();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line triggers PDOException: SQLSTATE[HY000] [2002] No such file or directory on my local machine

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced the MYSQL_HOST with MYSQL_DSN envar so you can test using a socket dsn. I think that's what your error is related to.


public function exists(Key $key): bool
{
return self::$locksAcquired[$this->getLockId($key)] ?? false;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's wrong, the implementation must assert that the lock is still present in the backend. (Ie the database restart, the network is down, ...)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This wont be possible since GET_LOCK is bound to the connection so its always released when the connection is terminated.

Copy link
Member

@jderusse jderusse Apr 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well.. getLockId is based on connection_id() which is (for MySQL 8 at least) an auto-increment that is reset when server starts.

This means, that each time the server restarts, the application might retrieve the connection_id previously affected to another process. Which makes the key used in self::$locksAcquired untrustable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should do it.

I cant add this test case to the since it requires rebooting mysql but I did run it manually to confirm proper handling even if the connection id is the same

class ManualMysqlStore extends MysqlStoreTest
{
    public function testMysqlClosedConnection()
    {
        // manually restart mysql to reset conn id counter
        $pdo = $this->getPdo();
        $storeA = new MysqlStore($pdo);
        $key = new Key('foo');
        $storeA->save($key);

        $connIdA = $pdo->query('SELECT CONNECTION_ID()')->fetchColumn();

        while (true) {
            try {
                $this->assertTrue($storeA->exists($key));
                sleep(1);
                // manually restart mysql to reset conn id counter
            } catch (\PDOException $e) {
                // mysql has gone away
                break;
            }
        }

        $pdo = $this->getPdo();
        $storeB = new MysqlStore($pdo);
        $connIdB = $pdo->query('SELECT CONNECTION_ID()')->fetchColumn();

        $this->assertFalse($storeB->exists($key));
        $this->assertSame($connIdA, $connIdB);
    }
}

$storeB = $this->getStore();

try {
$storeB->save(new Key('foo'));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test is incorrect. You are testing both:

  • 2 instances of Key
  • 2 instances of store

You don't know why the storeB->save throws an exception.

For the record the below code is OK:

$resource = uniqid(__METHOD__, true);
$key1 = new Key($resource);
$key2 = new Key($resource);

$storeA->save($key1);
$storeb->save($key1);

Because stores are stateless, and you are allowed to use 2 instances of store to manipulate the same lock.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because stores are stateless, and you are allowed to use 2 instances of store to manipulate the same lock.

makes sense I see where you're going now

$name = (string) $key;
$name = \strlen($name) > 64 ? hash('sha256', $name) : $name;

$stmt = $this->conn->prepare('SELECT IF(IS_USED_LOCK(:name) = CONNECTION_ID(), -1, GET_LOCK(:name, 0))');
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. They should not share the same instance of InMemoryStore

Copy link
Contributor

@kaznovac kaznovac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great work @rtek, if you don't mind i have added a few remarks

return false;
}

$stmt = $this->getConnection()->prepare('SELECT IF(IS_USED_LOCK(:name) = CONNECTION_ID(), 1, 0)');
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this prepares the statement each time the method is called, consider moving it to after the connection is established and capturing statement in an object property

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IF statement is technically not needed, return the result of comparison

Suggested change
$stmt = $this->getConnection()->prepare('SELECT IF(IS_USED_LOCK(:name) = CONNECTION_ID(), 1, 0)');
$stmt = $this->getConnection()->prepare('SELECT IS_USED_LOCK(:name) = CONNECTION_ID()');

you would get the 1 when the lock is held, and null when it's not (safer to assume not 1 value)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agree

Comment on lines 101 to 102
$stmt->bindValue(':name', self::getLockName($key), \PDO::PARAM_STR);
$stmt->execute();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consider using execute with an argument for binding (the Key name has been already 'forced' to a string by letLockName) - it's easier to read without semantic drawbacks

Suggested change
$stmt->bindValue(':name', self::getLockName($key), \PDO::PARAM_STR);
$stmt->execute();
$stmt->execute([':name' => self::getLockName($key)]);

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agree - did the others too

}

$name = self::getLockName($key);
$stmt = $this->getConnection()->prepare('SELECT IF(IS_USED_LOCK(:name) = CONNECTION_ID(), -1, GET_LOCK(:name, 0))');
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some mysql peculates should be documented (and maybe this store should prevent instantiation on unsupported version or scenario):

https://dev.mysql.com/doc/refman/5.7/en/locking-functions.html

  •   SELECT GET_LOCK('lock1',10);
      SELECT GET_LOCK('lock2',10);
      SELECT RELEASE_LOCK('lock2');
      SELECT RELEASE_LOCK('lock1');
    

    In MySQL 5.7 or later, the second [GET_LOCK()] (https://dev.mysql.com/doc/refman/5.7/en/locking-functions.html#function_get-lock) acquires a second lock and both RELEASE_LOCK() calls return 1 (success). Before MySQL 5.7, the second GET_LOCK() releases the first lock ('lock1') and the second RELEASE_LOCK() returns NULL (failure) because there is no 'lock1' to release.

  • MySQL 5.7 and later enforces a maximum length on lock names of 64 characters. Previously, no limit was enforced.

  • This function is [All lock functions are] unsafe for statement-based replication. A warning is logged if you use this function when binlog_format is set to STATEMENT.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please investigate the isolation of this lock acquire.

i'm not sure the IF statement is atomic nor serialized, so i've posted the question on sf https://stackoverflow.com/questions/71957583/is-mysql-if-statement-atomic

Copy link
Contributor Author

@rtek rtek Apr 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MySQL 5.7 and later enforces a maximum length on lock names of 64 characters.

Handled by getLockName

...unsafe for statement-based replication

After some digging, I don't believe this is relevant to the use case.

  1. The locks are "locked on a server-wide basis." so i dont one would expect a replica to respect locks on the master.
  2. The STATEMENT based replication warning is applicable if these various functions are used within DML (e.g. insert into foo values(is_used_lock('bar')) so a replica would get a different value when replicating that statement (since the lock situation on the replica is not the same as the master).

please investigate the isolation of this lock acquire.

These functions do not interact with transactions per "Locks obtained with GET_LOCK() are not released when transactions commit or roll back."

Heres a quick demo:

$pdoA = ...
$pdoB = ...

$dump = function (\PDO $pdo, string $sql) {
    printf("%s : %s\n",
         $sql,
        var_export($pdo->query($sql)->fetchColumn(), true)
    );
};

$dump($pdoA, "select get_lock('foo',0)");
$dump($pdoB, "select get_lock('foo',0)");
$dump($pdoA, "select release_lock('foo')");

$pdoA->beginTransaction();
$dump($pdoA, "select get_lock('foo',0)");
$dump($pdoB, "select get_lock('foo',0)");
select get_lock('foo',0) : 1
select get_lock('foo',0) : 0
select release_lock('foo') : 1
select get_lock('foo',0) : 1
select get_lock('foo',0) : 0

* @author rtek
* @author Jérôme TAMARELLE <jerome@tamarelle.net>
*/
class MysqlStore implements PersistingStoreInterface
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe rename it to MysqlLocksStore to indicate Mysql user locks are used (not to be confused with data storage in Pdo store

@fabpot fabpot modified the milestones: 6.1, 6.2 Apr 22, 2022
@nicolas-grekas nicolas-grekas removed this from the 6.2 milestone Nov 5, 2022
@nicolas-grekas nicolas-grekas added this to the 6.3 milestone Nov 5, 2022
@GromNaN
Copy link
Member

GromNaN commented Jan 19, 2023

Hello @rtek do you want to continue this PR? It is fine if you want to work on in a few weeks. Let me know if you want to hand it over.

@rtek
Copy link
Contributor Author

rtek commented Jan 21, 2023

Yes - Id like to wrap this up. Should I close and re-open to get a fresh review going?

@GromNaN
Copy link
Member

GromNaN commented Jan 25, 2023

Yes - Id like to wrap this up. Should I close and re-open to get a fresh review going?

You can rebase and update this PR.

@nicolas-grekas nicolas-grekas modified the milestones: 6.3, 6.4 May 23, 2023
@nicolas-grekas nicolas-grekas modified the milestones: 6.4, 7.1 Nov 15, 2023
@xabbuh xabbuh modified the milestones: 7.1, 7.2 May 15, 2024
@fabpot fabpot modified the milestones: 7.2, 7.3 Nov 20, 2024
}

if (-1 === $result) {
throw new LockConflictedException('Lock already acquired by this connection.');
Copy link

@javarr javarr Dec 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, thanks for your work!, I have one feedback:
I think that is necessary to throw different type of exception if user already has LOCK, i sow the implementation in vendor/symfony/lock/Lock.php:69, is better to ignore it if the user/connection already has the lock, or i am wrong?

image

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a Lock already acquired by this connection should not throw a LockConflictedException (btw, this is what your exists check tries to avoid also)

// mysql limits lock name length to 64 chars
$name = (string) $key;

return \strlen($name) > 64 ? hash('xxh128', $name) : $name;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hashing only longer names will mean that there is 2 resources that share the same MySQL lock name

}

if (-1 === $result) {
throw new LockConflictedException('Lock already acquired by this connection.');
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a Lock already acquired by this connection should not throw a LockConflictedException (btw, this is what your exists check tries to avoid also)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Lock] MySQL non-expiring locking using GET_LOCK
10 participants
Morty Proxy This is a proxified and sanitized view of the page, visit original site.