Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

[Lock] Add MysqlStore #45982

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 11 commits into
base: 7.4
Choose a base branch
Loading
from
Open
Show file tree
Hide file tree
Changes from 7 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions 6 .github/workflows/integration-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,9 @@ jobs:
./phpunit install
echo "::endgroup::"

- name: Start MySQL
run: sudo systemctl start mysql.service

- name: Run tests
run: ./phpunit --group integration -v
env:
Expand All @@ -174,6 +177,9 @@ jobs:
MESSENGER_SQS_FIFO_QUEUE_DSN: "sqs://localhost:9494/messages.fifo?sslmode=disable&poll_timeout=0.01"
KAFKA_BROKER: 127.0.0.1:9092
POSTGRES_HOST: localhost
MYSQL_HOST: localhost # ubuntu-20.04 mysql defaults
MYSQL_USERNAME: root
MYSQL_PASSWORD: root

#- name: Run HTTP push tests
# if: matrix.php == '8.1'
Expand Down
3 changes: 3 additions & 0 deletions 3 src/Symfony/Component/Lock/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
CHANGELOG
=========
6.2
---
* Add `MysqlStore` based on MySQL `GET_LOCK()` functionality

6.0
---
Expand Down
1 change: 1 addition & 0 deletions 1 src/Symfony/Component/Lock/Store/DoctrineDbalStore.php
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,7 @@ private function prune(): void
private function getCurrentTimestampStatement(): string
{
$platform = $this->conn->getDatabasePlatform();

return match (true) {
$platform instanceof \Doctrine\DBAL\Platforms\MySQLPlatform,
$platform instanceof \Doctrine\DBAL\Platforms\MySQL57Platform => 'UNIX_TIMESTAMP()',
Expand Down
141 changes: 141 additions & 0 deletions 141 src/Symfony/Component/Lock/Store/MysqlStore.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
<?php

/*
* This file is part of the Symfony package.
*
* (c) Fabien Potencier <fabien@symfony.com>
*
* For the full copyright and license information, please view the LICENSE
* file that was distributed with this source code.
*/

namespace Symfony\Component\Lock\Store;

use Symfony\Component\Lock\Exception\InvalidArgumentException;
use Symfony\Component\Lock\Exception\LockAcquiringException;
use Symfony\Component\Lock\Exception\LockConflictedException;
use Symfony\Component\Lock\Key;
use Symfony\Component\Lock\PersistingStoreInterface;

/**
* @author rtek
* @author Jérôme TAMARELLE <jerome@tamarelle.net>
*/
class MysqlStore implements PersistingStoreInterface
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe rename it to MysqlLocksStore to indicate Mysql user locks are used (not to be confused with data storage in Pdo store

{
private \PDO $conn;

private string $dsn;

private array $options;

private int $connectionId;

/** @var bool[] */
private static array $locksAcquired = [];

public function __construct(\PDO|string $connOrDsn, array $options = [])
{
if ($connOrDsn instanceof \PDO) {
$this->conn = $connOrDsn;
$this->assertMysqlDriver();
if (\PDO::ERRMODE_EXCEPTION !== $this->conn->getAttribute(\PDO::ATTR_ERRMODE)) {
throw new InvalidArgumentException(sprintf('"%s" requires PDO error mode attribute be set to throw Exceptions (i.e. $pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION)).', __METHOD__));
}
} else {
$this->dsn = $connOrDsn;
}

$this->options = $options;
}

public function save(Key $key): void
{
$id = $this->getLockId($key);

if (self::$locksAcquired[$id] ?? false) {
return;
}

$stmt = $this->conn->prepare('SELECT IF(IS_USED_LOCK(:name) = CONNECTION_ID(), -1, GET_LOCK(:name, 0))');
Copy link
Member

@GromNaN GromNaN Apr 10, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostgreSqlStore uses the InMemoryStore to avoid duplicate lock on the same connection.

return self::$storeRegistry[$namespace] ?? self::$storeRegistry[$namespace] = new InMemoryStore();

Copy link
Contributor Author

@rtek rtek Apr 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you elaborate on this requirement?

Im concerned that using the InMemoryStore can cause unexpected behavior in situations where either:

  1. A combined store is used with with Mysql and Postgres
  2. A the same key is passed to two LockFactory::createLockFromKey, one using Mysql and the other using Postgres

I moved the state into the store since the tests assert that exists() is bound to the Key instance, while save() is bound to the Key::$resource. Does this work?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. They should not share the same instance of InMemoryStore

$stmt->bindValue(':name', self::getLockName($key), \PDO::PARAM_STR);
$stmt->execute();
$result = $stmt->fetchColumn();

// lock acquired
if (1 === $result) {
self::$locksAcquired[$id] = true;

return;
}

if (0 === $result) {
throw new LockConflictedException('Lock already acquired by other connection.');
}

if (-1 === $result) {
throw new LockConflictedException('Lock already acquired by this connection.');
Copy link

@javarr javarr Dec 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, thanks for your work!, I have one feedback:
I think that is necessary to throw different type of exception if user already has LOCK, i sow the implementation in vendor/symfony/lock/Lock.php:69, is better to ignore it if the user/connection already has the lock, or i am wrong?

image

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a Lock already acquired by this connection should not throw a LockConflictedException (btw, this is what your exists check tries to avoid also)

}

throw new LockAcquiringException('Failed to acquire lock due to mysql error.');
}

public function putOffExpiration(Key $key, float $ttl): void
{
// noop - GET_LOCK() does not have a ttl
}

public function delete(Key $key): void
{
$stmt = $this->conn->prepare('DO RELEASE_LOCK(:name)');
$stmt->bindValue(':name', self::getLockName($key), \PDO::PARAM_STR);
$stmt->execute();

unset(self::$locksAcquired[$this->getLockId($key)]);
}

public function exists(Key $key): bool
{
return self::$locksAcquired[$this->getLockId($key)] ?? false;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's wrong, the implementation must assert that the lock is still present in the backend. (Ie the database restart, the network is down, ...)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This wont be possible since GET_LOCK is bound to the connection so its always released when the connection is terminated.

Copy link
Member

@jderusse jderusse Apr 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well.. getLockId is based on connection_id() which is (for MySQL 8 at least) an auto-increment that is reset when server starts.

This means, that each time the server restarts, the application might retrieve the connection_id previously affected to another process. Which makes the key used in self::$locksAcquired untrustable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should do it.

I cant add this test case to the since it requires rebooting mysql but I did run it manually to confirm proper handling even if the connection id is the same

class ManualMysqlStore extends MysqlStoreTest
{
    public function testMysqlClosedConnection()
    {
        // manually restart mysql to reset conn id counter
        $pdo = $this->getPdo();
        $storeA = new MysqlStore($pdo);
        $key = new Key('foo');
        $storeA->save($key);

        $connIdA = $pdo->query('SELECT CONNECTION_ID()')->fetchColumn();

        while (true) {
            try {
                $this->assertTrue($storeA->exists($key));
                sleep(1);
                // manually restart mysql to reset conn id counter
            } catch (\PDOException $e) {
                // mysql has gone away
                break;
            }
        }

        $pdo = $this->getPdo();
        $storeB = new MysqlStore($pdo);
        $connIdB = $pdo->query('SELECT CONNECTION_ID()')->fetchColumn();

        $this->assertFalse($storeB->exists($key));
        $this->assertSame($connIdA, $connIdB);
    }
}

}

private function getConnection(): \PDO
{
if (!isset($this->conn)) {
$this->conn = new \PDO(
$this->dsn,
$this->options['db_username'] ?? null,
$this->options['db_password'] ?? null,
$this->options['db_connection_options'] ?? null
);
$this->conn->setAttribute(\PDO::ATTR_ERRMODE, \PDO::ERRMODE_EXCEPTION);
$this->assertMysqlDriver();
}

return $this->conn;
}

private function assertMysqlDriver(): void
{
if ('mysql' !== $driver = $this->conn->getAttribute(\PDO::ATTR_DRIVER_NAME)) {
throw new InvalidArgumentException(sprintf('The adapter "%s" does not support the "%s" driver.', __CLASS__, $driver));
}
}

private function getLockId(Key $key): string
{
if (!isset($this->connectionId)) {
$this->connectionId = $this->getConnection()->query('SELECT CONNECTION_ID()')->fetchColumn();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line triggers PDOException: SQLSTATE[HY000] [2002] No such file or directory on my local machine

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced the MYSQL_HOST with MYSQL_DSN envar so you can test using a socket dsn. I think that's what your error is related to.

}

return $this->connectionId.'_'.spl_object_id($key);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

spl_object_id is not reliable.

The above code should work

$key = new Key('foo');
$store->save($key);

$key = clone($key);
$this->assertTrue($store->exists($key));

}

private static function getLockName(Key $key): string
{
// mysql limits lock name length to 64 chars
$name = (string) $key;

return \strlen($name) > 64 ? hash('xxh128', $name) : $name;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hashing only longer names will mean that there is 2 resources that share the same MySQL lock name

}
}
79 changes: 79 additions & 0 deletions 79 src/Symfony/Component/Lock/Tests/Store/MysqlStoreTest.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
<?php

namespace Symfony\Component\Lock\Tests\Store;

use Symfony\Component\Lock\Exception\InvalidArgumentException;
use Symfony\Component\Lock\Exception\LockConflictedException;
use Symfony\Component\Lock\Key;
use Symfony\Component\Lock\PersistingStoreInterface;
use Symfony\Component\Lock\Store\MysqlStore;

class MysqlStoreTest extends AbstractStoreTest
{
protected function getEnv(): array
{
if (!$host = getenv('MYSQL_HOST')) {
$this->markTestSkipped('Missing MYSQL_HOST env variable');
}

if (!$user = getenv('MYSQL_USERNAME')) {
$this->markTestSkipped('Missing MYSQL_USERNAME env variable');
}

if (!$pass = getenv('MYSQL_PASSWORD')) {
$this->markTestSkipped('Missing MYSQL_PASSWORD env variable');
}

return [$host, $user, $pass];
}

protected function getPdo(): \PDO
{
[$host, $user, $pass] = $this->getEnv();

return new \PDO('mysql:host='.$host, $user, $pass);
}

protected function getStore(): PersistingStoreInterface
{
return new MysqlStore($this->getPdo());
}

public function testDriverRequirement()
{
$this->expectException(InvalidArgumentException::class);
new MysqlStore(new \PDO('sqlite::memory:'));
}

public function testExceptionModeRequirement()
{
$this->expectException(InvalidArgumentException::class);
$pdo = $this->getPdo();
$pdo->setAttribute(\PDO::ATTR_ERRMODE, \PDO::ERRMODE_SILENT);
new MysqlStore($pdo);
}

public function testOtherConflictException()
{
$storeA = $this->getStore();
$storeA->save(new Key('foo'));

$storeB = $this->getStore();

try {
$storeB->save(new Key('foo'));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test is incorrect. You are testing both:

  • 2 instances of Key
  • 2 instances of store

You don't know why the storeB->save throws an exception.

For the record the below code is OK:

$resource = uniqid(__METHOD__, true);
$key1 = new Key($resource);
$key2 = new Key($resource);

$storeA->save($key1);
$storeb->save($key1);

Because stores are stateless, and you are allowed to use 2 instances of store to manipulate the same lock.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because stores are stateless, and you are allowed to use 2 instances of store to manipulate the same lock.

makes sense I see where you're going now

$this->fail('Expected exception: '.LockConflictedException::class);
} catch (LockConflictedException $e) {
$this->assertStringContainsString('acquired by other', $e->getMessage());
}
}

public function testDsnConstructor()
{
$this->expectNotToPerformAssertions();

[$host, $user, $pass] = $this->getEnv();
$store = new MysqlStore("mysql:$host", ['db_username' => $user, 'db_password' => $pass]);
$store->save(new Key('foo'));
}
}
Morty Proxy This is a proxified and sanitized view of the page, visit original site.