-
-
Notifications
You must be signed in to change notification settings - Fork 9.6k
[Lock] Add MysqlStore
#45982
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: 7.3
Are you sure you want to change the base?
[Lock] Add MysqlStore
#45982
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for taking over this feature. It's too late for 6.1, but will be a good addition for 6.2.
Since the first PR, @jderusse created PostgreSqlStore, which is very close to this MysqlStore.
Also, we split stores for PDO Vs DBAL connection. Once the PDO version is complete, you can create the DoctrineDbalMysqlStore
$name = (string) $key; | ||
$name = \strlen($name) > 64 ? hash('sha256', $name) : $name; | ||
|
||
$stmt = $this->conn->prepare('SELECT IF(IS_USED_LOCK(:name) = CONNECTION_ID(), -1, GET_LOCK(:name, 0))'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
PostgreSqlStore
uses the InMemoryStore to avoid duplicate lock on the same connection.
return self::$storeRegistry[$namespace] ?? self::$storeRegistry[$namespace] = new InMemoryStore(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you elaborate on this requirement?
Im concerned that using the InMemoryStore
can cause unexpected behavior in situations where either:
- A combined store is used with with Mysql and Postgres
- A the same key is passed to two
LockFactory::createLockFromKey
, one using Mysql and the other using Postgres
I moved the state into the store since the tests assert that exists()
is bound to the Key
instance, while save()
is bound to the Key::$resource
. Does this work?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- They should not share the same instance of
InMemoryStore
$this->connectionId = $this->getConnection()->query('SELECT CONNECTION_ID()')->fetchColumn(); | ||
} | ||
|
||
return $this->connectionId.'_'.spl_object_id($key); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
spl_object_id
is not reliable.
The above code should work
$key = new Key('foo');
$store->save($key);
$key = clone($key);
$this->assertTrue($store->exists($key));
private function getLockId(Key $key): string | ||
{ | ||
if (!isset($this->connectionId)) { | ||
$this->connectionId = $this->getConnection()->query('SELECT CONNECTION_ID()')->fetchColumn(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This line triggers PDOException: SQLSTATE[HY000] [2002] No such file or directory
on my local machine
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Replaced the MYSQL_HOST with MYSQL_DSN envar so you can test using a socket dsn. I think that's what your error is related to.
|
||
public function exists(Key $key): bool | ||
{ | ||
return self::$locksAcquired[$this->getLockId($key)] ?? false; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's wrong, the implementation must assert that the lock is still present in the backend. (Ie the database restart, the network is down, ...)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This wont be possible since GET_LOCK is bound to the connection so its always released when the connection is terminated.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well.. getLockId
is based on connection_id()
which is (for MySQL 8 at least) an auto-increment that is reset when server starts.
This means, that each time the server restarts, the application might retrieve the connection_id
previously affected to another process. Which makes the key used in self::$locksAcquired
untrustable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this should do it.
I cant add this test case to the since it requires rebooting mysql but I did run it manually to confirm proper handling even if the connection id is the same
class ManualMysqlStore extends MysqlStoreTest
{
public function testMysqlClosedConnection()
{
// manually restart mysql to reset conn id counter
$pdo = $this->getPdo();
$storeA = new MysqlStore($pdo);
$key = new Key('foo');
$storeA->save($key);
$connIdA = $pdo->query('SELECT CONNECTION_ID()')->fetchColumn();
while (true) {
try {
$this->assertTrue($storeA->exists($key));
sleep(1);
// manually restart mysql to reset conn id counter
} catch (\PDOException $e) {
// mysql has gone away
break;
}
}
$pdo = $this->getPdo();
$storeB = new MysqlStore($pdo);
$connIdB = $pdo->query('SELECT CONNECTION_ID()')->fetchColumn();
$this->assertFalse($storeB->exists($key));
$this->assertSame($connIdA, $connIdB);
}
}
$storeB = $this->getStore(); | ||
|
||
try { | ||
$storeB->save(new Key('foo')); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The test is incorrect. You are testing both:
- 2 instances of Key
- 2 instances of store
You don't know why the storeB->save
throws an exception.
For the record the below code is OK:
$resource = uniqid(__METHOD__, true);
$key1 = new Key($resource);
$key2 = new Key($resource);
$storeA->save($key1);
$storeb->save($key1);
Because stores are stateless, and you are allowed to use 2 instances of store to manipulate the same lock.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Because stores are stateless, and you are allowed to use 2 instances of store to manipulate the same lock.
makes sense I see where you're going now
$name = (string) $key; | ||
$name = \strlen($name) > 64 ? hash('sha256', $name) : $name; | ||
|
||
$stmt = $this->conn->prepare('SELECT IF(IS_USED_LOCK(:name) = CONNECTION_ID(), -1, GET_LOCK(:name, 0))'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- They should not share the same instance of
InMemoryStore
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
great work @rtek, if you don't mind i have added a few remarks
return false; | ||
} | ||
|
||
$stmt = $this->getConnection()->prepare('SELECT IF(IS_USED_LOCK(:name) = CONNECTION_ID(), 1, 0)'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this prepares the statement each time the method is called, consider moving it to after the connection is established and capturing statement in an object property
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IF statement is technically not needed, return the result of comparison
$stmt = $this->getConnection()->prepare('SELECT IF(IS_USED_LOCK(:name) = CONNECTION_ID(), 1, 0)'); | |
$stmt = $this->getConnection()->prepare('SELECT IS_USED_LOCK(:name) = CONNECTION_ID()'); |
you would get the 1
when the lock is held, and null
when it's not (safer to assume not 1
value)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
agree
$stmt->bindValue(':name', self::getLockName($key), \PDO::PARAM_STR); | ||
$stmt->execute(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
consider using execute
with an argument for binding (the Key
name has been already 'forced' to a string by letLockName
) - it's easier to read without semantic drawbacks
$stmt->bindValue(':name', self::getLockName($key), \PDO::PARAM_STR); | |
$stmt->execute(); | |
$stmt->execute([':name' => self::getLockName($key)]); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
agree - did the others too
} | ||
|
||
$name = self::getLockName($key); | ||
$stmt = $this->getConnection()->prepare('SELECT IF(IS_USED_LOCK(:name) = CONNECTION_ID(), -1, GET_LOCK(:name, 0))'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
some mysql peculates should be documented (and maybe this store should prevent instantiation on unsupported version or scenario):
https://dev.mysql.com/doc/refman/5.7/en/locking-functions.html
-
SELECT GET_LOCK('lock1',10); SELECT GET_LOCK('lock2',10); SELECT RELEASE_LOCK('lock2'); SELECT RELEASE_LOCK('lock1');
In MySQL 5.7 or later, the second [GET_LOCK()] (https://dev.mysql.com/doc/refman/5.7/en/locking-functions.html#function_get-lock) acquires a second lock and both RELEASE_LOCK() calls return 1 (success). Before MySQL 5.7, the second GET_LOCK() releases the first lock ('lock1') and the second RELEASE_LOCK() returns NULL (failure) because there is no 'lock1' to release.
-
MySQL 5.7 and later enforces a maximum length on lock names of 64 characters. Previously, no limit was enforced.
-
This function is[All lock functions are] unsafe for statement-based replication. A warning is logged if you use this function when binlog_format is set to STATEMENT.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please investigate the isolation of this lock acquire.
i'm not sure the IF
statement is atomic nor serialized, so i've posted the question on sf https://stackoverflow.com/questions/71957583/is-mysql-if-statement-atomic
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
MySQL 5.7 and later enforces a maximum length on lock names of 64 characters.
Handled by getLockName
...unsafe for statement-based replication
After some digging, I don't believe this is relevant to the use case.
- The locks are "locked on a server-wide basis." so i dont one would expect a replica to respect locks on the master.
- The
STATEMENT
based replication warning is applicable if these various functions are used within DML (e.g.insert into foo values(is_used_lock('bar')
) so a replica would get a different value when replicating that statement (since the lock situation on the replica is not the same as the master).
please investigate the isolation of this lock acquire.
These functions do not interact with transactions per "Locks obtained with GET_LOCK() are not released when transactions commit or roll back."
Heres a quick demo:
$pdoA = ...
$pdoB = ...
$dump = function (\PDO $pdo, string $sql) {
printf("%s : %s\n",
$sql,
var_export($pdo->query($sql)->fetchColumn(), true)
);
};
$dump($pdoA, "select get_lock('foo',0)");
$dump($pdoB, "select get_lock('foo',0)");
$dump($pdoA, "select release_lock('foo')");
$pdoA->beginTransaction();
$dump($pdoA, "select get_lock('foo',0)");
$dump($pdoB, "select get_lock('foo',0)");
select get_lock('foo',0) : 1
select get_lock('foo',0) : 0
select release_lock('foo') : 1
select get_lock('foo',0) : 1
select get_lock('foo',0) : 0
* @author rtek | ||
* @author Jérôme TAMARELLE <jerome@tamarelle.net> | ||
*/ | ||
class MysqlStore implements PersistingStoreInterface |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe rename it to MysqlLocksStore
to indicate Mysql user locks are used (not to be confused with data storage in Pdo store
Hello @rtek do you want to continue this PR? It is fine if you want to work on in a few weeks. Let me know if you want to hand it over. |
Yes - Id like to wrap this up. Should I close and re-open to get a fresh review going? |
You can rebase and update this PR. |
} | ||
|
||
if (-1 === $result) { | ||
throw new LockConflictedException('Lock already acquired by this connection.'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
a Lock already acquired by this connection should not throw a LockConflictedException (btw, this is what your exists
check tries to avoid also)
// mysql limits lock name length to 64 chars | ||
$name = (string) $key; | ||
|
||
return \strlen($name) > 64 ? hash('xxh128', $name) : $name; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hashing only longer names will mean that there is 2 resources that share the same MySQL lock name
} | ||
|
||
if (-1 === $result) { | ||
throw new LockConflictedException('Lock already acquired by this connection.'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
a Lock already acquired by this connection should not throw a LockConflictedException (btw, this is what your exists
check tries to avoid also)
6.16.2Add
MysqlStore
based on MySQLGET_LOCK()
functionalityBased on #25578
Welcoming feedback before adding docs.