Skip to content

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

[Lock] add mongodb store #31889

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Dec 11, 2019
Merged

Conversation

kralos
Copy link

@kralos kralos commented Jun 6, 2019

Q A
Branch? 4.4
Bug fix? no
New feature? yes
BC breaks? no
Deprecations? no
Tests pass? yes (requires ext-mongodb and mongodb/mongodb to test)
Fixed tickets #27345
License MIT
Original Doc PR symfony/symfony-docs#9807
Remove from 4.3 Doc PR symfony/symfony-docs#11686
Add to 4.4 5.1 Doc PR symfony/symfony-docs#11735

Looks like I messed up kralos:27345-lock-mongodb with a force push (trying to fix ci issues) right before it was merged to master (4.3.0).

see #27648

Description
We should support Semaphore Locks with a MongoDB back end to allow those that already use MongoDB as a distributed storage engine.

Symfony already partially supports MongoDB for session storage: Symfony\Component\HttpFoundation\Session\Storage\Handler\MongoDbSessionHandler

Example

$client = new MongoDb\Client();

$store = new Symfony\Component\Lock\Store\MongoDbStore(
    $client
    array(
        'database' => 'my-app',
    )
);
$lockFactory = new Symfony\Component\Lock\Factory($store);
$lock = $lockFactory->createLock('my-resource');

@kralos
Copy link
Author

kralos commented Jun 6, 2019

@fabpot Sorry, this was supposed to be part of #27648 but got missed due to a messed up force push.

I've based this PR on 4.3, is that ok?

rebased to 4.4

@kralos
Copy link
Author

kralos commented Jun 6, 2019

This PR was based on the version approved by @jderusse 1f4d601

which had a failing test (we last minute removed the return index name from MongoDbStore::createTtlIndex).

I have then fixed that failing test 71091fa..d319a2a

@nicolas-grekas nicolas-grekas added this to the 4.3 milestone Jun 6, 2019
@nicolas-grekas
Copy link
Member

nicolas-grekas commented Jun 6, 2019

Please target master 4.4, that's a new feature for sure.

@nicolas-grekas nicolas-grekas modified the milestones: 4.3, next Jun 6, 2019
@nicolas-grekas nicolas-grekas changed the title 27345 mongodb lock store [Lock] add mongodb store Jun 6, 2019
@nicolas-grekas nicolas-grekas changed the base branch from 4.3 to 4.4 June 6, 2019 07:08
@nicolas-grekas nicolas-grekas changed the base branch from 4.4 to 4.3 June 6, 2019 07:09
@kralos kralos force-pushed the 27345-mongodb-lock-store-43 branch from 55526dc to 0e77ec6 Compare June 6, 2019 09:26
@kralos
Copy link
Author

kralos commented Jun 6, 2019

rebased from 4.4

@kralos
Copy link
Author

kralos commented Jun 6, 2019

I've stripped it from symfony/symfony-docs 4.3 symfony/symfony-docs#11686

Will add back to symfony/symfony-docs 4.4 once the above PR is merged and upstreamed to symfony/symfony-docs 4.4

Added in symfony/symfony-docs#11735

javiereguiluz added a commit to symfony/symfony-docs that referenced this pull request Jun 12, 2019
This PR was merged into the 4.3 branch.

Discussion
----------

[Lock] mongodb store removed from symfony 4.3

MongoDbStore wasn't actually added in symfony 4.3, it will be added in 4.4.

See symfony/symfony#31889

Commits
-------

75ad033 #27345 Removed Lock MongoDbStore, it will be added in symfony 4.4

/**
* MongoDbStore is a StoreInterface implementation using MongoDB as a storage
* engine.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about adding that it requires an external lib?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pattern public static function isSupported(): bool has been added to MongoDbStore as well as @requires extension mongodb on the class. I've also moved the CAUTION: phpdoc up to the class to be inline with other stores.

@kralos kralos force-pushed the 27345-mongodb-lock-store-43 branch 2 times, most recently from 753d398 to f3f8568 Compare July 4, 2019 00:17
@kralos
Copy link
Author

kralos commented Jul 4, 2019

I'm looking into the failing test later today

Copy link
Contributor

@Simperfit Simperfit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice feature !
Status: Needs Work

*
* @internal
*/
public static function isSupported(): bool
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

imho this is not needed

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did this because some of Jérémy's stores do it. I thought it might be useful later if a bundle extension needs to check support to auto default/configure something supported, but maybe we could add an interface for this later to all stores if we ever did something like that. Also thinking about it more... you could never auto configure a MongoDbStore since the database name option is mandatory. So I have removed static::isSupported().

*/
public function __construct(Client $mongo, array $options, float $initialTtl = 300.0)
{
if (!static::isSupported()) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can directly check if it is supported here.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

*
* @see https://docs.mongodb.com/manual/applications/replication/
*/
public function __construct(Client $mongo, array $options, float $initialTtl = 300.0)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you should not type-hint the mongodb client, what happens if he does not exist ?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then the developer wouldn't be able to construct a MongoDB\Client. If they pass something that isn't a MongoDB\Client they would get a TypeError:

Uncaught TypeError: Argument 1 passed to Symfony\Component\Lock\Store\MongoDbStore::__construct() must be an instance of MongoDB\Client, x given

This is what I would want to happen

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if you don't have MongoDB ?

Copy link
Author

@kralos kralos Jul 5, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you don't have mongodb/mongodb you can't construct a MongoDB\Client

if you don't have ext-mongodb you can't composer require mongodb/mongodb

if you somehow have installed mongodb/mongodb and don't have ext-mongodb you will fail to construct a MongoDB\Client (this is mongodb/mongodb's responsibility):

<?php

require __DIR__.'/vendor/autoload.php';

$mongoClient = new MongoDB\Client('mongodb://127.0.0.1');
$store = new Symfony\Component\Lock\Store\MongoDbStore($mongoClient, [
    'database' => 'test',
]);
$ docker run --rm -it -v $PWD:$PWD -w $PWD php php ./test.php

Fatal error: Uncaught Error: Class 'MongoDB\Driver\Manager' not found in /home/kralos/src/symfony/vendor/mongodb/mongodb/src/Client.php:87
Stack trace:
#0 /home/kralos/src/symfony/test.php(5): MongoDB\Client->__construct('mongodb://127.0...')
#1 {main}
  thrown in /home/kralos/src/symfony/vendor/mongodb/mongodb/src/Client.php on line 87

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MongoDB\Driver\Manager is provided by ext-mongodb

MongoDB\Client is provided by mongodb/mongodb

Symfony\Component\Lock\Store\MongoDbStore depends on mongodb/mongodb

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you not type-hint MongoDB\Client for that reason. maybe i'm wrong cc @jderusse

Copy link
Author

@kralos kralos Jul 6, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed the if (!\extension_loaded('mongodb')) { check in the constructor since it's impossible to hit it. You can't pass a MongoDB\Client without mongodb/mongodb which you can't have without ext-mongodb.

We still have in the class docblock @requires extension mongodb


if (!isset($this->options['database'])) {
throw new InvalidArgumentException(
'You must provide the "database" option for MongoDBStore'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you should inline this

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done


if ($this->options['gcProbablity'] < 0.0 || $this->options['gcProbablity'] > 1.0) {
throw new InvalidArgumentException(sprintf(
'"%s" gcProbablity must be a float from 0.0 to 1.0, "%f" given.',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you should inline this

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

src/Symfony/Component/Lock/Store/MongoDbStore.php Outdated Show resolved Hide resolved
src/Symfony/Component/Lock/Store/MongoDbStore.php Outdated Show resolved Hide resolved
src/Symfony/Component/Lock/Store/MongoDbStore.php Outdated Show resolved Hide resolved
src/Symfony/Component/Lock/Store/MongoDbStore.php Outdated Show resolved Hide resolved
src/Symfony/Component/Lock/Store/MongoDbStore.php Outdated Show resolved Hide resolved
@kralos kralos force-pushed the 27345-mongodb-lock-store-43 branch 2 times, most recently from 6df4973 to a1fe6d0 Compare August 22, 2019 04:22
@kralos
Copy link
Author

kralos commented Aug 22, 2019

git rebase origin/4.4 and squashed

@javiereguiluz
Copy link
Member

This one looks ready for Symfony 4.4/5.0 👍

|| (random_int(0, PHP_INT_MAX) / PHP_INT_MAX) <= $this->options['gcProbablity']
)
) {
if ($this->getDatabaseVersion() < '2.2') {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This works by accident, version_compare should be used for these

>>> '2.10' < '2.2'
=> true

instances: 2

But why are we adding support for legacy version anyway? 2.2 was EOL-ed in 2014. We are paying the price not just with added complexity but with extra query on each request as well.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I'd prefer not to support <2.2. removing support and documenting requirements

{
$this->getCollection()->deleteMany([ // filter
'expires_at' => [
'$lte' => $this->createDateTime(microtime(true)),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is inconsistent. IMHO parameter should either be non-optional, or not specified here

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed, I've opted for not optional. the objective of the function is to create a MongoDb compliant UTCDateTime to millisecond precision.

if (1 === \count($writeErrors)) {
$code = $writeErrors[0]->getCode();
} else {
$code = $e->getCode();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor thing, but else branch could be easily avoided here by specifying this as default. This is super cheap call

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated

@kralos
Copy link
Author

kralos commented Oct 15, 2019

@nicolas-grekas can you let me know if you're happy for this to be merged? I'll rebase/squash etc if everyone is happy. The appveyor CI failure is not Lock component by the way...

Testing src\Symfony/Component/HttpClient

1) Symfony\Component\HttpClient\Tests\CurlHttpClientTest::testToStream
Symfony\Component\HttpClient\Exception\TransportException: Couldn't connect to server for "http://localhost:8057/".

Copy link
Member

@jderusse jderusse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you update the StoreFactory to add the MongoDB\Client and mongodb:// in the list of managed DSN.
Need to update the CHANGELOG.md

@kralos
Copy link
Author

kralos commented Nov 5, 2019

Could you update the StoreFactory to add the MongoDB\Client and mongodb:// in the list of managed DSN.
Need to update the CHANGELOG.md

In order to add MongoDbStore support to StoreFactory the MongoDbStore::__construct needs to be able to construct without $options.

I have therefore added support for passing a MongoDB\Collection to MongoDbStore::__construct.

I have also added DSN support to MongoDbStore::__construct.

Neither Client nor DSN alone will be enough to construct a MongoDbStore as the database needs to be specified in $options therefore StoreFactory only supports MongoDB\Collection.

CHANGELOG.md has been updated.

CI is currently passing (the errors from travis are related to components other then Component\Lock). All Component\Lock tests are passing.

I've left the commit history for review purposes, if we are happy with the current iteration let me know and I can squash (if required).

@@ -62,7 +65,6 @@ public static function createStore($connection)

case 0 === strpos($connection, 'flock://'):
return new FlockStore(substr($connection, 8));

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revert

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed, something funny happened with my rebase

}

if ($this->options['gcProbablity'] < 0.0 || $this->options['gcProbablity'] > 1.0) {
throw new InvalidArgumentException(sprintf('"%s" gcProbablity must be a float from 0.0 to 1.0, "%f" given.', __METHOD__, $this->options['gcProbablity']));
throw new InvalidArgumentException(sprintf('%s() gcProbablity must be a float from 0.0 to 1.0, "%f" given.', __METHOD__, $this->options['gcProbablity']));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove " in "%f" given too (for consistency)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated

}

if (null === $this->collection && !isset($this->options['database'])) {
throw new InvalidArgumentException(sprintf('%s() requires the "database" option when constructing with a %s', __METHOD__, \is_object($mongo) ? \get_class($mongo) : \gettype($mongo)));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

database and collection names could be defined in the DSN (similar to PDO). dsn = mongodb://[username:password@]host1[:port1][,host2[:port2:],...]/[database/][collection] ?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The database you see there is the authentication database, not the database to use for data

Copy link
Member

@jderusse jderusse Nov 5, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't it the purpose of the querystring parameter authSource? https://symfony.com/doc/master/bundles/DoctrineMongoDBBundle/installation.html#authentication

anyway, if the path component is a standard to declare the auth Database, we can use custom queryString parameters like mongodb://localhost/?lockDatabase=foo&lockCollection=bar

If we can not create the Store from a DSN, developpers using the FrameworkBundle won't be able to use it from the configuration framework.lock = '%env(LOCK_DNS)% (see symfony/symfony-docs#12523)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't it the purpose of the querystring parameter authSource?

Actually I just tested this (mongodb 4.2.0) and if both /path and ?authSource are specified /path means application database and ?authSource means authentication database. We are still stuck for getting the collection name.

we can use custom queryString parameters

Seems to be the only way, do you have any objection to us taking ?collection=bar? I feel like if we ever had a conflict it would be because MongoDB is using this param for the exact same purpose.

I'm going to enforce when $mongo is a:

  • MongoDB\Collection:
    • $options['database'] is ignored
    • $options['collection'] is ignored
  • MongoDB\Client:
    • $options['database'] is mandatory
    • $options['collection'] is mandatory
  • MongoDB Connection URI string:
    • /path is used otherwise $options['database'], at least one is mandatory
    • ?collection= is used otherwise $options['collection'], at least one is mandatory

Comment on lines 120 to 126
$database = parse_url($mongo, PHP_URL_PATH);
if (null !== $database) {
$this->options['database'] = $database;
}
$query = parse_url($mongo, PHP_URL_QUERY);
$queryParams = [];
parse_str($query ?? '', $queryParams);
$collection = $queryParams['collection'] ?? null;
if (null !== $collection) {
$this->options['collection'] = $collection;
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's about something similare to Messenger PDO

if (false === $parsedUrl = parse_url($dsn)) {
    throw new InvalidArgumentException(sprintf('The given Mongodb DSN "%s" is invalid.', $dsn));
}

$query = [];
if (isset($parsedUrl['query'])) {
    parse_str($parsedUrl['query'], $query);
}
$this->option = $option + $query;
$this->options['database'] = $this->options['database'] ?? ltrim($parsedUrl['path'] ?? '', '/') ?: null;

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You prefer $options to take precedence over /path?collection=?

<?php

declare(strict_types=1);

$mongo = 'mongodb://127.0.0.1/uri-db?collection=uri-col';
$options = [
    'database' => 'option-db',
    'collection' => 'option-col',
];

if (false === $parsedUrl = parse_url($mongo)) {
    throw new \InvalidArgumentException(sprintf('The given MongoDB Connection URI "%s" is invalid.', $mongo));
}
$query = [];
if (isset($parsedUrl['query'])) {
    parse_str($parsedUrl['query'], $query);
}
$options += $query;
$options['database'] = $options['database'] ?? ltrim($parsedUrl['path'] ?? '', '/') ?: null;

print_r($options);
/* Array
(
    [database] => option-db
    [collection] => option-col
) */

Copy link
Author

@kralos kralos Nov 5, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've implemented this but without polluting $this->options with querystring params other than collection.

if (false === $parsedUrl = parse_url($mongo)) {
    throw new InvalidArgumentException(sprintf('The given MongoDB Connection URI "%s" is invalid.', $mongo));
}
$query = [];
if (isset($parsedUrl['query'])) {
    parse_str($parsedUrl['query'], $query);
}
$this->options['collection'] = $this->options['collection'] ?? $query['collection'] ?? null;
$this->options['database'] = $this->options['database'] ?? ltrim($parsedUrl['path'] ?? '', '/') ?: null;

@kralos
Copy link
Author

kralos commented Nov 11, 2019

@jderusse happy with current implementation? want me to squash?

Copy link
Member

@jderusse jderusse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @kralos

@kralos
Copy link
Author

kralos commented Nov 20, 2019

@nicolas-grekas can we get this merged? do you need a squash first? I don't want to miss v4.4

@fabpot fabpot changed the base branch from 4.4 to master December 11, 2019 00:40
@fabpot fabpot force-pushed the 27345-mongodb-lock-store-43 branch from af7825d to a6bfa59 Compare December 11, 2019 00:40
@fabpot
Copy link
Member

fabpot commented Dec 11, 2019

Thank you @kralos.

@fabpot fabpot closed this in e8ecfe3 Dec 11, 2019
@fabpot fabpot merged commit a6bfa59 into symfony:master Dec 11, 2019
@nicolas-grekas nicolas-grekas modified the milestones: next, 5.1 May 4, 2020
@fabpot fabpot mentioned this pull request May 5, 2020
@kralos kralos deleted the 27345-mongodb-lock-store-43 branch August 17, 2020 23:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants
Morty Proxy This is a proxified and sanitized view of the page, visit original site.