[Intl] New Intl API #9206

webmozart · Oct 3, 2013

Q	A
Bug fix?	yes
New feature?	no
BC breaks?	yes
Deprecations?	yes
Tests pass?	yes
Fixed tickets	#5802
License	MIT
Doc PR	TODO

The following description is outdated. This PR has turned into an overhaul of the Intl API to basically rebuild ICU4J/ICU4C (the official ICU APIs) under PHP. Once I'm done, I'll update the text below.

This PR contains a new, extensive integration test suite for the Intl component that check whether the methods of the resource bundles work correctly and consistently in practice. When I wrote this test suite, I discovered numerous bugs that have been fixed in this PR.

I had to do two major BC breaks:

I changed the Intl::get*Bundle()->get*() methods to throw exceptions instead of just returning null on errors. Otherwise it would have been impossible to fix the existing bugs. Consequently, anybody using these methods directly with invalid languages, locales etc. will now experience an uncaught exception. People using the Form layer only should not be affected.
I added a new method getLocaleAliases() to LocaleBundleInterface.

I also did two BC breaks in our internal code, which shouldn't affect our users:

I changed the default value of $fallback in the protected AbstractBundle::readEntry() method from false to true. The method is just a proxy to StructuredBundleReader::readEntry(), which also has the default true for this parameter. We had numerous bugs in the past that were caused by this parameter being false by default. This BC break obviously only affects people who extended AbstractBundle (I'd be surprised if anyone did).
I removed the internal interfaces CompilationContextInterface and StubbingContextInterface.

The fixed bugs had mainly to do with locale aliases. When locales get renamed in ICU, the old locale remains as an alias for (think: pointer to) the new locale. For example, the locale "mo" is an alias for "ro_MD". That locale falls back to "ro" if entries can't be found. Previously, the fallback from "mo" to "ro" did not work since aliases were ignored.

Additionally, the .res files for aliases (i.e. pointers to the new actual locales) were not generated for the LocaleBundle, so locale aliases there could not be accessed at all (e.g. Intl::getLocaleBundle()->getLocaleName('en', 'mo') -> bang).

The PR also contains improvements in efficiency. When accessing the Intl::get*Bundle()->getLocales(), the bundles won't scan the filesystem for available locales anymore (slow), but read a cached entry from a pregenerated misc.res file (speedy gonzales).

The new test suite (see the *ConsistencyTest) is a very nice documentation of the current state of the ICU data, so go and check it out. Some tests are very slow, so I created a new group icu-consistency for them, which is excluded by default.

TODO:

Test and upgrade symfony/Icu 1.2.x
Test and upgrade symfony/Icu 1.1.x
Test and upgrade symfony/Icu 1.0.x
Make sure no BC breaks leak through to the Form component
Update Intl component documentation

stloyd · Oct 3, 2013

src/Symfony/Component/Intl/ResourceBundle/Compiler/GenrbBundleCompiler.php

+     */
+    public function __construct($genrb = 'genrb', $envVars = '')
+    {
+        exec('which ' . $genrb, $output, $status);


Why not using symfony/process?

Good question :) but out of scope of this PR (this class was just renamed, the above code remained the same).

…never an invalid locale is given

…-existing languages, currencies, etc. are accessed

…n a generic misc.res file to improve reading performance

…ate locale names when fallback (en_GB->en) is possible

…formation

…king for fallback locales

…continues at fallback locale (unless disabled)

…file to enable proper locale fallback

…ss through numeric strings without converting them to integers

…y::forNumericCode()

pvgnd · Dec 16, 2013

What is the status of this overhaul ?
Is there a workaround on Symfony 2.3 or Symfony 2.4 to be able to use Intl ?

stof · Jan 24, 2014

@bschussek can you comment about the status ?

And it will need to be rebased

webmozart · Jan 24, 2014

@stof Currently on hold due to other, more important, issues. I'd however be interested in feedback about the new API. I designed the API to reflect ICU4Js API (more or less), i.e. classes with static methods, such as:

$currencies = Currency::getCurrencies();
$names = Currency::getNames();

if (Language::exists('de_DE')) // etc.

Now we all know that static methods are boo. However, these methods are used to access "global system information", so I'm not sure whether it makes sense to always pass objects around for accessing this global information.

Opinions?

/cc @pierrejoye

stof · Jan 24, 2014

src/Symfony/Component/Intl/Currency.php

+     *
+     * @throws InvalidArgumentException If the currency or the locale is invalid
+     *
+     * @api


All the @api tags should be removed IMO. We should not decide to enforce full BC on them before they are even final.

stof · Jan 24, 2014

@bschussek Teh issue I see with using a static API is that it means that code using it cannot be tested in isolation anymore, because you cannot mock the static calls being done in the middle.

fabpot · Mar 3, 2014

@webmozart If we want this PR to be included in 2.5, we should finish it asap.

pierrejoye · Mar 3, 2014

@stof what is the problem actually? I do not see much point to create an instance for only checking the existence of a locale (or other ICU related data).

…cessibleResourceBundle (webmozart) This PR was merged into the 2.3 branch. Discussion ---------- [Intl] Removed non-working $fallback argument from ArrayAccessibleResourceBundle | Q | A | ------------- | --- | Bug fix? | yes | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | - | License | MIT | Doc PR | - The code in question didn't actually work. This was extracted from #9206. Commits ------- 5feda5e [Intl] Removed non-working $fallback argument from ArrayAccessibleResourceBundle

…webmozart) This PR was merged into the 2.3 branch. Discussion ---------- [Intl] Added exception handler to command line scripts | Q | A | ------------- | --- | Bug fix? | no | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | - | License | MIT | Doc PR | - Extracted from #9206. Commits ------- 9052efc [Intl] Added exception handler to command line scripts

This PR was merged into the 2.3 branch. Discussion ---------- [Intl] Updated icu.ini up to ICU 53 | Q | A | ------------- | --- | Bug fix? | no | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | - | License | MIT | Doc PR | - Extracted from #9206. Commits ------- 260e2fe [Intl] Updated icu.ini up to ICU 53

This PR was merged into the 2.3 branch. Discussion ---------- [Intl] Fixed a few bugs in TextBundleWriter | Q | A | ------------- | --- | Bug fix? | yes | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | - | License | MIT | Doc PR | - See the included test cases for more information. This code was extracted from #9206. Commits ------- 7b4a35a [Intl] Fixed a few bugs in TextBundleWriter

This PR was merged into the 2.3 branch. Discussion ---------- [Intl] Improved bundle reader implementations | Q | A | ------------- | --- | Bug fix? | yes | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | - | License | MIT | Doc PR | - This PR extracts bundle reader improvements from #9206. The code is internal and used for resource bundle generation only, so I did not care about BC too much. Commits ------- c3cce5c [Intl] Improved bundle reader implementations

fabpot · Dec 24, 2014

@webmozart Is it still relevant?

stloyd reviewed Oct 3, 2013
View reviewed changes

webmozart added 23 commits October 4, 2013 10:22

[Intl] Made TextBundleWriter able to write "nofallback" resource bundles

77cbd73

[Intl] Made TextBundleWriter able to write traversables

ba12cd4

[Intl] The various Intl methods now throw a NoSuchLocaleException whe…

e14dd3a

…never an invalid locale is given

[Intl] Changed Intl methods to throw NoSuchEntriesExceptions when non…

953a29c

…-existing languages, currencies, etc. are accessed

[Intl] The available locales of each resource bundle are now stored i…

0b5408b

…n a generic misc.res file to improve reading performance

[Intl] Fixed LocaleScanner to actually remove blacklisted locales

2ad4d32

[Intl] Improved LocaleBundleTransformationRule to not generate duplic…

d6805b4

…ate locale names when fallback (en_GB->en) is possible

[Intl] Added LocaleScannerTest

958a4dc

[Intl] Removed unneeded method BundleReaderInterface::getLocales()

b3bd0aa

[Intl] Fixed CS

15419bb

[Intl] Fixed currency transformation rule to clean up after the trans…

fbe2853

…formation

[Intl] Fixed StructuredBundleReader to follow locale aliases when loo…

95de438

…king for fallback locales

[Intl] Added exception handler to data generation scripts

5d14465

[Intl] Decoupled transformation rules from Intl class

72398a3

[Intl] Improved test coverage of BinaryBundleReader

2bb2508

[Intl] Improved exception formatting in transformation scripts

9d202dc

[Intl] StructuredBundleReader now catches NoSuchLocaleExceptions and …

fec3933

…continues at fallback locale (unless disabled)

[Intl] Changed LocaleBundleTransformationRule to generate a root.res …

945cfef

…file to enable proper locale fallback

[Intl] Improved error messages in BinaryBundleReader

d19e20a

[Intl] Changed AbstractEntry::readEntry() to fallback by default

127d9c0

[Intl] Added period to exception message in RecursiveArrayAccess

921f9fb

[Intl] Renamed BundleCompiler to GenrbBundleCompiler

6251c7e

[Intl] Added missing entries to the UPGRADE file

1b3d920

webmozart added 2 commits October 15, 2013 14:52

[Intl] Fixed RecursiveArrayAccess/ArrayAccessibleResourceBundle to pa…

05e4761

…ss through numeric strings without converting them to integers

[Intl] Added Currency::exists(), Currency::canonicalize() and Currenc…

1cc7c7a

…y::forNumericCode()

This was referenced Oct 16, 2013

en_GB locale and missing currency selection options #9262

Closed

[Intl] Support ISO 3166-1 country codes #8492

Closed

stof reviewed Jan 24, 2014
View reviewed changes

webmozart added the Feature label Aug 5, 2014

webmozart mentioned this pull request Sep 8, 2014

[intl] Support all locales by using CLDR data directly #11737

Closed

This was referenced Sep 9, 2014

Determine the file format to use for the shipped ICU data #11885

Closed

Flag deprecated currencies in the bundled ICU data, filter them out by default #11886

Closed

Use classes and interfaces to represent the shipped ICU data #11887

Closed

This was referenced Sep 11, 2014

[Intl] Improved bundle reader implementations #11907

Merged

[Intl] Removed non-working $fallback argument from ArrayAccessibleResourceBundle #11905

Merged

[Intl] Fixed a few bugs in TextBundleWriter #11906

Merged

This was referenced Sep 12, 2014

[Intl] Updated icu.ini up to ICU 53 #11913

Merged

[Intl] Added exception handler to command line scripts #11914

Merged

fabpot closed this Feb 5, 2015

webmozart mentioned this pull request Mar 30, 2016

Simplify access to CLDR/ICU data #18368

Closed

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[Intl] New Intl API #9206

[Intl] New Intl API #9206

Uh oh!

webmozart commented Oct 3, 2013

Uh oh!

stloyd Oct 3, 2013

Uh oh!

webmozart Oct 3, 2013

Uh oh!

pvgnd commented Dec 16, 2013

Uh oh!

stof commented Jan 24, 2014

Uh oh!

webmozart commented Jan 24, 2014

Uh oh!

stof Jan 24, 2014

Uh oh!

stof commented Jan 24, 2014

Uh oh!

fabpot commented Mar 3, 2014

Uh oh!

pierrejoye commented Mar 3, 2014

Uh oh!

fabpot commented Dec 24, 2014

Uh oh!

Uh oh!

Search code, repositories, users, issues, pull requests...

Uh oh!

[Intl] New Intl API #9206

[Intl] New Intl API #9206

Uh oh!

Conversation

webmozart commented Oct 3, 2013

Uh oh!

stloyd Oct 3, 2013

Choose a reason for hiding this comment

Uh oh!

webmozart Oct 3, 2013

Choose a reason for hiding this comment

Uh oh!

pvgnd commented Dec 16, 2013

Uh oh!

stof commented Jan 24, 2014

Uh oh!

webmozart commented Jan 24, 2014

Uh oh!

stof Jan 24, 2014

Choose a reason for hiding this comment

Uh oh!

stof commented Jan 24, 2014

Uh oh!

fabpot commented Mar 3, 2014

Uh oh!

pierrejoye commented Mar 3, 2014

Uh oh!

fabpot commented Dec 24, 2014

Uh oh!

Uh oh!