Hi,

A while ago I contributed on trying to reduce memory consumption in building the container through #17377. This impacted me because I use Platform.sh with Sylius, a heavy application in a limited RAM environment.

However, despite the above PR, a growth in our Sylius application meant that we recently started swapping on container compile and could not deploy to Platform.sh because any attempt at running a console command would be killed after 5 mins of swapping.

After some investigation it is simply token_get_all here.

In our application, checking memory consumption either side of this line on PHP 7.0.3:

21:33:54 pre token_get_all  49MB 
21:33:55 post token_get_all 341MB 

That's just under 300MB to tokenize the container, I'm not sure how much it is on php5.X but it's likely to be more.

The workaround I've used for our application was to override the Kernel methods and use php_strip_whitespace instead:

    /**
     * The following 2 methods workaround the enormous memory usage of token_get_all() which is used in
     * stripComments() below.
     *
     * Instead of calling token_get_all() we use php_strip_whitespace() which is actually more aggressive stripping
     * and results in a smaller cached container.
     *
     * However this can only be run against a file (not a string).
     * So we dump the container as normal without any stripping, then afterwards strip and re-save the container
     *
     * This resulted in a decrease in total memory consumption of the process that builds the container
     * from 350mb to <50mb in PHP7, which will allow it to be built on Platform.
     *
     * PW, 03/2016
     *
     * {@inheritdoc}
     */
    public function dumpContainer(ConfigCache $cache, ContainerBuilder $container, $class, $baseClass)
    {
        parent::dumpContainer($cache, $container, $class, $baseClass);

        if (!$this->debug) {
            $cache->write(php_strip_whitespace($cache->getPath()), $container->getResources());
        }
    }

    /**
     * {@inheritdoc}
     */
    public static function stripComments($source)
    {
        return $source;
    }

The memory consumption after dumping the container now was only an extra 10MB compared to 290MB with token_get_all. The time difference was fairly negligible, slightly slower though. The resulting container was 4.8MB compared to 5.6MB.

Clearly we are a heavier use-case here, probably one of the larger containers around, but as systems and platforms grow I could see more people in our situation.

The original comment on the stripComments() method is:

     * We don't use the PHP php_strip_whitespace() function
     * as we want the content to be readable and well-formatted.

My questions are:

• Seeing as stripComments is only being used when debug is false anyway (usually in the production environment) how important is it that this file is readable? I understand there could be a use-case for trying to debug and manually change this file - maybe?
• If it's really important to keep it readable then maybe there's a more memory-efficient way of stripping while keeping readability that doesn't consume such an enormous amount of memory on large strings?
• Does stripping really matter anyway since you would hope now that everyone using Symfony is using opcache and any savings here will be irrelevant in production anyway (I think...)?

If you want to keep the current approach then I would be more than happy to contribute a Cookbook chapter about this since it took me a while to work around this.