Merged CSS/JS files require pub/static/_cache writeable, contrary to documentation

[scottsb]

Preconditions

1. Have Magento installed with two filesystem owners as documented, with all code files (source and CLI-generated) read-only for web server user. This includes the full pub/static directory.
2. Turn on the merging of CSS or JS files.

Steps to reproduce

Clear static file cache and load home page.

Expected result

No exception should be generated.

Actual result

Exception is logged:

[2018-01-16 16:18:56] main.CRITICAL: Directory "/var/www/html/___/pub/static/_cache/merged" cannot be created Warning!mkdir(): Permission denied {"exception":"[object] (Magento\\Framework\\Exception\\FileSystemException(code: 0): Directory \"/var/www/html/___/pub/static/_cache/merged\" cannot be created Warning!mkdir(): Permission denied at /var/www/html/___/vendor/magento/framework/Filesystem/Driver/File.php:225)"} []

Commentary

This could arguably be seen as a documentation issue in which case I can report it in the devdocs repo. However, as I understand it, it seems that the documentation does describe the intended state, meaning the code should be changed instead. I'm not sure if it's possible, but if it were, ideally the merged files would be generated by static-content:deploy as alluded to in #9542. This would allow multiple servers in a cluster to be certain to have the merged files present in response to requests without the directory being shared among the nodes.

[davidalger]

+1 here guys! Nothing should be written to pub/static when in production mode, and a setup:static-content:deploy should write out merged files to static/_cache. This causes problems in load-balanced environments where a snapshot of a pre-compiled build is deployed.

[dan-ding]

bump
sort of related: #9542

[ghost]

Hi @scottsb i cannot reproduce you scenario, which version of magento you've tested ?

[graemel]

I see this behaviour in 2.1.

[davidalger]

This is a problem with 2.2.x as well. It starts with the files in static/_cache/merged not being written during static content deploy, and then the issue is compounded (on either a read-only file system or ) when Magento attempts to write merged JS / CSS files to disk during a web request.

[ghost]

@scottsb , we are closing this issue due to inactivity. If you'd like to update it, please reopen the issue.

[ishakhsuvarov]

@davidalger @scottsb Sorry for the confusion. This ticket was closed mistakenly. We will take a look at it again and let you know.

[ghost]

Hi @scottsb , thank you for your report.
We've acknowledged the issue and added to our backlog.

[kirkmadera]

We also build separate from deploy and ran into this issue. We are expecting pub/static to actually be static and read only. We have also made the workaround to make the cache dir writable. Would be cleaner if these merged files generated as part of static content deploy.

[scrivvles]

We are experiencing this same issue on our production Magento 2.2.6 server.

[gwharton]

Looking forward to an update on this. Running 2.2.7 on a cluster and had to disable merging of css, not because of non writeable /pub/static/_cache folder, as I have allowed it, but individual servers don't seem to be triggering the generation of the cache files. I seem to get many different min.css files generated, with different hash's, but not always the one that the browser is requesting. I'd love to see these generated in the static:content:deploy phase.

[kirkmadera]

I think I understand the reasoning behind this. It's because it has to create the merged file for every possible frontend page, which would be tough to generate statically, especially without a full running app present to crawl.

I honestly think documenting in Magento docs that pub/static/_cache needs to be writable and shared across all frontend servers would be enough.

[gwharton]

Why shared. Does magento keep track of which ones have been generated, and in effect only generates it on one of the cluster nodes and 404s it from the rest of the cluster as all cluster nodes think its already generated, not knowing it is only present on one node.

That would certainly explain what i am seeing.

[scottsb]

I was mulling over this last night and was thinking of how this could be solved. This is rough and may not be workable, but it might be a start:

1. Change the generation of the files to happen on-demand rather than as part of the triggering page request. This would allow whatever server receives the request to generate the file locally, even if it's a different server than triggered it originally. This file could then be cached using the general cache config (such as in Redis or at least in the expected var cache directory if using disk). In order for this to work the URL either has to include sufficient data or reference some cache key (which would be stored during the initial page request).

2. The first step alone would avoid the present filesystem permissions issue but at the significant cost of requiring a Magento boot to serve the merged files, even if actual processing is quick once cached. So the next step would be changing the default Varnish VCL config to include these merged static files. (Currently all non-HTML assets are excluded by default.) This only solves the problem for those using Varnish, but that's recommended config for prod in any case.

A different, more exotic option would be to keep the merging in the initial page request but to push the data into the full page cache inline. This wouldn't be hard for Magento's built-in FPC, but for Varnish it would require us to push data out to it rather than waiting on it to cache while proxying. This is possible but is a rather non-standard approach.

[scottsb]

@gwharton Yes, Magento merges the file as part of the page request and saves it the local disk. It's expected that the file is then served directly by the web server as a static file from then on (it never routes through PHP/Magento). But if the merged asset request goes to a different server in the cluster, then it won't be present locally if the directory isn't shared.

[gwharton]

@scottsb Yeah, I just deployed a shared writable pub/static/_cache folder across my cluster and re-enabled merging of css files and all seems ok.

[geerlingguy]

I've also had to mount pub/static/_cache as a shared folder in my Kubernetes environment since any time more than one Pod (container) is running, the file will be generated on one, but not another, and that kinda messes things up—plus I have to make sure it's writeable (which it wasn't earlier).

[Stoyvo]

Magento 2.4.4 - Issue still present.

CSS/JS should never be generated on web-request, these are static files, they do not change, having this part of the web request is a critical design flaw. It severely affects containerized applications and forces infrastructure work-arounds.

Merged JS / CSS should be built in static content deploy and not generated by frontend request.

[mattgrul]

With the current "PR Created" label being added I'm hoping we are close to a possible solution for this issue. However I can't seem to find any corresponding PR, does anyone have a link?

[bsp-cory]

@mattgrul if there is a PR, I would LOVE to see it. This has been driving me nuts for a couple of years now, and it would really help my quality of life if I didn't have to keep selectively changing permissions on these static files.

[kanevbg]

I do not see PR either. Would be very pleased to see such too. Same opinion as of @Stoyvo , it's really unacceptable and pretty strange such a flaw to exists so much time on such advanced system. Someone should add more labels, like "platform health"

[erfanimani]

For those wondering, the PR is likely created on Adobe's internal VCS, so you know it's being worked on.

[engcom-Hotel]

Hello @mattgrul @bsp-cory @kanevbg @erfanimani,

The label Progress: PR Created was added due to the wrong status made in JIRA. It was corrected. Currently, there is no PR created for this issue.

We have tried to reproduce the issue in the latest development of Magento ie 2.4-develop but the issue is not reproducible. It might the issue is fixed and merged with the development branch.

The steps we have followed are as follows:

1. Followed the below documentation to create 2 users one is for the web server and the other one is for CLI:
   https://experienceleague.adobe.com/docs/commerce-operations/configuration-guide/deployment/file-system-permissions.html
   https://experienceleague.adobe.com/docs/commerce-operations/installation-guide/prerequisites/file-system/configure-permissions.html?lang=en#about-the-shared-group
2. Install Magento in the folder.
3. Change with production mode: bin/magento deploy:mode:set production
4. Ran static content deploy command: bin/magento setup:static-content:deploy
5. Try to access the Homepage.

We are able to access the homepage without any issues. Please let us know in case we have missed anything.

Thanks

[dooblem]

Did you enable merge js/css as mentionned in the issue ?

[mattgrul]

@engcom-Hotel It seems like you are not fully understanding the issue. If you read through the entire thread you will see this is an issue with load-balanced environments. The contents of static/_cache should be written during the compile stage, not, during the first initial web request.

[Stoyvo]

@engcom-Hotel

The appropriate way to test this will be:

1. Magento 2 is installed, functional, visible on frontend.
2. Change to Production mode php bin/magento deploy:mode:set production
3. Enable "merged css": php bin/magento config:set dev/css/merge_css_files 1
4. Flush Cache just incase php bin/magento cache:flush
5. On server, verify no merged css files exist in ls pub/static/_cache/merged/
6. Run static content deploy: php bin/magento setup:static-content:deploy
7. Verify merged css is visible: ls pub/static/_cache/merged/
8. Load frontend by loading Magento 2 in web browser
9. Check again for merged css ls pub/static/_cache/merged/

IF merged css is missing in Step 7 AND missing in Step 9, then merged CSS is not working correctly, please retry the steps.

IF merged CSS is missing in Step 7, but is visible in Step 9, then you have correctly recreated the issue. Static content deploy did not generate the file as it's expected to.

[engcom-Hotel]

Hello @Stoyvo,

Thanks for the detailed steps. We have followed the above steps as well. and for is the issue is still not reproducible. I have to add some points here to notice:

1. We have used the shared group as mentioned in this document..
2. We already have a www-data user in the www-data group, so we create one more user in the same group with CLI/SSH access.

Please let us know if we have missed anything.

Thanks

[github-jira-sync-bot]

❌ Cannot export the issue. This GitHub issue is already linked to Jira issue(s): https://jira.corp.adobe.com/browse/AC-1005

[erfanimani]

Related issue: #29172

[p24-max]

I'm also facing this issue.
pub/static/_cache/merged/ files are not generated with command bin/magento setup:static-content:deploy.
The first request(s) are extremly slow due to this.

Only "workaround" what I currently found is disabling the CSS merging, but this is not a solution.
Does someone found a better workaround to keep merging enabled?

--
bump @engcom-Hotel

[Stoyvo]

@p24-max

Does someone found a better workaround to keep merging enabled?

I use GCP containers with a "gfuse" mount to a public storage bucket where first request builds it, but it is used on every request going forward. This adds overhead to my container start time, but it's faster than rebuilding merged css.

[gwharton]

7 years on, and we're all still generating static content triggered by the first few browsers of our sites.