Cannot find module 'postcss' in source install update when compiling assets

Summary

While trying to upgrade our test instance of GitLab from 14.6.3 to 14.8.1 via the scripted workflow we have used for that upgrade many times in the past (script based on upgrade instructions, I verified that the instructions still say the same) I encountered a problem in the asset compilation step. it displays an error "Cannot find module 'postcss'. It also still exits with a successful exit code despite that error.

Steps to reproduce

  • Source install according to install guide
  • Update according to update guide for source installs
  • Encounter missing module error in asset compilation step

What is the current bug behavior?

Missing module 'postcss'

`gitlab:assets:fix_urls` finished in 0.522036264 seconds
internal/modules/cjs/loader.js:905
  throw err;
  ^

Error: Cannot find module 'postcss'
Require stack:
- /var/git/gitlabhq/scripts/frontend/check_page_bundle_mixins_css_for_sideeffects.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:902:15)
    at Function.Module._load (internal/modules/cjs/loader.js:746:27)
    at Module.require (internal/modules/cjs/loader.js:974:19)
    at require (internal/modules/cjs/helpers.js:101:18)
    at Object.<anonymous> (/var/git/gitlabhq/scripts/frontend/check_page_bundle_mixins_css_for_sideeffects.js:13:13)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:12)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:75:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/var/git/gitlabhq/scripts/frontend/check_page_bundle_mixins_css_for_sideeffects.js'
  ]
}
`gitlab:assets:check_page_bundle_mixins_css_for_sideeffects` finished in 0.134506489 seconds

What is the expected correct behavior?

All required modules should be installed.

Output of checks

The checks do not display any errors when running after this. Login fails on the 2FA step, the browser console mentions some scripts that were blocked because they are mime-type text/plain, this might be unrelated to this error.

Results of GitLab environment info

Expand for output related to GitLab environment info 


System information
System:         Debian 10
Current User:   git
Using RVM:      no
Ruby Version:   2.7.3p183
Gem Version:    3.1.6
Bundler Version:2.3.8
Rake Version:   13.0.6
Redis Version:  5.0.14
Sidekiq Version:6.4.0
Go Version:     go1.16.8 linux/amd64

GitLab information
Version:        14.8.1
Revision:       479d579719c
Directory:      /var/git/gitlabhq
DB Adapter:     PostgreSQL
DB Version:     13.6
URL:            https://gitlab-dev.saltation.de
HTTP Clone URL: https://gitlab-dev.saltation.de/some-group/some-project.git
SSH Clone URL:  git@gitlab-dev.saltation.de:some-group/some-project.git
Using LDAP:     no
Using Omniauth: no

GitLab Shell
Version:        13.23.2
Repository storage paths:
- default:      /var/git/repositories
GitLab Shell path:              /var/git/gitlab-shell

Results of GitLab application Check

Expand for output related to the GitLab application check Note: the init script matches the one in the repository apart from the app_root setting. 
Checking GitLab subtasks ...


Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 13.23.2 ? ... OK (13.23.2)
Running /var/git/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Systemd unit files or init script exist? ... yes
Systemd unit files or init script up-to-date? ... no
Try fixing it:
Install the Service
For more information see:
doc/install/installation.md in section "Install the Service"
Please fix the error above and rerun the checks.
Projects have namespace: ...
7/3 ... yes
135/5 ... yes
[... lots more number pairs, all with yes]
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.3)
Git user has default SSH configuration? ... yes
Active users: ... 60
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

Presumably the issue is that

scripts/frontend/check_page_bundle_mixins_css_for_sideeffects.js

uses postcss but that module is not installed by an earlier step.