<!--Before adding details, please check if there is a more focused issue template available to address your needs.--> ## Details During the Game day https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21110, we have noticed that the nodes were taking too longer to launch and upon investigation we found the nodes were failing in the startup scripts and failures were ``` [ 748.190299] google_metadata_script_runner[2737]: startup-script: W: Tried to start delayed item http://security.ubuntu.com/ubuntu jammy-security InRelease, but failed [ 748.190380] google_metadata_script_runner[2737]: startup-script: W: Tried to start delayed item http://security.ubuntu.com/ubuntu jammy-security InRelease, but failed [ 748.190450] google_metadata_script_runner[2737]: startup-script: + apt-get install -y build-essential [ 748.190515] google_metadata_script_runner[2737]: startup-script: E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 1842 (apt-get) [ 748.190607] google_metadata_script_runner[2737]: startup-script: E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it? [ 748.190781] google_metadata_script_runner[2737]: Script "startup-script" failed with error: exit status 100 [ 748.190849] google_metadata_script_runner[2737]: Finished running startup scripts. ``` The script has a **race condition** between its own package installations and Ubuntu's automatic background processes (like `unattended-upgrades` that starts on boot). Investigating through the strart up scripts https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/blob/main/packer/scripts/bootstrap.sh?ref_type=heads In the `install_chef()` function around line 345-350: ``` apt-get update -y apt-get install -y gnupg ca-certificates ubuntu-keyring software-properties-common add-apt-repository universe add-apt-repository main apt-get install -y build-essential # <-- This is line 57.633331 in your error ``` This section runs **without waiting for apt locks** to be released, unlike the `enroll_ubuntu_advantage()` function which does it correctly. ### This might be happening due to 1. VM boots up 2. Ubuntu's `unattended-upgrades` service starts automatically (process 1839) 3. Your bootstrap script starts almost immediately 4. Both try to acquire the dpkg lock 5. Bootstrap script fails because `unattended-upgrades` got there first <!--If you know the team you need assistance from, please include the team label (uncomment and delete extra teams): /label ~"group::Networking & Incident Management" ~"NIM::Requests" /label ~"group::Runners Platform" /label ~"group::Observability"--> <!--Please do not edit the below-->