Add Workspace Suspension with Delayed Termination

What does this MR do and why?

Adds Workspace Suspension with Delayed Termination

For more details see:

• Issue: Rails: Add Workspace Suspension with Delayed Te... (#489078 - closed)
• Epic: Workspace Suspension with Delayed Termination (gitlab-org#14910 - closed)

Acceptance Criteria

Migrations

• Add max_active_hours_before_stop and max_stopped_hours_before_termination fields to the remote_development_agent_config table
• max_active_hours_before_stop will be of type smallint with default of 1.5 days. See Default values section in the epic for the reasoning behind this value.
• max_stopped_hours_before_terminations will be of type smallint with defaults of 1 month. See Default values section in the epic for the reasoning behind this value.
• Add constraints. See Validations section in the epic

Models

• Update the active record model for RemoteDevelopmentAgentConfig to include the needed validations. See Validations section in the epic)

Domain Logic

• Adds logic to agent_config ROP chain to set new fields from agent in workspaces_agent_configs table in database, with defaults read from Settings Module.
• Add new logic to reconciler ROP chain to conditionally change desired_state. See gitlab-org#14910 (closed) for details of logic
  • This should done before the WorkspacesToBeReturnedFinder step in the chain, as this forwards back to the agent all workspaces whose desired state has been updated more recently than last response to agent
• Change token lifetime to point to hardcoded MAX_HOURS_BEFORE_TERMINATION constant instead of max_hours_before_termination from client / workspaces table.
• Add integration testing of new behavior

Frontend

• Remove max_hours_before_termination field from being displayed in UI, but do not yet remove from component logic, frontend code, or GraphQL API. This will be removed later in follow-up issues in Remove old workspace termination logic (&15251 - closed).

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

No UI changes, the observable behavior is in the state change.

How to set up and validate locally

1. Add overrides for new settings to the agent config, setting them to 1 hour:

remote_development:
  max_active_hours_before_stop: 10
  max_stopped_hours_before_termination: 20

2. Create a new workspace and ensure it successfully starts and is in the Running state

3. In Rails console, validate these settings got populated to agent config record for the agent

w = RemoteDevelopment::Workspace.last;ermination;
p w.workspaces_agent_config.max_active_hours_before_stop;
p w.workspaces_agent_config.max_stopped_hours_before_termination;

4. Manually update the desired_state_updated_at on the workspace to more than 10 hours:

w.update!(desired_state_updated_at: w.desired_state_updated_at - 10.hours)

5. Force a full reconcile by restarting the agent

6. Verify that the workspace automatically stops.

7. Manually update the desired_state_updated_at on the workspace to more than 20 hours:

w.update!(created_at: w.created_at - 20.hour)

8. Force a full reconcile by restarting the agent

9. Verify that the workspace automatically terminates.