Rollout of duo_workflow_compress_checkpoint

Summary

This issue tracks the rollout of the duo_workflow_compress_checkpoint feature flag.

Feature

This feature flag controls whether the checkpoint payload between Duo Workflow Service and GitLab monolith is compressed using zlib and encoded using base64 to decrease payload size.

Owners

• Team: Agent Foundations
• Most appropriate slack channel to reach out to: #g_agent_foundations
• Best individual to reach out to: @a_akgun

Expectations

What are we expecting to happen?

Reduced payload size for checkpoint data transferred between Duo Workflow Service and GitLab monolith, leading to:

• Lower network bandwidth usage
• Faster data transfer
• Reduced memory footprint

What might happen if this goes wrong?

• Increased CPU usage due to compression/decompression overhead
• Potential compatibility issues if compression/decompression fails
• Possible performance degradation if compression overhead exceeds network transfer savings

What can we monitor to detect problems?

• Duo Workflow Service response times
• Error rates in Duo Workflow Service logs
• CPU utilization on Duo Workflow Service instances
• Network bandwidth metrics
• Checkpoint payload size metrics

Beta groups/projects

If applicable, any groups/projects that are happy to have this feature turned on early. For example, gitlab-org/gitlab or gitlab-org.

Roll Out Steps

• Enable on staging
• Test on staging with Duo Workflow
• Enable on GitLab.com for internal testing (gitlab-org group)
• Monitor for 24 hours
• Enable globally on GitLab.com
• Monitor for 48 hours
• Announce on #g_agent_foundations

Rollback Steps

• Disable the feature flag
• Monitor for recovery
• Investigate root cause