Use node headless executor for DAP

What does this MR do and why?

Use node headless executor for DAP

This change will allow us to centralise around single client for DAP

This change introduces a new way to run AI-powered workflows in GitLab. Instead of always downloading and running a binary executable file, the system can now optionally use a Node.js-based command-line tool when a feature flag is enabled.

The code adds several new environment variables that workflows can access, including project and namespace IDs, and a GitLab token.

The main improvement is that when the new feature flag is turned on, workflows will use a simpler npx command to run the GitLab Duo CLI tool, rather than the more complex process of downloading, extracting, and running a binary file. This makes the workflow execution more streamlined and potentially more reliable.

The change includes comprehensive tests to ensure both the old and new execution methods work correctly, and that setup scripts (if provided) are properly combined with the main workflow commands in both scenarios.

References

#576424

Screenshots or screen recordings

How to set up and validate locally

1. Setup DAP for remote flows https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/duo_agent_platform.md?ref_type=heads#environment-setup-2-remote-flows
2. Via rails console bin/rails c enable Feature.enable :ai_dap_use_headless_node_executor
3. Run DAP's flow session (eg: issue to MR, or fix pipeline)

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.