subhopam das closed issue #4: Explore: Trigger ingestion job via a Duo Workflow flow at GitLab AI Hackathon / Participants / kapillamba4

subhopam das
2026-03-20T01:42:57Z

Priority: 🟡 High (Enables self-service data population)

Currently the ingestion pipeline (ingestion-flow/) must be run manually via CLI. Explore how to trigger it as a Duo Workflow flow so users can say something like @oracle index this project and have the knowledge base populated automatically.

Options to explore

Option A: Ingestion as a separate flow

Create a new flow definition (e.g., flows/ingest-flow.yml)

Triggered by mentioning the Oracle with "index" or "ingest" keywords

The flow agent runs python3 -m ingestion --project-id $DUO_WORKFLOW_PROJECT_ID

Requires the ingestion dependencies (python-gitlab, sentence-transformers, psycopg) to be available in the Docker image

Option B: Ingestion via API endpoint

Add a POST /ingest endpoint to the query service

The Oracle flow agent calls it via curl when it detects no data exists

Ingestion runs server-side on the query service host (where DB access is direct)

More reliable since dependencies are already installed

Option C: GitLab CI/CD scheduled pipeline

Add a scheduled pipeline job in .gitlab-ci.yml that runs ingestion nightly

Simplest approach but not on-demand

Could be combined with Option B for both scheduled and on-demand

Key considerations

The ingestion job needs a GitLab API token with read access to the target project

The agent-config.yml image needs ingestion dependencies if using Option A

Database credentials must be available in the execution environment

Rate limiting: ingestion of large projects takes time (67 min for 10K MRs per REQUIREMENTS.md)

Action items

Evaluate which option is most feasible for the hackathon timeline

Prototype the chosen approach

Test with a small project (10-20 MRs)

Document how to trigger ingestion