Proof of concept for agentic GitLab Docs localization (into French)

Oleksandr Pysaryuk
2026-04-05T14:58:50Z

### What was built:

In a single agentic Claude Code session, I _manually_ prototyped what the **agentic localization architecture** could look like. :tada:

In the session, I translated the GitLab Duo section of GitLab Docs https://docs.gitlab.com/user/gitlab_duo/feature_summary/ (35 files) into French, with an LLM operating on **raw markdown (!), with no Phrase TMS**, no XLIFF, no tag repair cycles, no qa checker tool, etc.

See translated files in this MR: https://gitlab.com/gitlab-com/localization/tech-docs-forked-projects/test/gitlab/-/merge_requests/111+

Not **only** _translated_, but also **_internationalized_**: Claude Code helped me build language selector, side navigation works too, some other basics of i18n with json and yaml are in place. See this MR: https://gitlab.com/gitlab-com/localization/tech-docs-forked-projects/test/gitlab-docs/-/merge_requests/71+

See screen capture videos on Google Drive of the **GitLab Docs site in French**, from my localhost: https://drive.google.com/drive/folders/1ngWC-XZGgisfFSu5lFDBmBIpCyt1s9Rs

![CleanShot 2026-04-04 at 14.23.37@2x.png](/uploads/3a19b6130d7a7ac4a6937da78106fb9f/CleanShot_2026-04-04_at_14.23.37_2x.png){width="511" height="338"}

It was done completely by-passing the pipeline we had built in [tech-docs-ai-powered-translation](https://gitlab.com/gitlab-com/localization/tech-docs-ai-powered-translation)

Worth noting, I still used what we had built in [tech-docs-ai-powered-translation](https://gitlab.com/gitlab-com/localization/tech-docs-ai-powered-translation) and in QA checker, but differently, i.e. I manually ran every quality pass that our production pipeline performs: UI label consistency against the [French .po file](https://gitlab.com/gitlab-org/gitlab/-/tree/master/locale/fr?ref_type=heads), Phrase TMS TBX terminology compliance, and structural QA that the [tech-docs-mr-qa-checker](https://gitlab.com/gitlab-com/localization/tech-docs-mr-qa-checker) would do.

By the end, the output was:

1. \~35 translated French markdown files committed in `doc-locale/fr-fr/user/gitlab_duo/` https://gitlab.com/gitlab-com/localization/tech-docs-forked-projects/test/gitlab/-/merge_requests/111
2. A reusable translation rules slash command ( see attached file`translate-docs-fr.md`) that encodes all quality rules as a machine-readable agent instruction file. This is the spec, the skill, the translation rules - however we want to call it)
3. Working navigation, language selector, and partially translated navigation yaml
4. **A working proof that full-file markdown translation works without a TMS, wihtout intermediary XLIFF format, without filter configurations**

#### What I set out to do

I wanted to answer one question practically: **can an AI agent translate GitLab docs files at production quality, operating directly on markdown, using only version-controlled spec files as guidance?**

The existing pipeline (Argo - Phrase TMS - XLIFF - AI translation by Gemini from GCP - tag repair - QA checker tool - MR on fork - upstream MR) works. It launched Japanese docs at scale. But it carries complexity: custom file filter configurations, MXLIFF processing, up to 5 iterative tag repair passes, and a multi-system workflow where visibility is split across Argo, Phrase, and GitLab.

The hypothesis was, a capable AI agent (or rather, an LLM i.e. Claude Sonnet 4.6, running in a Claude Code agentic session) operating on raw markdown, guided by a version-controlled rules file `translate-docs-fr.md`, could produce comparable quality output instantly, without any of that infrastructure.

### Preparing the specs / rules file

Before any translation happened, I started with a rules file, fed all info form our epics and issues from the past when we had cerated Phrase filters, a produced a markdown slash command (`translate-docs-fr.md`) that defines how to handle every element in a GitLab docs .md file:

- YAML frontmatter (translate `title:` and `description:`, preserve everything else)
- Fenced code blocks (preserve verbatim, never translate code or comments)
- Hugo shortcodes (preserve structure; translate `title="..."` values; never translate `name=`, `data=`, etc.)
- Alert boxes in GLFM format (preserve `> [!note]` marker, translate the text)
- Inline code in backticks (preserve verbatim)
- etc. etc.

### Running first translations pass

With the rules file in place, I started translating files in `doc/user/gitlab_duo/`. The first few files went quickly, in minutes. Raw translation quality was high. Hugo shortcodes were OK. Code blocks were untouched. The structure held fine.

### The UI label problem

After several files were translated, I asked: _"How were you translating UI labels? Did you consult the French PO file?"_

The answer was no , the agent was translating UI labels like "Save changes" using general French, producing "Enregistrer". But the official GitLab French UI translation (in `locale/fr/gitlab.po`) is "Sauvegarder les modifications".

I grepped the PO file (85,720 lines) and the agent found 4 systemic errors affecting 68 locations across all files already translated:

| **Wrong** | **Correct (from PO file)** | **Occurrences** |
|-----------|----------------------------|-----------------|
| Enregistrer | Sauvegarder les modifications | 23 |
| Rechercher ou accéder à | Rechercher ou aller à | 28 |
| Admin | Administrateur | 6 |
| Général | Généralités | 11 |

Agent ceated a Python script bulk-fixed all 68 locations, knew to skip code blocks and any mentions of those terms in inline code, to avoid over-fixing false positives. The rules file was updated iteratively to add rules about how to handle UI labels

This is exactly the PO file lookup mechanism described in [tech-docs-ai-powered-translation/-/blob/main/doc/**ui_labels.md**](https://gitlab.com/gitlab-com/localization/tech-docs-ai-powered-translation/-/blob/main/doc/ui_labels.md)**,** and the agentic session proved the simplified mechanism works, too.

### Terminology compliance and TBX parsing

Next question I asked Claude Code: _"What about the Phrase TMS termbase? Are you translating according to glosary? Are you using forbidden terms?"_

I exported TBX from Quickterm (`GitLab termbase - QuickTerm-20011158.tbx`, 49 MB). Agent wrote a Python script to extract 3,713 English-Frrench term pairs, and scan all 30 translated files against it.

52 glossary violations found: (now, I don't know the state of our glossary, so it's up to a language professional to judge the below), some examples are

| **Forbidden term** | **Correct term** | **Occurrences** |
|--------------------|------------------|-----------------|
| Prérequis | Avant de commencer | 36 |
| conversation | discussion | 12 |

All 52 were fixed by a bulk replacement script. A **F**orbidden terms section was iteratively added to the rules file, forbidden terms were sourced directly from the TBX.

This is the first step toward the JSON-in-git termbase (with Quickterm still remaining the SSOT) where terminology is codified in version-controlled markdown, exported from SSOT

### Launching the localhost docs site and real bugs found

At this point I had 30 translated files. To see how they actually rendered, I launched the Hugo docs site locally

**Some real bugs I found that none of the text-level checks had caught:**

* Wrong tab placement (`turn_on_off.md`), a genuine translation error by LLM where content placed in the wrong versioned tab during translation.
* Missing image (`turn_on_off_earlier.md`). The screenshot (`![Cascading setting](img/disable_duo_features_v17_1.png)`) was completely dropped during translation by LLM
* Extra backtick (`` markdown.md ``) `` (``` ``) \`\` in an inline code example had 5 backticks instead of 4.
* Empty link text (`markdown.md`) `[](../administration/wikis/_index.md#allow-uri-includes-for-asciidoc)` had empty display text, the link existed but was invisible. Agent manually restored as `[paramètre d'application](...)`.

**All bugs were structural or content-level issues that Phrase's segment-level QA would not have caught** (not sure 100% so worth checking with Phrase experts).

Three of them (extra backtick, empty link text, wrong tab placement) are the kind caught by existing CI linting jobs or human review. One of them (dropped image) is completeness mistake by LLM, no linter can detect because it leaves no trace. This is for humans to find somehow

This was addressed addressed by a _self-review pass_ added to the rules file, where before writing output, the agent verifies that every image, every numbered step, and every link from the source is present in the translation. We can iterate on those rules, test each of them

### Hugo rendering failure: the 16 KB buffer

This was completely new to me, so the following section is written by Claude - I am not vouching for its technical accuracy:

> _While browsing the rendered site, `mcp_clients.md` failed to render entirely with:_
>
> ```
> slice bounds out of range [:16669] with capacity 16384
> ```
>
> _Root cause: Hugo's internal Goldmark markdown renderer allocates a buffer sized to the file's byte count, rounded to the nearest power of 2. The French file was 16,047 bytes — close to 16,384. After Hugo replaces shortcode syntax with internal `HAHAHUGOSHORTCODE...` placeholder tokens (which are longer than the original shortcode syntax), the total exceeded the buffer capacity._
>
> _Agent identified the longest over-translated prose sections by comparing line-by-line byte counts between French and English, then made 9 linguistic simplifications that preserved full meaning while reducing the file from 16,047 to 15,832 bytes. Hugo rendered the page successfully._
>
> _This is an important constraint for the production pipeline: French translation runs \~14% longer than English by byte count. Files over \~13,500 bytes in English need to be kept tight in French to stay under the limit._

### The anchor ID regression and wrong anchor text, wrong test environment

Worth noting, this happened because I did not tell the agent initially that have a service (Argo Agent) that injects anchor IDs to markdown files as a pre-processing step. https://gitlab.com/gitlab-com/localization/docs-site-localization/-/work_items/155+

Agent added `{#french-anchor-id}` suffixes to all translated headings, derived from the French heading text. The correct approach (used by the Argo Agent pipeline for Japanese) is to derive anchors from the **English** heading text. All 275 French-derived anchors were removed during the session (as iteration one). Then English-derived anchor IDs were injected into all French headings via script. See result in this commit: https://gitlab.com/gitlab-com/localization/tech-docs-forked-projects/test/gitlab/-/merge_requests/111/diffs?commit_id=38cf2271c34036a35477cdf645b8fee93be68fba

### A second translation pass and FR term `issue` vs. `ticket`

A follow-up session translated four files from `doc/user/project/`: `issues/_index.md`, `create_issues.md`, `managing_issues.md`, and `description_templates.md`. These are the core GitLab project management pages, and they use the word "issue" constantly.

A quick TBX lookup confirms that the term: `issue` (en-us) is recorded as `ticket` (fr-fr), status: Preferred, Approved.

So it seemed like a simple substitution to ask the agent to do

Except it wasn't.

An agent-written script replaced all occurrences of "issue" with "ticket" across the files. It worked but it also changed words in links like `gitlab.com/-/issues/523713` to `gitlab.com/-/tickets/523713`, `#edit-an-issue` anchor references to `#edit-an-ticket`, and file paths like `issues/_index.md` to `tickets/_index.md`. The translated files became structurally broken. Asked the agent to revert it

The script was rewritten to replace terms in display text only, preserve URL/anchor/path verbatim. But then \~40 grammatical agreement patterns broke

Why? Because _issue_ is a feminine noun in French. _Ticket_ is masculine. Swapping the noun without swapping everything that agrees with it produces broken grammar. Here is what agent summarized for me that needed attention in French

| Pattern | Before (with _issue_, feminine) | After (with _ticket_, masculine) |
|---------|---------------------------------|----------------------------------|
| Indefinite article | `une issue` | `un ticket` |
| Definite article | `l'issue` | `le ticket` |
| Contraction de + article | `d'issue` | `de ticket` |
| `de le` → `du` | `de l'issue` | `du ticket` |
| Adjective | `confidentielle` | `confidentiel` |
| Adjective | `existante` | `existant` |
| Adjective | `nouvelle` | `nouveau` |
| Past participle | `créée` | `créé` |
| Past participle | `clôturées` | `clôturés` |
| Pronoun | `elle` (referring to issue) | `il` (referring to ticket) |
| Determiner | `toutes les issues` | `tous les tickets` |

`managing_issues.md` alone required changes in 223+ lines (!!)

The fix was fully agentic. Two more targeted passes: 1) a regex script that caught the most common patterns (articles, the most frequent adjective and participle forms), followed by 2) a direct string replacement pass for specific sentences that resisted the pattern approach. After both passes, every instance looked correct.

Worth noting: I have enough basic French knowledge to validate. But a quality pass by a human could be necessary. The beautiful thing is, agent did it with my guidance and via scripting. **Conclusion: we need humans, and we need terminology :smile:**

_Engineering_ takeaway for the production pipeline: the future proposed `terms/fr-fr.jsonl` termbase format should include a `gender` field on each noun entry (`"gender": "m"` or `"gender": "f"`). A gender field in the termbase means a script could better automatically fix grammatical agreements before humans validate.

### External titles and further rule refinement iterations

While scanning for translation quality issues, I found that YouTube video titles and blog post titles had been translated:

- "GitLab Duo Chat (agentic) - MCP tool call approval" was translated into French
- "Streamline DevSecOps engineering workflows with GitLab Duo" was translated into French

Rule needed to be updated to instruct LLM that published external titles (YouTube videos, blog posts, conference talks etc.) must be preserved verbatim in English.

Some version-range tab titles ("In 17.8 to 18.1", "In 17.4 and later") had been left in English. An updated script fixed all 11 instances across 3 files:

- "In 17.8 to 18.1" --\> "Dans 17.8 à 18.1"
- "In 17.4 and later"-\> "Dans 17.4 et versions ultérieures"
- "In 17.3 and earlier" --\> "Dans 17.3 et versions antérieures"

### Local preview and French sidebar navigation fix

When browsing the local Hugo site, the left sidebar navigation rendered in English for all fr-fr pages even though `data/fr-fr/navigation.yaml` has French titles. Japanese worked correctly.

Root cause (found by asking GitLab Duo on the codebase, but by Claude Code): `fr-fr` was not registered in `locales/index.js` alongside `en-us` and `ja-jp`. The Vue sidebar component falls back to English for any locale not in that file. Fixed locally by creating `locales/fr-fr.json` and adding `fr-fr` to the languages object. Here is what an actual MR _could_ look like: https://gitlab.com/gitlab-com/localization/tech-docs-forked-projects/test/gitlab-docs/-/merge_requests/71/diffs

### What the rules file became

Through iteration, `translate-docs-fr.md` evolved from a basic decision tree into a comprehensive agent instruction file:

| Section | Content |
|---------|---------|
| Decision tree (13 rules) | How to handle every structural element in GitLab docs markdown |
| Glossary | \~40 terms that must never be translated (product names, technical terms, tools) |
| Structural invariants | 8 constraints including file size limit (Hugo 16 KB buffer) |
| More rules | 6 explicit rules for every ambiguous case (register, glossary precedence, imperative tone, etc.) |

UI label compliance and forbidden term enforcement were validated during the session but are not yet encoded in the rules file. Both depend on data sources (PO file, TBX glossary) and terminology is still being cleaned up

**This file is the spec layer in the agentic localization.** It is version-controlled, reviewable, editable as a standard MR, and readable directly by any AI agent. It couold further be split into: `fr-fr-style-guide.md`, `terms/fr-fr.jsonl`, and `translation-rules.md`. Up to us to define the specs that make most sense and separate the rules from sgtyle from glossary from UI labels concerns, or other future concerns

---

### What the Claude Code session proved

| **Hypothesis** | **Result** |
|----------------|------------|
| Full-file markdown translation works without XLIFF segmentation | Proven. 35 files, all structure preserved |
| Hugo shortcodes survive without Phrase filter configurations | Proven. Tabs, alerts, history blocks, icons, all intact |
| Tag repair cycles are unnecessary with full-file translation | Proven . Zero tag corruption in 35 files |
| UI label compliance is achievable at agent level from PO file | Proven 68 UI label isnatnces corrected |
| TBX termbase compliance is achievable at agent level | Proven. 52 terminology violations found and fixed |
| Whole-file QA catches bugs segment-level QA cannot | Proven. 4 bugs found by rendering, not text analysis. TBD how this QA should be done - human or more specs for an agent |
| A version-controlled rules file can serve as the agent's system prompt | Proven. Every quality pass improved the rules file iteratively; rules were immediately re-applied to all files |

### What's missing to make this the real pipeline

**At least these things** separate this manual session from fully agentic yet:

1. **Argo creates a GitLab issue**. (today Argo creates new Phrase projects, or adds files to continuous projects in Pharse). The change detection and scoping logic already exists in Argo, the pivot will be routing to an issue instead of to Phrase.
2. **The agent pushes a branch and opens an MR.** The translation is proven manually in this POC. Automation needs be built
3. **Structural CI is already in place.** The existing jobs like `docs-i18n-lint markdown` and `docs-i18n-lint links` run automatically on any MR touching `doc-locale/`. They need to be further streamlined and matured to get pipelines to green. Other less deterministic more human aspects, could be addressed by the agent self-review pass now encoded in `translate-docs-fr.md`. And need to determine what to do with QA checker tool that's standalone
4. etc. TBD collaboratively :collaboration:

**Immediate next step proposed:**

- [ ] Share `translate-docs-fr.md` with the French translation professional (vendor, Content Manager) as the spec to get their feedback will drive the next iteration of the rules

**Not doing yet:**

- `terms/fr-fr.jsonl` termbase - terminology governance process is still in progress; premature to encode. Need to evolve Quickterm as an SSOT. and test JSON export
- UI label CI enforcement - PO file quality needs improvement before it can be treated as authoritative source of quality UI translations
- `fr-fr-style-guide.md` - linguistic style rules (punctuation spacing, number formatting, etc.) are a longer-term investment once the basic workflow is proven. They do not exist today. Style philosophy is outlined here: https://gitlab.com/gitlab-com/localization/docs-site-localization/-/work_items/889#note_3171894105

#### Where this goes next

This session is a proof of concept, not a production system. The translation works, the quality passes are validated, and the rules file is ready to use. The strategic decision on what happens next is for Leadership, the Program Management and Localization Engineering team collaboratively. The options range from running this as a parallel track alongside the existing [tech-docs-ai-powered-translation](https://gitlab.com/gitlab-com/localization/tech-docs-ai-powered-translation) pipeline, to a fuller pivot from TMS. That call is not made yet.

Once this work is completed: https://gitlab.com/groups/gitlab-com/localization/-/work_items/166+, **what I'd like to see happen on the engineering side is,** a prototype of the automation layer that removes the human trigger. There are two options, and they are trade-offed differently:

* **Option A. Argo calls the translation agent.** Argo detects changed English source files, creates a GitLab issue, and the existing GDATP / GitLab Translation Agent picks it up, but instead of the current [tech-docs-ai-powered-translation](https://gitlab.com/gitlab-com/localization/tech-docs-ai-powered-translation) pipeline, it uses the French rules file (`translate-docs-fr.md`) as its spec and opens a translation MR directly. Requires engineering work.
* **Option B. Localization Engineer runs Claude Code.** An engineer runs Claude Code on a batch of files, the LLM translates using the rules file, and the engineer pushes the branch and opens the MR. Perhaps on a select set of files. To keep polishing the POC. Requires engineering time to run translations.
* Open to more options.

My recommendation as the author of this proof of concept is **define Option B better and start with B**. The overhead is low enough that it proves the end-to-end workflow without committing to building an automation infrastructure yet. If the output quality holds and the review process works, Option A becomes the obvious next investment.