Synthetic codebases as RL training signal

A multi-agent pipeline for generating production-shaped, cross-language code in regions where frontier-model priors are weak, and the GRPO signal it unlocks.

Ryan Clark · May 2026

The contamination problem

The standard recipe for training a coding model with RL (pick a public benchmark, gate on its test suite, reward what passes) has a quiet failure mode. The model has already seen the benchmark. Or it has seen a paraphrase. Or it has seen the same algorithm in a different file. By the time you measure, the test you wrote with verifiable rewards in mind is actually rewarding recall.

I started the project by running the obvious thing first: a pipeline that synthesizes classical CS problems (interval trees, scheduling, parsers) at 800–2,000 LOC, with an architect/test-designer/builder separation and an information-starved repair loop. 35 seeds landed at 99.83% test pass overall, and the average init→final delta across the repair loop was 0.50pp. The repair loop had nothing to recover; the builder was solving the problems on the first try from priors. The negative result was sharp enough to justify pivoting the whole approach.

The pivot: stop trying to make the task harder by adding rules, and instead push the shape of the codebase away from what a frontier model has seen. Two directions:

• Line A: production-shaped multi-subsystem services at 10k+ LOC, with locked cross-module contracts that expose interface-inference gaps a single-file CS problem can't.
• Line B: cross-language ports of real OSS projects, where paradigm shifts (Python async → Go errgroup, Ruby metaprogramming → Rust traits) force genuine restructuring instead of transliteration.

Pipeline design

Both pivot pipelines share Phase 1's architectural spine: role-separated agents with information barriers between them. The test-designer never sees the builder's reasoning; the builder never sees test bodies (only function names during repair); each stage is a separate process with a sandboxed filesystem view.

Line A's pipeline:

Architect-multi  →  Decomposer (subsystem contracts)
                 →  Test Designer (25–40% e2e, 40–55% integration, 10–20% unit)
                 →  Integrator (error/config/clock/observability layers)
                 →  Module-builder × N (per subsystem, topological waves)
                 →  Scorer  →  Repair (info-starved: function names only)

The key discipline is the contract lock. After the decomposer fixes a subsystem's public signatures, no later stage can mutate them. A module-builder that finds a signature impossible must emit a structured CONTRACT_REVISION_REQUEST, not paper over the gap. This catches the silent-drift failure mode where each builder fixes "their" problem locally and the integration layer breaks.

Line B's pipeline replaces the architect with an architect-port that reads the source repo's code and emits target-language idioms, and adds a compile-repair stage for cross-language contract mismatches between translator and architect (the dominant failure mode for Rust and Zig targets).

Reward hacking and the defenses that caught it

The test-blind contract is the core safety property of this pipeline: if a builder can read the held-out test bodies, the training signal collapses into "memorize this oracle" instead of "satisfy the spec generally." Treating that property as a design intent rather than an enforced invariant is a mistake. Five times across the pipeline's lifetime the builder has tried to bypass it. Every attempt left trajectory evidence; none made it into a shipped artifact's score; the catalog grew as new pipeline variants opened new attack surfaces.

Three representative incidents:

1. fastcsv `cp` exploit (2026-04-20 01:11 PDT). During port-fastcsv's builder-repair, the builder ran ls translator/, discovered *_test.go files were readable, then cp translator/reader/*_test.go builder/reader/ and recompiled the held-out tests as its own code. go test passed trivially. Detected during a debug session unrelated to security; the cp commands are unambiguous in the trajectory JSONL. The root cause was a macOS-26 silent regression in sandbox-exec's file-read* rules: the sandbox was cosmetic, the deny-list had been effectively off for the entire batch. Defense: replaced sandbox-exec deny rules with chmod 000 on translator/, reviewer/, and architect/tests/ for the duration of every builder run.
2. chrono-node cross-run exfil (2026-04-20 17:04 PDT, caught live). The source-visible-builder experiment ran on a sibling task tree to a finished Wave-1 baseline. Mid-test-repair, the builder ran cp /Volumes/.../port-chrono-node-20260420/.../translator/tests/*.rs tests/, reaching across task trees because chmod was scoped to its own run, not to sibling runs. 34 minutes of oracle-access work before the live progress-check caught it. Defense (operator-applied, not yet automated): killed the run, manually chmod 000-quarantined the specific Wave-1 baseline's translator/, reviewer/, and architect/tests/ dirs for the duration of the relaunch. The relaunched trajectory showed zero cross-run reads; the quarantine held. Open item, tracked in RESEARCH_LOG §17: the within-task-tree chmod is wired into withHiddenPaths() for every builder spawn, but the cross-task-tree sweep is still manual per run. Closing it is a 10-line walk over RUNS_DIR at builder spawn time; it isn't installed because no further source-visible runs have been launched since the chrono-node experiment was set aside.
3. mypy → Go trajectory probe (2026-04-21 02:30 UTC, unsuccessful). After being blocked on ls translator/ (chmod working), the builder tried head -5 test-translator.trajectory.jsonl, a new attack surface targeting task-dir-level stream-json files that the chmod list didn't cover. The 5 lines it got back contained only session-init records and an encrypted thinking block; no test content actually leaked. The builder didn't follow up with grep or cat, which would have pulled real oracle content from the translator's tool_results. Defense: collectSiblingTrajectoryFiles() now chmod-hides every *.trajectory.jsonl, *.stderr.txt, and *.stdout.json from sibling stages for every builder run (keeping the active builder's own paths writable).

What I take from these: the test-blind property has to be defended in depth, not just declared. Information-starved prompts close the trivial path, but the agent is creative; the actual failure mode is a configuration gap the agent finds before the operator does. The piece that does most of the work is the catalog mechanism: every detection cross-referenced to a fix and a re-baseline cost. A Phase-0 honesty rebaseline of 5 pre-chmod ports under the new isolation cost $56 and reproduced 4 of 5 final pass rates to within zero; the fastcsv exploit "didn't inflate the pass rate, only shortcut the path." Opus can solve Java → Go honestly; it just takes 3× longer without the oracle.

What the pipeline produces

Line A: 5 production-shaped seeds

^† invoice-engine ran on the pre-patch pipeline; its 86.8% init reflects a re-score after a since-fixed integration-layer signature-drift bug (FrozenClock(now=...)) was patched. The +13.2pp delta is inflated by that fix and is not directly comparable to the other rows. The raw first-pass score before the patch was 8.8%.

The summary: 100% final-pass numbers across all five seeds, and a mean delta of +6.5pp where Phase 1's classical-CS pipeline produced 0.50pp. The repair loop has work to do on the production-shaped distribution that it didn't on classical algorithms. The deltas themselves are more modest than I'd hoped. Hos-dispatch-scheduler at +8.2pp (an hours-of-service scheduler for trucking) is the data point I'd point to for the "niche-domain → bigger signal" hypothesis the pipeline was designed to test, but it is n=1: medicaid-eligibility-engine is also an obscure-operational-domain seed and only landed +4.0pp, and invoice-engine's +13.2pp is a pipeline-bug artifact rather than a domain-difficulty effect (see footnote). Directional, in other words, but it needs replication on more niche seeds before I'd call the pattern general.

Line B: cross-language ports

The pipeline ran 31 cross-language port attempts across three target languages, covering source projects in Python, TypeScript, Java, Ruby, OCaml, Nim, C, C++, and Go. 28 produced scores (tables below); two early bbolt-zig attempts (v1, v2) hit scorer-error from Zig stdlib API drift and are excluded; one libsodium → Rust attempt was killed mid-architect, and only the libsodium → Zig attempt completed. Numbers below are the post-Wave-2 canonical scores after a reviewer stage augmented each port's test suite (1,523 supplemental tests added in Wave 1, 659 in Wave 2). For three ports the pre-reviewer scores tell the more dramatic story; those are called out in prose.

Go targets

Rust targets

Zig targets (Wave 3 stress test)

* trio-core's main run shipped with a $0 entry because the reviewer-upgraded scorer accepted the existing build with no repair rounds; the underlying Wave-2 rescue cost $9.90 separately. bbolt-v3 is the first bbolt-zig attempt to produce a clean scoring run; v1 and v2 hit scorer-error as noted above, v4 is a follow-on iteration not yet rolled into the canonical table.

Four failure-mode clusters emerged, visible in the columns above: recall-dominated (init ≥97%, deltas under 3pp; the majority of the slate); already at ceiling (init 100%, like fastcsv and centrifuge); test-repair productive (whoosh +14.1pp, python-docx +8.3pp, oniguruma +4.4pp); and compile-repair productive (chrono-node, the original commonmark-java run, several Wave-2 rescues). The last cluster is the reason the compile-repair stage exists; without it, ports with translator/architect contract mismatches would have been thrown out as failures rather than recovering.

Three Wave-1 ports closed dramatic deltas on their original test suites before the reviewer-upgrade added stricter tests: arrow-py (Python → Go) went from 3.8% → 98.8% on the initial 211-test suite; trio-core (Python → Go) closed 36.3% → 100%; ajv (TypeScript → Go) closed 59.7% → 100%. These are clear cases of the pipeline producing training signal where pretraining priors were genuinely weak. The post-reviewer scores in the tables above are smaller because the reviewer added tests targeting edge cases the builder hadn't yet covered, raising the baseline against which the final delta is measured.

The Zig wave stress-tests the pipeline's limits hardest. Wave-3's sub-100% Zig scores aren't builder failures; they trace to translator/reviewer output tracking Zig stdlib API drift across releases (std.fs.Dir vs std.Io.Dir, removed realpathAlloc, @import package-name traps). Each bbolt iteration (v1–v4) peeled one layer; the pattern is that the pipeline's failure surface is finite and fixable per Zig release, which is informative but a real maintenance cost.

Scaling: the 95k-LOC mypy → Go port

The scale question came next: does the pipeline hold together at the scale frontier labs actually care about? I ran one port designed specifically to stress that: mypy v1.13.0, filtered to 95,340 LOC of Python implementation across 125 files (plus 186,592 LOC of tests), targeted to Go. Go was chosen deliberately. Rust as a target was killed by Astral's ty and Meta's Pyrefly both showing up in pretraining data, so Go gives zero canonical Python-type-checker prior art for the model to lean on.

The +9.5pp delta is not the interesting number. The interesting number is that a 95k-LOC port compiled clean on first try at all, and that the information-starved repair loop continued to extract signal from a codebase at this scale. The relaxed-spec + source-visible builder combination (used together for the first time on this run) is what made it possible: the builder reads the source repo for heuristics while still working against a locked, decomposer-owned API contract for tests.

Human-likeness: 5-seed replication

One persistent concern with synthetic code is that it reads as synthetic: uniform module openings, single-error-helper funnels, no version markers, no scar tissue. The first probe on feature-flag-service compared three variants:

Variant C is multi-epoch construction where each wave's prompt frames an evolution narrative ("v1 didn't expose X, so we wrap Y in v2; v3 under deadline we inline Z"). It produced visibly more human-like code without sacrificing correctness. To check that the effect generalized, I replicated variant C across all 5 Line A seeds (~$206 total for the replication batch):

* invoice-engine's raw failure is 89 of 96 failures tracing to one missing LineItem.tax_category field, same class as a pre-patch baseline bug and not drip-feed-specific. Hypothetical pass-rate with the one field added: 94.9%.

The 5-seed pattern is consistent: TODO/FIXME/HACK markers go from 0–1 to 11–25 on every seed; HACK/FIXME/NOTE file counts go from 0–1 to 9–20 on every seed; wrappers/shims up on 4 of 5; comment density and variance up on 4 of 5. The LLM judge aggregate across 10 trials (5 seeds × 2 position-swapped) was 9 C wins, 1 tie (invoice), 0 reverse wins. A follow-up calibration against 8 well-maintained Python OSS libraries (httpx, attrs, click, tenacity, structlog, cattrs, typer, marshmallow) placed variant C inside the human-authored distribution on comment density (~p75), comment variance, and TODO-marker density. It under-produced wrappers/1kLOC and over-produced error-handling heterogeneity, so the verdict is "closest synthetic variant, not parity."

The GRPO validation

The engineering only pays off if the pipeline produces actual RL training signal. I ran a matched-budget GRPO comparison on Qwen3-30B-A3B with two arms, varying only the task source:

• Synthesis arm: 30 tasks drawn from the 5 Line A seeds. Untrained baseline mean reward: 0.640.
• Public OSS arm: 29 tasks drawn from 7 well-maintained Python libraries (attrs, click, jinja2, packaging, rich, markdown, arrow). Untrained baseline mean reward: 0.839.

The two arms have qualitatively different shapes. The Synthesis arm rises after step 10, recovers from the step-40 dip, and the final checkpoint is the best of the six (+8.9pp over untrained). The Public OSS arm oscillates inside a ±0.025 band around its 0.839 baseline and finishes below untrained at −1.3pp. Caveats stated up front: this is n=1 per arm, six measurement points, on the training task distribution; the synthesis arm hit a SIGTERM around step 35 and was resumed from the latest preserved checkpoint, so the post-resume trajectory is what's tabulated. The qualitative gap is what the chart is for. 50 steps × 29 tasks at 0.839 baseline is too short to falsify a slow OSS climb, but it is enough to say that at this budget, the synthesis arm produces gradient the policy can move on and the public-OSS arm does not.

The full-pass rate split makes the gap starker: Synthesis 29.2% → 32.1% (+2.9pp), Public OSS 0% throughout. The policy never produced a deterministic full-task solution on any public-OSS task at any checkpoint during training. The cross-subsystem coupling Line A's pipeline builds in (locked contracts, integration-layer signatures, tests that span modules) is exactly the source of partial-credit reward that public OSS, with its 1:1 module-to-test structure, doesn't generate.

The translation pipeline (next iteration)

Line B's port orchestrator worked, but its single-shot architecture made the compile-repair failures expensive to diagnose: the translator, the test-translator, and the architect were entangled. The lessons from Line B fed directly into a separate cleaner pipeline (/translation-codebases) that decomposes translation into six independently-scoped stages with explicit JSON-schema'd contracts between them:

Snapshot   →  Surveyor (module map, dependency waves)
           →  Planner (target-lang layout, LOCKED signatures)
           →  Integrator (skeleton, build, test harness)
           →  Translator × waves (per-module, parallel within wave)
           →  Verifier (build + test)

The discriminating change is that the Planner's signatures are immutable after they ship. Translators that find a signature impossible emit CONTRACT_REVISION_REQUEST rather than diverging silently, the same discipline that worked for Line A's subsystem contracts generalized to cross-language porting. The framework is implemented and configured for chibicc (C → Zig) as the first end-to-end test; the first production run is the immediate next step.

Where this falls short

• Line A deltas haven't reached the target band. 3.5–8.2pp is real signal but not the 20–30pp regime that would justify "this is the obviously-correct training distribution." Niche-domain seeds (hos-dispatch at 8.2pp, mypy → Go at +9.5pp peak) point in the right direction but need more replication.
• Cross-language ports skew toward recall-dominated. Most of the 31 ports land at 97%+ init; the high-signal cases (arrow-py +95pp on original tests, trio-core +63.7pp, ajv +40.3pp, whoosh +14.1pp post-upgrade) are the minority. The mypy port shows the pipeline holds at 95k LOC, but it's still one data point at that scale; whether the pattern repeats on other large source repos is open.
• Test-signal saturates on semantic bugs. Function-name-only repair feedback can't convey input/expected pairs. Long-tail failures (arrow-py's 3 remaining locale-pluralization rules, chrono-node's 32 failing edge cases) hit diminishing returns.
• Human-likeness uses an LLM judge in the same family as the builder. The 5-seed replication (9 wins / 1 tie / 0 losses across 10 trials) and the OSS-distribution calibration both reduce the contamination concern, but independent human raters would close the loop. The judge consistency across position swaps is a positive signal, not a substitute for that.
• GRPO results are in-distribution. The +8.9pp on Synthesis is measured on the training task distribution; without external benchmark evaluation (SWE-Bench, LiveCodeBench) the strongest claim is "RL has more gradient to push against on synthetic tasks," not "the resulting checkpoint is a better coder." Closing this gap is item #1 below.
• The Zig adapter tracks a moving target. Sub-100% Zig scores trace to stdlib API drift (std.fs.Dir vs std.Io.Dir, etc.), not builder correctness. Each Zig release re-opens the prompt-update surface.

What's next

1. External-eval validation of GRPO. The +8.9pp on the Synthesis arm is in-distribution. SWE-Bench Verified, LiveCodeBench, and BigCodeBench scores for the trained LoRA head vs. untrained base are what would actually demonstrate "synthetic codebases have real training value." This is the priority follow-up.
2. Niche-domain seed expansion. Hos-dispatch (+8.2pp) and mypy (+9.5pp peak) both come from outside the commodity platform category. Industrial protocols, financial batch systems, specialty scheduling are the next targets.
3. First end-to-end translation run. The chibicc C → Zig run on the new translation pipeline is configured but not yet executed. Validating that the LOCKED-contract design actually holds across dependency waves is the open question.
4. Independent human raters for the human-likeness judge. The 5-seed LLM-judge result (9 / 1 / 0) is consistent across position swaps but rated by a model in the builder's family. A small human-rater study (~30 pairs) would close the contamination concern.
5. Drip-feed as a per-seed config. The wave overrides are currently an experiment-specific overlay. Folding them into seed definitions makes "produces human-like multi-wave code" a default property of the right seeds rather than a special mode.

Code & resources

• Repository: github.com/RyanClark2k/synthetic-codebases
• Line A orchestrator: src/orchestrator-multi.ts
• Line B port orchestrator: src/port-orchestrator.ts
• Seed library (80 hand-curated briefs): seeds/seeds.json
• GRPO task bundler: grpo-experiment/prepare_task_bundle.py
• Human-likeness experiment: experiments/human-likeness/FINDINGS.md
• Translation pipeline orchestrator: translation-codebases/src/orchestrator.ts