ProgramGym: Scaling Training Environments for Long-Horizon Program Reproduction Agents

Chengyi Yang^1*, Renxu Bao^1*, Jue Hou^1*, Zongpei Teng¹, Ziheng Wang¹, Yujie Lin¹, Chengsong Huang², Junnan Zhu³, Delai Qiu⁴, Yining Wang⁴, Shengping Liu⁴, Fei Long^1†, Jinsong Su^1†

¹Xiamen University · ²Washington University in St. Louis
³Institute of Automation, Chinese Academy of Sciences · ⁴Unisound

^*Equal contributions · ^†Corresponding authors

Agents need coding like software engineers. So we build the gym

HuggingFace Paper

Overview

What the gym holds, and what training in it buys

Program-Env · Corpus

Executable tasks by source language

Program-Env 0 executable tasks

Go 818 49.64%
C++ 378 22.94%
Rust 303 18.39%
C 149 9.04%

ProgramBench · Results

Tests Passed Score (%)

Opus 4.8xhigh

GPT-5.6xhigh

GLM-5.2

Gemini 3.7Flash

DS-V4Flash-0731

Qwen3.8-SFTProgram-Agent

Qwen3.8-27BProgram-Agent

Gemini 3.6Flash

Kimi K2.7Code

Qwen3.8-SFTClaude Code

Qwen3.8-27B

70.9

69.9

64.6

61.2

57.68

56.25

55.71

55.7

53.6

42.84

34.87

Ours Baseline Program-Env Program-Agent

Tests Passed Score on ProgramBench (%)

Figure 1. Overview of ProgramGym. Program-Env contributes 1,648 executable tasks across four systems languages. Training in it carries Qwen3.8-27B from 34.87 to 42.84 (solid), and running either model under Program-Agent carries it to 55.71 and 56.25 (dashed) — closing much of the gap to far larger proprietary models.

Program-Env composition — 1,648 tasks
Language	Tasks	Share
Go	818	49.64%
C++	378	22.94%
Rust	303	18.39%
C	149	9.04%

Tests passed on ProgramBench (%)
Model	Group	Score
Claude Opus 4.8 (xhigh)	Baseline	70.9
GPT-5.6 Sol (xhigh)	Baseline	69.9
GLM-5.2	Baseline	64.6
Gemini 3.7 Flash	Baseline	61.2
DeepSeek-V4-Flash-0731	Baseline	57.68
Qwen3.8-27B-SFT + Program-Agent	ProgramGym	56.25
Qwen3.8-27B + Program-Agent	ProgramGym	55.71
Gemini 3.6 Flash	Baseline	55.7
Kimi K2.7 Code	Baseline	53.6
Qwen3.8-27B-SFT + Claude Code	ProgramGym	42.84
Qwen3.8-27B	Baseline	34.87

Program-Env

How a repository becomes an environment

Subagent Open full screen ↗

Program-Agent

How an agent rebuilds a program from its black box

psampaz/go-mod-outdated · Go · five revisions, in 19 beats. Click a line to replay it; hover a box for its file. Open full screen ↗

Program-Env · Results

1,648 repositories, eight domains, one contract.

Each environment is a real GitHub project, rebuilt into a sandboxed executable with a black-box suite. Developer tools and system utilities dominate, and almost every subagent targets I/O behaviour.

Figure 4 · Repositories by functional category

Distribution of the 1,648 repositories across eight functional categories.

Table 1 · Black-box interaction categories

Distribution of the black-box interaction categories over 5,918 subagents from 1,648 repositories.

Tests Category	# Subagents	% Percentage
Arguments	4,901	82.82%
Configuration	2,388	40.35%
Help	4,070	68.77%
I/O	5,873	99.24%
Subcommands	1,305	22.05%
TUI	716	12.10%

Cost and quality

136.77B tokens. 72.99% coverage. 404 tests per repository.

Building the test suites takes almost the whole budget: 130.83B of the 136.77B tokens. Those suites cover 72.99% of the gold executable's lines on average (median 81.63%) and keep 404 tests per repository (median 287).

Cost and quality · (a) tokens by stage, (b) mean coverage, (c) mean test cases

Per Repository Mean: 82.99M Median: 55.48M

(a)

(b)

(c)

Distributions

Small tools and large codebases, the same pipeline.

Code size spans four orders of magnitude; coverage and test counts stay high across all four languages, with C++ carrying the longest tail.

Figure 5 · Distributions of code lines, coverage, and test cases in Program-Env

DensityInterquartile rangeMedianMean1.5 IQROutliers

Distributions of code lines, coverage, and test cases in Program-Env (all 1,648 repositories).

Across the 1,648 repositories, test construction accounts for 130.83B of the 136.77B tokens spent; a repository costs 82.99M tokens on average.

Repositories by functional category (n = 1648)
Category	Repositories	Share
Developer Tools	441	26.76%
CLI & System Utilities	372	22.57%
Data & Scientific Computing	243	14.75%
Security & Forensics	227	13.77%
Media & Graphics	115	6.98%
Games & Education	110	6.67%
Productivity & Communication	73	4.43%
Languages & Runtimes	67	4.07%

Black-box interaction categories over 5,918 subagents
Category	Subagents	Share
Arguments	4,901	82.82%
Configuration	2,388	40.35%
Help	4,070	68.77%
I/O	5,873	99.24%
Subcommands	1,305	22.05%
TUI	716	12.10%

Per language: repositories, mean coverage, mean test cases, median code lines / coverage / test cases
Language	n	Mean cov.	Mean tests	Med. lines	Med. cov.	Med. tests
Go	818	72.60%	255.5	914	81.7	210
C++	378	64.83%	483.9	4,215	72.2	360
Rust	303	80.92%	572.2	1,159	87.3	488
C	149	79.66%	672.7	2,110	86.0	623

Tokens by stage (total, including 9 imputed repositories)
Stage	Tokens
Executable generation	4,659,772,234
Dispatcher	1,285,441,527
Test construction	130,826,470,528

Program-Agent · Results on ProgramBench

Training lifts the whole curve. Program-Agent lifts it again.

Share of the 200 ProgramBench instances whose rebuilt program passes at least a given fraction of the behavioural tests. Program-Agent lifts the curve for both models, and the fine-tuned one holds the most instances at the high thresholds.

Test pass-rate distribution · hover or tap for the exact values at any threshold

Qwen3.8-27B + Claude Code 34.87% 198 / 200 usableQwen3.8-27B + Program-Agent 55.71% 198 / 200 usableQwen3.8-27B-SFT + Claude Code 42.84% 198 / 200 usableQwen3.8-27B-SFT + Program-Agent 56.25% 199 / 200 usable

ProgramBench · Across models and agents

A 27B open model, trained here, lands beside the frontier.

Test pass rate against output tokens per task, for the public ProgramBench leaderboard and our own runs. The mascot marks Program-Agent and the fine-tuned model.

Program-Agent · Round budget

Each implement-review round raises the pass rate; five rounds raise it furthest.

The same fine-tuned model under Program-Agent at K = 1, 3 and 5 implementation rounds, against Claude Code and DeepSeek-V4-Flash. More rounds cost more tokens and keep paying off.

Test pass rate versus output tokens · K rounds

Program-Agent · What the agent does, turn by turn

Under Program-Agent the agent works longer, and more of each turn goes to reading and writing code.

One unit of mass per agent turn, split across the tool actions it made and summed over each setting's 200 trajectories. Under Program-Agent the fine-tuned model averages about 1,340 turns per task against 730 under Claude Code.

Normalized action count per agent turn · hover or tap for the breakdown at any turn

Normalized Action Count

Qwen3.8-27B +
Claude Code

Qwen3.8-27B +
Program-Agent

Qwen3.8-27B-SFT +
Claude Code

Qwen3.8-27B-SFT +
Program-Agent

ReadWriteProbeExecuteGitOther

Cite this work

arXiv:2610.13395

@misc{yang2026programgym,
  title = {{ProgramGym}: Scaling Training Environments for Long-Horizon Program Reproduction Agents},
  author = {Chengyi Yang and Renxu Bao and Jue Hou and Zongpei Teng and
            Ziheng Wang and Yujie Lin and Chengsong Huang and Junnan Zhu and
            Delai Qiu and Yining Wang and Shengping Liu and Fei Long and Jinsong Su},
  year = {2026},
  eprint = {2610.13395},
  archivePrefix = {arXiv},
  primaryClass = {cs.SE},
  url = {https://arxiv.org/abs/2610.13395}
}