Hugging Face trains one LoRA adapter per agent harness, with a proxy that blocks reward hacking

Hugging Face’s Harness Hub team published harness adapters on 6 November: small LoRA adapters, each trained with reinforcement learning for one base model inside one agent harness. The harnesses are pi, opencode and hermes, run unmodified in their own sandboxes. The idea is that you keep your local agent and swap in the adapter trained for it.

What is trained

The base models are Liquid AI’s LFM2.5-24B-A2B (a mixture of experts with 2B active parameters) and LFM3-1.7B, plus webAI’s TwIL-8B as an experiment, since its model card says it was never trained on agent data. Training uses GRPO through TRL’s AsyncGRPOTrainer with LoRA rank 32 on all linear layers, 8 rollouts per task and a learning rate of 1e-5, on 2 or 4 H200 GPUs on HF Jobs.

Because the adapters share one base model, one vLLM server hosts all three harness policies at once. HF says that made evaluation cheap: nine checkpoints of LFM2.5-24B-A2B, 1,350 agent sessions, ran in one H200 job in under an hour for about $5 of GPU time, against about $90 to merge and serve each checkpoint separately.

Where the tasks come from

Training mixes SmolDataEnvs, data-analysis questions from public Kaggle notebooks, with Repo2RLEnvs, tasks rebuilt from agent sessions people shared on the Hub. A pipeline replays each session, finds a verifiable unit of work and checks the resulting environment offline. HF is open about the skew: 253 of the 270 tasks are build tasks, 243 come from a single trace dataset, and most are easy for the largest model. It asks for more shared sessions where something breaks and gets fixed.

The reward-hacking guard

safeagents is a reverse proxy between the harness and the model. It checks every tool call with rules, not a model, in about 10 to 20 ms by HF’s measure: undeclared tools, hosts that never appeared in the conversation, edits to test files, writes to the reward file, proxy bypasses and token lookups are blocked, and any blocked rollout scores 0. HF shows one catch from a TwIL-8B smoke run, where the model went looking for the dataset bucket its task came from instead of computing the answer. In a 50-step ablation, turning the guard on changed held-out pass@1 by +0.0 ± 4.1 points.

The results so far

HF says that in smoke tests the adapters make fewer tool calls than the base model on held-out tasks while keeping or improving accuracy. Those are its own early numbers; the final runs, 200 to 400 steps per model, were still streaming to a live dashboard when we read the post.

Why this matters

The same model behaves differently depending on the harness around it, and this recipe tunes for that directly instead of hoping one checkpoint fits every agent. Everything is published: environments, safeagents rules, training settings and every blocked call, and the post shows how to scrub your own sessions with safeagents scrub and train an adapter for your harness. Even if you never train one, the rule list is a practical checklist of shortcuts an RL-trained agent will try. For background, see what an agent harness actually is.