Build agent harnesses that improve with feedback.
Run reproducible experiments, build for a new domain, or turn corrections into changes to your own harness.
🧭 Contents · 🧪 Experiments · 🛠️ Customize · 🌱 Evolve · 📚 Docs · 💬 Community · 📖 Citation
🧪 Run experiments Follow how a harness changes across rounds. Open experiments → |
🛠️ Customize a harness See a domain harness collect evidence and answer. Open customization → |
🌱 Evolve your harness Turn a correction into a proposed change. Open personal evolution → |
- 🧪 Run experiments
- 🛠️ Customize a harness
- 🌱 Evolve your personal harness
- 💬 Community
- 📚 Documentation
- 📖 Citation
- 🔗 Related work
- 📜 License
Tip
Expand a workflow to see what you can build with HarnessGrad and how to get started.
🧪 Run experiments — compare harnesses and RSI methods under recorded conditions
| ALFWorld | AppWorld | ScienceWorld |
|---|---|---|
![]() ![]() |
![]() ![]() |
![]() ![]() |
These curves come from experiments run with HarnessGrad. Click any panel to inspect the original chart.
Choose a harness → Select an RSI method → Run tasks → Inspect the evidence. Use the same recorded conditions when comparing results, then explore the learning curve, versions and task trajectories in the Console.
A research run is one point in a harness × method × benchmark space: a harness solves tasks, a method proposes edits to that harness between rounds, and a benchmark scores each candidate. HarnessGrad provides tasks and scoring, isolated execution, and version control for candidate harnesses, with a recorded curve for each experiment.
[!NOTE] The inventory lists declared integrations; each harness and benchmark has its own preparation and runtime requirements.
Full harness, method and benchmark inventory
The table records the inventory at this README revision; the benchmark row is checked against the declared adapters.
| axis | what ships | where |
|---|---|---|
| harnesses | 8: loop, codex, opencode, deepseek-harness, openclaw, miniagent, miniswe, oracle |
base_harness/<name>/ |
| RSI methods | 19: dgm, rrsi, adas, ahe, ace, skillopt, evoprompt, evo_harness, opro, tthe, dream_rsi, sica_ci, gepa, darwinx, arbor, hyperagents, meta_harness, harnessx, modular_rsi |
experiments/method-<name>.json |
| benchmarks | 56 declared adapters; preparation and runtime prerequisites vary — every one listed: agentdojo, aime, alfworld, appworld, authored_example_fix_bug, bbeh, bfcl_v3, bird_minidev, browsecomp, chessbench_puzzles, cooperbench, costbench, costbench_dynamic, deepsearchqa, deepswe, ds1000, fever, gaia, gsm8k, harvey_lab, hle, hotpotqa, jericho, jobbench, kumo, livecodebench, livemathbench, locomo, math, math500, memoryagentbench, musique, nq, officeqa_pro, personamem_v2, physicsquestions, reasoning_gym, scienceworld, searchqa, sportsunderstanding, spreadsheetbench, spreadsheetbench2, strategyqa, strongreject, supergpqa, swe_bench, swe_bench_pro, tau2_bench, tau3_bench, tau_bench, terminal_bench, travelplanner, triviaqa, twowikimultihopqa, widesearch, workbench |
data/datasets/, detail in DATASETS.md |
- Bring a harness: supply a directory with
harness.jsonand an entrypoint accepting--taskand--workdir. See bring your own harness. - Bring a method: supply a program that reads a trajectory and writes a candidate tree. See bring your own method.
- Bring a benchmark: write an adapter that materialises tasks and runs its official scorer. See dataset onboarding.
A harness supplies harness.json and an entrypoint accepting --task and --workdir (interface). The platform runs dataset tasks and scoring, isolates execution, versions candidate harnesses, and records each run's outcomes and receipts. Dataset and harness prerequisites still need to be installed separately.
Each round records the candidate harness hash and its changes, alongside the run's model roles and measurement metadata.
[!IMPORTANT] Compare results only when the task selection, scorer, models and other relevant conditions support it. Failed or incomplete measurements are not scores; do not substitute proxy scores under a benchmark name. See the measurement policy.
Requires Linux, Python 3.12+, Git, uv, and a working bubblewrap. See the first-use guide for setup details.
git clone https://github.com/zjunlp/HarnessGrad.git
cd HarnessGrad
uv venv --seed --python 3.12 ~/.venvs/harnessgrad
source ~/.venvs/harnessgrad/bin/activate
uv pip install -e .
./install.sh --checkSet the shared external dataset directory in your local .env:
HG_DATA_ROOT=/path/to/datasetsSee downloads for preparation and locations, and datasets for adapter details.
[!TIP] New to HarnessGrad? The first-use guide includes an installation control that does not need a model call or dataset download.
Prepare Loop and GSM8K, then configure your OpenAI-compatible solver and editor endpoints. Replace the placeholders below and use a fresh run ID for each experiment. See the experiment guide for details.
# The provided configuration uses GSM8K.
python tools/configure_harness.py --harness base_harness/loop --fetch
python tools/datasets.py --fetch --dataset gsm8k
python tools/prepare_dataset.py --dataset gsm8k
python tools/datasets.py --check --dataset gsm8k
export HG_AGENT_BACKEND=openai
export HG_AGENT_BASE_URL='YOUR_SOLVER_ENDPOINT_URL'
export HG_AGENT_MODEL='YOUR_SOLVER_MODEL_NAME'
export HG_AGENT_API_KEY='YOUR_SOLVER_API_KEY'
export HG_METHOD_BASE_URL='YOUR_EDITOR_ENDPOINT_URL'
export HG_METHOD_MODEL='YOUR_EDITOR_MODEL_NAME'
export HG_METHOD_API_KEY='YOUR_EDITOR_API_KEY'
HG_QUICKSTART_ID="dgm-$(python -c 'import uuid; print(uuid.uuid4().hex[:12])')"
harnessgrad run --config experiments/method-dgm.json --run-id "$HG_QUICKSTART_ID" --check-only --check-net
harnessgrad run --config experiments/method-dgm.json --run-id "$HG_QUICKSTART_ID" --fail-on-incomplete
python tools/analysis/run_summary.py "runs/$HG_QUICKSTART_ID"Use the shared Console to inspect experiments, harness versions and task trajectories.
export HG_CONSOLE_RUNS="$PWD/runs" # existing run-record directory
python console/server.py --port 4460Open http://127.0.0.1:4460/ from the repository root, or point HG_CONSOLE_RUNS to another existing run-record directory. The Console binds to localhost by default.
![]() Experiments |
![]() Versions |
![]() Trajectories |
Thank you to everyone who contributes harnesses, methods, benchmarks, and improvements to HarnessGrad! The three charts show the most recent 30 days; each line includes contributions made before that window.
Want to join? Bring a harness, add a method, or onboard a benchmark, then open a pull request. We will add your name to the relevant contributor chart.
🛠️ Customize a harness — build for a model, environment or task
Ocean Harness is one example built with HarnessGrad. It combines domain sources, research tools, and an answer workflow to answer ocean questions with citations. Build a harness for your own domain using the workflow below.
Use Python 3.12+ and install from the HarnessGrad checkout. For model-generated edits, configure an editor endpoint in the customization guide.
python -m pip install .Choose a starting harness → Edit a candidate → Inspect the diff → Try it on tasks.
Start from a base or saved candidate, edit its files with an assistant and evaluate the result. See bring your own harness. The customization proposal command can also generate an unevaluated candidate from explicitly supplied feedback and text, with options to refine instructions from failure feedback, propose memory-backed skills and export a source-checked patch for review.
[!TIP] Already have a harness? Run
harnessgrad customize inspect --root ./my-harnessto see the components declared in its manifest, or see bring your own harness for the entrypoint and manifest format.
Install HarnessGrad and configure an OpenAI-compatible editor endpoint as described in the customization guide. Choose the files and feedback you want to share with that endpoint; for example, put the following in feedback.json:
[{"id": "review-1", "kind": "failure", "text": "The task ended without running the project tests."}]harnessgrad customize collect --root ./my-harness --skill skills/main.md --feedback ./feedback.json > request.json
harnessgrad customize propose --request request.json --max-calls 8 > proposal.json
harnessgrad customize review --request request.json --proposal proposal.json --root ./my-harnessReplace ./my-harness and skills/main.md with your own selected paths. The proposal includes candidate text and a diff for review; use the guide to export a patch and test it on a copy.
The same Console used to inspect experiments also provides a workspace for editing harness candidates. See the workbench guide for setup and sessions.
🌱 Evolve your harness — turn feedback into improvements to your own harness
Xiao Lü is one example of personal Harness evolution built with HarnessGrad. The desktop assistant shows how a correction from a session can become a scoped change proposal for review.
Use Python 3.12+ and install from the HarnessGrad checkout. To build the native desktop shell, follow the desktop setup guide.
python -m pip install .Open the local personal workspace with a synthetic session. Choose an existing project directory and a separate state directory outside the checkout and project.
harnessgrad personal demo --state-root /path/to/state --project /path/to/project
harnessgrad personal serve --state-root /path/to/stateOpen the printed local URL and select the service's connection.json in the browser. This example previews the workspace without model calls. For the desktop workflow shown above, see the personal evolution guide and the proposal preview in PR #64.
If the platform, the measurement protocol or a curve produced here is useful in your work:
@misc{xu2026harnessgrad,
title = {HarnessGrad: Harness as A Continual Learner},
author = {Buqiang Xu and Yenan Wang and Yutong Yu and Rongyao Li and
Jintian Zhang and Shuofei Qiao and Xuehai Wang and Zirui Xue and
Yijun Chen and Ningyu Zhang},
year = {2026},
}MIT. Third-party benchmark code keeps its own licence and provenance next to the code that uses it (PROVENANCE.json, LICENSE.upstream).














