Skip to content

About

A platform for running and comparing harness self-improvement methods.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

26 stars

Watchers

1 watching

Forks

Repository files navigation

HarnessGrad: Harness as A Continual Learner

Build agent harnesses that improve with feedback.
Run reproducible experiments, build for a new domain, or turn corrections into changes to your own harness.

🧭 Contents · 🧪 Experiments · 🛠️ Customize · 🌱 Evolve · 📚 Docs · 💬 Community · 📖 Citation

Quality checks Website deployment License: MIT Python 3.12 or newer

Project website Documentation Contributing guide Join the WeChat community

✨ Choose your path

ScienceWorld experiment curve
🧪 Run experiments
Follow how a harness changes across rounds.
Open experiments →
Ocean Harness showing research evidence
🛠️ Customize a harness
See a domain harness collect evidence and answer.
Open customization →
Xiao Lü desktop assistant alongside a chat
🌱 Evolve your harness
Turn a correction into a proposed change.
Open personal evolution →

🧭 Table of Contents

Tip

Expand a workflow to see what you can build with HarnessGrad and how to get started.

🧪 Run experiments — compare harnesses and RSI methods under recorded conditions

ALFWorld AppWorld ScienceWorld
alfworld panel-1 curve
alfworld panel-2 curve
appworld panel-1 curve
appworld panel-2 curve
scienceworld training curve
scienceworld validation curve

These curves come from experiments run with HarnessGrad. Click any panel to inspect the original chart.

🔬 From a question to an experiment

Choose a harness → Select an RSI method → Run tasks → Inspect the evidence. Use the same recorded conditions when comparing results, then explore the learning curve, versions and task trajectories in the Console.

Learning loop: tasks lead to feedback, editing the harness and evaluation, which returns to tasks.

🧩 Experiment integrations

A research run is one point in a harness × method × benchmark space: a harness solves tasks, a method proposes edits to that harness between rounds, and a benchmark scores each candidate. HarnessGrad provides tasks and scoring, isolated execution, and version control for candidate harnesses, with a recorded curve for each experiment.

[!NOTE] The inventory lists declared integrations; each harness and benchmark has its own preparation and runtime requirements.

Full harness, method and benchmark inventory

The table records the inventory at this README revision; the benchmark row is checked against the declared adapters.

axis what ships where
harnesses 8: loop, codex, opencode, deepseek-harness, openclaw, miniagent, miniswe, oracle base_harness/<name>/
RSI methods 19: dgm, rrsi, adas, ahe, ace, skillopt, evoprompt, evo_harness, opro, tthe, dream_rsi, sica_ci, gepa, darwinx, arbor, hyperagents, meta_harness, harnessx, modular_rsi experiments/method-<name>.json
benchmarks 56 declared adapters; preparation and runtime prerequisites vary — every one listed: agentdojo, aime, alfworld, appworld, authored_example_fix_bug, bbeh, bfcl_v3, bird_minidev, browsecomp, chessbench_puzzles, cooperbench, costbench, costbench_dynamic, deepsearchqa, deepswe, ds1000, fever, gaia, gsm8k, harvey_lab, hle, hotpotqa, jericho, jobbench, kumo, livecodebench, livemathbench, locomo, math, math500, memoryagentbench, musique, nq, officeqa_pro, personamem_v2, physicsquestions, reasoning_gym, scienceworld, searchqa, sportsunderstanding, spreadsheetbench, spreadsheetbench2, strategyqa, strongreject, supergpqa, swe_bench, swe_bench_pro, tau2_bench, tau3_bench, tau_bench, terminal_bench, travelplanner, triviaqa, twowikimultihopqa, widesearch, workbench data/datasets/, detail in DATASETS.md

Where to start

  • Bring a harness: supply a directory with harness.json and an entrypoint accepting --task and --workdir. See bring your own harness.
  • Bring a method: supply a program that reads a trajectory and writes a candidate tree. See bring your own method.
  • Bring a benchmark: write an adapter that materialises tasks and runs its official scorer. See dataset onboarding.

⚙️ Why one command is enough

A harness supplies harness.json and an entrypoint accepting --task and --workdir (interface). The platform runs dataset tasks and scoring, isolates execution, versions candidate harnesses, and records each run's outcomes and receipts. Dataset and harness prerequisites still need to be installed separately.

⚖️ Why the comparison is fair

Each round records the candidate harness hash and its changes, alongside the run's model roles and measurement metadata.

[!IMPORTANT] Compare results only when the task selection, scorer, models and other relevant conditions support it. Failed or incomplete measurements are not scores; do not substitute proxy scores under a benchmark name. See the measurement policy.

📦 Installation

Requires Linux, Python 3.12+, Git, uv, and a working bubblewrap. See the first-use guide for setup details.

git clone https://github.com/zjunlp/HarnessGrad.git
cd HarnessGrad
uv venv --seed --python 3.12 ~/.venvs/harnessgrad
source ~/.venvs/harnessgrad/bin/activate
uv pip install -e .
./install.sh --check

🗂️ Dataset storage

Set the shared external dataset directory in your local .env:

HG_DATA_ROOT=/path/to/datasets

See downloads for preparation and locations, and datasets for adapter details.

[!TIP] New to HarnessGrad? The first-use guide includes an installation control that does not need a model call or dataset download.

🚀 Quick start

Prepare Loop and GSM8K, then configure your OpenAI-compatible solver and editor endpoints. Replace the placeholders below and use a fresh run ID for each experiment. See the experiment guide for details.

# The provided configuration uses GSM8K.
python tools/configure_harness.py --harness base_harness/loop --fetch
python tools/datasets.py --fetch --dataset gsm8k
python tools/prepare_dataset.py --dataset gsm8k
python tools/datasets.py --check --dataset gsm8k
export HG_AGENT_BACKEND=openai
export HG_AGENT_BASE_URL='YOUR_SOLVER_ENDPOINT_URL'
export HG_AGENT_MODEL='YOUR_SOLVER_MODEL_NAME'
export HG_AGENT_API_KEY='YOUR_SOLVER_API_KEY'
export HG_METHOD_BASE_URL='YOUR_EDITOR_ENDPOINT_URL'
export HG_METHOD_MODEL='YOUR_EDITOR_MODEL_NAME'
export HG_METHOD_API_KEY='YOUR_EDITOR_API_KEY'
HG_QUICKSTART_ID="dgm-$(python -c 'import uuid; print(uuid.uuid4().hex[:12])')"
harnessgrad run --config experiments/method-dgm.json --run-id "$HG_QUICKSTART_ID" --check-only --check-net
harnessgrad run --config experiments/method-dgm.json --run-id "$HG_QUICKSTART_ID" --fail-on-incomplete
python tools/analysis/run_summary.py "runs/$HG_QUICKSTART_ID"

🖥️ Experiment workbench

Use the shared Console to inspect experiments, harness versions and task trajectories.

export HG_CONSOLE_RUNS="$PWD/runs"  # existing run-record directory
python console/server.py --port 4460

Open http://127.0.0.1:4460/ from the repository root, or point HG_CONSOLE_RUNS to another existing run-record directory. The Console binds to localhost by default.

Experiments view
Experiments
Harness versions view
Versions
Task trajectories view
Trajectories

🌱 Contributors

Thank you to everyone who contributes harnesses, methods, benchmarks, and improvements to HarnessGrad! The three charts show the most recent 30 days; each line includes contributions made before that window.

Want to join? Bring a harness, add a method, or onboard a benchmark, then open a pull request. We will add your name to the relevant contributor chart.

Harnesses: who contributed, by day

RSI methods: who contributed, by day

Benchmarks: who contributed, by day

🛠️ Customize a harness — build for a model, environment or task

Ocean Harness assembles research evidence and returns a sourced answer

Ocean Harness is one example built with HarnessGrad. It combines domain sources, research tools, and an answer workflow to answer ocean questions with citations. Build a harness for your own domain using the workflow below.

📦 Installation

Use Python 3.12+ and install from the HarnessGrad checkout. For model-generated edits, configure an editor endpoint in the customization guide.

python -m pip install .

Current capabilities

Choose a starting harness → Edit a candidate → Inspect the diff → Try it on tasks.

Start from a base or saved candidate, edit its files with an assistant and evaluate the result. See bring your own harness. The customization proposal command can also generate an unevaluated candidate from explicitly supplied feedback and text, with options to refine instructions from failure feedback, propose memory-backed skills and export a source-checked patch for review.

[!TIP] Already have a harness? Run harnessgrad customize inspect --root ./my-harness to see the components declared in its manifest, or see bring your own harness for the entrypoint and manifest format.

🚀 Quick start

Install HarnessGrad and configure an OpenAI-compatible editor endpoint as described in the customization guide. Choose the files and feedback you want to share with that endpoint; for example, put the following in feedback.json:

[{"id": "review-1", "kind": "failure", "text": "The task ended without running the project tests."}]
harnessgrad customize collect --root ./my-harness --skill skills/main.md --feedback ./feedback.json > request.json
harnessgrad customize propose --request request.json --max-calls 8 > proposal.json
harnessgrad customize review --request request.json --proposal proposal.json --root ./my-harness

Replace ./my-harness and skills/main.md with your own selected paths. The proposal includes candidate text and a diff for review; use the guide to export a patch and test it on a copy.

Customization workbench

The same Console used to inspect experiments also provides a workspace for editing harness candidates. See the workbench guide for setup and sessions.

Harness editor view
Harness editor

🌱 Evolve your harness — turn feedback into improvements to your own harness

Xiao Lü demonstrates turning personal feedback into a proposed change

Xiao Lü is one example of personal Harness evolution built with HarnessGrad. The desktop assistant shows how a correction from a session can become a scoped change proposal for review.

📦 Installation

Use Python 3.12+ and install from the HarnessGrad checkout. To build the native desktop shell, follow the desktop setup guide.

python -m pip install .

🚀 Quick start

Open the local personal workspace with a synthetic session. Choose an existing project directory and a separate state directory outside the checkout and project.

harnessgrad personal demo --state-root /path/to/state --project /path/to/project
harnessgrad personal serve --state-root /path/to/state

Open the printed local URL and select the service's connection.json in the browser. This example previews the workspace without model calls. For the desktop workflow shown above, see the personal evolution guide and the proposal preview in PR #64.

📖 Citation

If the platform, the measurement protocol or a curve produced here is useful in your work:

@misc{xu2026harnessgrad,
  title  = {HarnessGrad: Harness as A Continual Learner},
  author = {Buqiang Xu and Yenan Wang and Yutong Yu and Rongyao Li and
            Jintian Zhang and Shuofei Qiao and Xuehai Wang and Zirui Xue and
            Yijun Chen and Ningyu Zhang},
  year   = {2026},
}

🔗 Related work

📜 License

MIT. Third-party benchmark code keeps its own licence and provenance next to the code that uses it (PROVENANCE.json, LICENSE.upstream).

About

A platform for running and comparing harness self-improvement methods.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

26 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages