Improving Your AI Coding Agent Without switching the Model or touching the harness

When an AI coding agent underperforms, most engineering teams reach for one of three levers:

1. Upgrade to a more capable model.
2. Add more context.
3. Build a more sophisticated agent harness.

All three can work. All three can also become expensive.

I recently came across a Microsoft research project called SkillOpt that suggests another approach: keep the model and harness unchanged, and systematically improve the skill that guides the agent.

SkillOpt treats a Markdown skill file almost like the parameters of a machine learning model. It runs tasks, studies failures, proposes small edits, validates them, and retains only the changes that improve performance.

No model fine tuning. No weight updates. No additional optimizer calls after the skill is deployed.

I think this points toward an emerging engineering discipline: SkillOps, the practice of measuring, optimizing, versioning, and deploying the procedural knowledge used by AI agents.

A skill is more than a prompt

A prompt usually describes the task directly in front of the agent.

A skill describes how the agent should repeatedly approach an entire class of tasks.

It might contain:

• Codebase conventions
• Tool selection policies
• Testing and verification steps
• Known failure modes
• Output requirements
• Rules about files or systems the agent must not modify

For example, an enterprise coding skill might tell an agent:

Before modifying a service:

1. Identify the owning module and its contract tests.
2. Use the internal API client instead of calling the service directly.
3. Never edit generated client files.
4. Run unit, contract, and dependency boundary tests.
5. Reopen the diff and verify that no unrelated files changed.

A human can write this file, but we are still guessing what the model actually needs.

SkillOpt replaces that guesswork with an evaluation loop.

How SkillOpt works

SkillOpt improves a skill while keeping the target model frozen.

The process looks like this:

1. The target model performs a batch of tasks using the current skill.
2. The system records its answers, tool calls, command output, and verifier results.
3. A separate optimizer LLM analyzes successful and failed trajectories.
4. It proposes a bounded set of additions, deletions, or replacements.
5. The modified skill is tested against a held out validation set.
6. The change is accepted only when it improves the validation score.

Rejected edits are not simply discarded

SkillOpt retains rejected edits as negative feedback. This helps the optimizer avoid repeating changes that sounded reasonable but reduced actual performance.

This distinction matters.

The optimizer LLM can propose what might improve the skill, but it does not get to declare that its own suggestion worked. The validation results make that decision.

That is much closer to software engineering than prompt tweaking:

Propose. Test. Measure. Accept or roll back.

Is the skill optimized for one LLM or every LLM?

A SkillOpt training run is anchored to a specific combination of:

• Target model
• Task or benchmark
• Execution harness
• Evaluator

The target model performs the tasks. The optimizer model studies its trajectories and edits the skill. The candidate skill is then evaluated using the same target model and harness.

The baseline is therefore specific to the model and environment. SkillOpt does not initially optimize one universal skill against every LLM.

However, the resulting skill may still transfer to other models and coding tools. That transfer is tested separately rather than assumed.

This is a useful enterprise design. Optimize against the model and coding tool your teams actually use, then test whether the resulting artifact transfers safely to other environments.

The results are significant

Across six benchmarks, seven target models, and three execution modes, SkillOpt was best or tied for best in all 52 evaluated model, benchmark, and harness combinations.

With GPT-5.5 in direct chat, average accuracy increased from 58.8% to 82.3%, an improvement of 23.5 percentage points.

Some of the largest gains appeared on procedural tasks:

Benchmark	Without optimized skill	With SkillOpt
SpreadsheetBench	41.8%	80.7%
OfficeQA	33.1%	72.1%
LiveMathematicianBench	37.6%	66.9%

Inside agentic harnesses, the average improvement over using no skill was:

• 24.8 percentage points for Codex
• 19.1 percentage points for Claude Code

These are exactly the kinds of tasks where raw intelligence is not enough. The agent must follow a process, use tools correctly, preserve state, and verify its output.

A smaller model with a better skill can beat a larger model

One of the most interesting findings appears when comparing results across models.

GPT-5.4-mini with SkillOpt outperformed the larger GPT-5.4 model without a skill on four of the six direct chat benchmarks:

Benchmark	GPT-5.4 without a skill	GPT-5.4-mini with SkillOpt
SearchQA	76.9	80.2
SpreadsheetBench	41.4	47.5
DocVQA	77.6	90.9
ALFWorld	75.4	85.8

This does not mean a smaller model will replace a larger model for every task.

It demonstrates something more practical: within a bounded enterprise workflow, the correct operating procedure can matter more than additional model capacity.

That creates an interesting cost model.

A team could use a powerful optimizer during an offline training process, deploy the resulting skill with a cheaper target model, and pay no optimizer cost during normal agent execution.

Skills can transfer across coding tools

The paper evaluated the same portable skill artifact in direct chat, Codex, and Claude Code.

The strongest transfer result came from SpreadsheetBench. A skill trained inside Codex was moved to Claude Code without further optimization.

Claude Code’s performance increased from 22.1% to 81.8%, slightly exceeding the result from the skill optimized directly inside Claude Code.

That does not prove every skill will work across every coding assistant. Tool APIs, context formats, and execution behavior still differ.

The official project currently includes integrations or plugin shells for Claude Code, Codex, GitHub Copilot, Devin, and OpenClaw. Cursor integration has been requested, but it is not currently listed as a shipped integration.

The safe enterprise approach is straightforward: treat skill portability like application portability. Reuse the same artifact where possible, but validate it separately in every target environment.

SkillOpt Sleep makes this operational

The original paper describes an offline optimization process. The newer SkillOpt 0.2 release extends the idea into something closer to continuous improvement.

SkillOpt Sleep can run a nightly cycle:

Harvest sessions
    ↓
Identify recurring tasks
    ↓
Replay them offline
    ↓
Propose bounded skill changes
    ↓
Validate against held out tasks
    ↓
Stage the improved skill for review

It includes experience replay, optional synthetic “dream” rollouts, rewards for accuracy, latency, and token usage, and long term memory.

More importantly, proposed changes are staged for review rather than silently replacing the deployed skill.

Imagine applying this to an enterprise codebase.

During the day, developers use coding agents to fix defects, add APIs, and update tests. At night, the system examines repeated failure patterns:

• The agent keeps modifying generated files.
• It misses a required security scan.
• It uses a public SDK instead of the company wrapper.
• It fails to run contract tests after changing an event schema.

The optimizer converts those failures into candidate procedural rules.

Those rules are deployed only when they improve performance on tasks the optimizer did not use to create them.

That is institutional learning captured as a small, inspectable text artifact.

Things to watch out for

This is not free optimization

SkillOpt avoids model training, but its offline optimization process can still consume substantial compute and tokens.

The paper’s GPT-5.5 experiments used between 20.8 million and 213.8 million training tokens, depending on the benchmark.

The final deployed skills, however, contained only 379 to 1,995 tokens and required between one and four accepted edits.

This makes SkillOpt most attractive when:

• The task occurs frequently.
• Success can be measured reliably.
• The optimized skill will be reused many times.
• Model inference or engineering mistakes are expensive.

It is less compelling for one time tasks or subjective work without a reliable evaluator.

Dependable tests, executable checks, or scoring signals are essential because they determine whether a proposed change is accepted or rejected.

How to try this in an enterprise workflow

Do not begin with a broad goal such as, “Make our coding agent better.”

Choose one narrow, recurring workflow with a clear answer key.

Good starting points might include:

• Upgrading a service dependency
• Fixing a known category of defects
• Generating database migrations
• Implementing APIs from an internal specification

Collect historical tasks and divide them into training, validation, and test sets.

Define success using signals such as:

• Unit and integration tests
• Static analysis
• Security rules
• Token usage
• Execution time
• Human review requirements

First, measure the current model and harness without an optimized skill.

Then run SkillOpt, evaluate the resulting artifact against untouched tasks, and publish the skill using the same controls you use for code:

• Version history
• Pull request review
• Automated tests
• Staged rollout
• Rollback support

The model provides general intelligence.

The harness provides tools.

The skill provides your organization’s operating procedure.

We have spent enormous energy improving the first two.

With SkillOpt, the third can now be trained too.