WikiSkill

The WikiSkill framework is a method for improving agent skills from repeated task experience. It separates three kinds of state that many self-improving systems mix together: raw execution trajectories, accumulated diagnostic knowledge, and the concise skills supplied to an agent during ordinary work.

The separation lets WikiSkill retain knowledge from a failed skill update without retaining the update itself. Candidate skills must improve validation performance before they become active. The wiki persists across accepted and rejected proposals, so later proposals can use the evidence and avoid repeating a failed intervention. Across five benchmarks and five models, the paper reports higher average test scores than three other skill-evolution methods. One Gemini-3.5-Flash ablation over four benchmarks found that the complete wiki pipeline helped skill development most when only the proposer could read the wiki during training.

Three layers with different jobs

WikiSkill begins each benchmark with no skills and an empty wiki. Its workspace then develops three layers:

Layer	Contents	Role in the loop
Raw	Immutable task trajectories with reasoning, actions, tool output, and final answers	Preserves exact evidence from training rollouts
Wiki	Pattern pages, an index, an evolution log, and a skill-impact record	Consolidates recurring successes, failures, prior proposals, and validation outcomes
Skills	Active SKILL.md files plus PURPOSE.md provenance files	Supplies concise procedural instructions to the task-performing agent

The raw layer answers what happened. The wiki records what the system currently thinks the experience means. The skill layer contains the procedure that has earned permission to affect later task execution.

That distinction gives rollback a useful asymmetry. A regressing skill patch disappears from the active skill set, but the skill-impact record keeps the proposal's unified diff, score, and rejection outcome. Rollback removes behavior without erasing the learning signal.

The evolution loop

Each WikiSkill iteration runs the following sequence:

1. The inference agent executes the complete training split with the active skills injected into its system prompt. It cannot read the wiki.
2. The wiki maintainer receives the existing wiki and at most eight sampled trajectories: up to five failures and three successes. Each trace is capped at 15,000 characters.
3. The maintainer creates or patches pattern pages, rewrites the pattern index, and appends an iteration summary.
4. The skill proposer starts with the wiki index, the skill-impact history, and a compact list of training outcomes. It can inspect selected pattern pages and raw traces through a file-reading tool.
5. The proposer submits one atomic action: create one skill, patch one skill, or make no change.
6. The system evaluates the candidate skill set on a validation split. It accepts only a strict score improvement; otherwise, it restores the previous skills.
7. The outer loop appends the proposal diff, score, and decision to the wiki whether the proposal passed or failed.

The proposer uses a ReAct-style search process rather than receiving every trace in one prompt. Its published prompt requires inspection of at least four trajectories before proposing a change. Under the reported full-batch setup, each iteration used one wiki-maintainer call plus roughly 10 to 20 proposer turns. The evaluated training splits contained only 16 to 80 examples, so the experiments do not establish whether proposer turns remain constant at larger scales. Token volume, trajectory generation, and validation work also contribute to total cost.

Reported results

The evaluation covers recent mathematics questions, scholarly web search, spreadsheet editing, long-context Treasury-document questions, and text-based household tasks. Training splits contain 16 to 80 examples, validation splits contain 10 to 40, and test splits contain 85 to 280. The authors repeat each complete evolution run three times. They use 1,000-iteration paired bootstrap tests over test instances and mark top-method comparisons at $p < 0.05$.

The following table reports the macro-average score across all five benchmarks. “Strongest comparison” is the best average among Trace2Skill, EvoSkill, and SkillOpt for that model in the paper's table.

Inference model	No skill	Strongest comparison	WikiSkill	Gain over no skill
Qwen-3.5-4B	26.2	35.2	38.5	+12.3
Qwen-3.5-9B	29.9	42.3	47.4	+17.5
Qwen-3.6-27B	39.4	53.3	63.3	+23.9
Gemma-4-31B	41.3	49.1	54.9	+13.6
Gemini-3.5-Flash	49.5	56.1	68.1	+18.6

The gains vary sharply by task. Qwen-3.5-4B with WikiSkill falls below its no-skill score on OfficeQA, while Qwen-3.6-27B gains 40.9 points on SpreadsheetBench. Gemini-3.5-Flash begins ALFWorld with a perfect validation score, so its evolution loop stops before producing a skill for that benchmark.

The paper also transfers evolved skills among models. Cross-model transfer often beats self-evolved skills, which separates two capabilities that a self-improvement loop normally bundles together: discovering a useful procedure and executing it reliably. Transfer can also fail badly. A spreadsheet skill evolved by Qwen-3.5-4B lowers Gemini-3.5-Flash from 50.5 to 18.1, while the Qwen-3.6-27B skill raises the same model to 63.4. The authors' trajectory analysis suggests that small-model workarounds constrained the stronger model and consumed its interaction budget.

The wiki ablation

The clearest test isolates who can read the wiki during skill evolution. The ablation uses Gemini-3.5-Flash on four benchmarks and excludes ALFWorld.

Inference agent reads wiki	Skill proposer reads wiki	Average score
No	No	48.7
Yes	No	45.3
Yes	Yes	60.9
No	Yes	63.7

Giving the proposer the persistent wiki raises the average from 48.7 to 63.7 when the inference agent cannot read it. This comparison changes two components together: when the proposer has no wiki access, the experiment also removes the wiki maintainer and therefore the persistent knowledge layer. Letting the inference agent read the maintained wiki lowers the result from 63.7 to 60.9 when the proposer already has access.

The authors hypothesize that wiki access lets the inference agent solve training tasks from information that never reaches the skill. Those trajectories then overstate what the active skill can do. The default design keeps the richer evidence available to the optimizer and gives the task-performing agent only the validated procedure.

Architectural reading

The paper's reported benchmark experiments provide evidence for retaining a separate, evolving knowledge representation during skill optimization. The result goes beyond keeping every proposal in a flat log. Pattern pages can combine evidence across iterations, the index supports selective inspection, and the skill-impact record ties interventions to outcomes. The work has not yet been independently reproduced.

Co's architectural reading is that the wiki acts as a compilation layer. Raw trajectories remain the source material, wiki pages are revisable intermediate representations, and skills are executable releases. Each layer has a different retention and evaluation policy. This resembles a versioned context repository connected to agent trajectory observability, but WikiSkill evaluates whether that structure improves benchmark-specific skill evolution rather than whether it supports a complete long-lived agent.

The proposer-only access result also sharpens agent memory. Useful memory depends on which role can retrieve it and at which phase of work. Supplying more retained context to every model call can make training trajectories less informative about the active skill, even when the retained context is accurate.

Evidence limits

WikiSkill is an August 2026 preprint. As of August 31, 2026, the reviewed paper and arXiv record did not link a public implementation. The reported comparisons therefore depend on the paper's description, prompts, and tables rather than an independently rerunnable release.

The benchmark design leaves several questions open:

• Active skills are injected in full, so the study does not test skill discovery, triggering, or progressive disclosure as a skill library grows.
• Every accepted patch must immediately improve a small validation split. Validation sets contain 10 to 40 tasks, which makes selection noise and repeated validation reuse important concerns. Three independently evolved skill sets and bootstrap resampling over test instances do not capture every source of optimizer-run uncertainty or remove adaptive validation pressure.
• The paper counts optimizer API calls, but call count does not equal token use, latency, or cost. WikiSkill's 10 to 20 proposer turns may be expensive even when the asymptotic call count is constant in training-set size.
• The evaluated loops run for eight iterations and do not cover tasks lasting hundreds of actions or several hours.
• The wiki has no automatic pruning or contradiction-resolution mechanism. Continued accumulation can eventually make retrieval and maintenance harder.
• The 27B Qwen result changes both parameter scale and model generation relative to the 4B and 9B models, so the Qwen trend is evidence that stronger evaluated models benefit more, not a controlled scaling law.

A production system would also need minimization, redaction, access-control, retention, and deletion rules for raw trajectories. Reasoning and tool output can contain credentials, personal data, proprietary documents, or sensitive third-party material. WikiSkill stores these records because they are useful optimization evidence; the paper studies performance rather than the custody of that evidence.

Longer-running deployments still need to show that the wiki remains searchable, correctable, private, and useful after far more than eight iterations. The paper's design makes that question inspectable because raw evidence, revisable interpretation, and active procedure remain separate objects rather than one continually rewritten prompt.