Reusable guidance is not automatically good guidance
Agent skills are meant to save an AI system from rediscovering the same procedure every time. A skill can tell an agent how to inspect a repository, use a tool, build an output and check its work.
That sounds helpful. It often is. A new study also shows the other side: a skill that looks relevant can push an agent into a wrong implementation, a misplaced file or a lot of unnecessary work.
Researchers from Huazhong University of Science and Technology, Microsoft Research, Microsoft and the University of Illinois Urbana-Champaign isolated 307 cases in which the loaded skill changed the outcome for the worse.
The lesson is not to stop using skills. It is to treat them as active configuration, not harmless documentation.
How the researchers isolated the skill
A failed run does not prove that a skill caused the failure. The model may simply be inconsistent, the task may be too hard or the test may miss something important.
The team used paired comparisons to narrow that problem. The task, verifier, agent framework, model and repository state stayed fixed. One run used the skill being audited; the reference run used no skill or another semantically matched skill that completed the same task.
They started with SkillsBench and SWE-Skills-Bench, then added public skills matched by name and description. From 20,664 possible pairs, the executed evaluations produced 665 candidates. After removing ambiguous, narrow and duplicate cases, the researchers kept 307 for analysis.
Of those, 125 were functional failures: the target run failed while its reference passed. Another 182 were efficiency regressions: both runs passed, but the skilled run used more time and tokens, with at least one of those costs more than doubling.
Most failures came from plausible instructions
Obviously irrelevant skills were not the main problem. In 86 of the 125 functional failures, the loaded guidance led the agent to implement a required field, API behaviour, calculation, output format or domain rule incorrectly, or leave it out.
Another 24 failures put a plausible artifact in the wrong place. Thirteen came from an environment mismatch, such as validating against a package or runtime state that the task's verifier did not share. Only two were classified as a basic applicability mismatch.
That makes these failures awkward to catch by topic alone. A skill can be about the right subject and still contain a path, template, dependency or default that does not fit this particular job.
The practical check is closer to a compatibility review: what does this task require, what assumptions does the skill add, and where do the two conflict?
The hidden cost was extra procedure
Longer prompts played a part in some expensive runs, but they were not the main explanation. Excessive procedure accounted for 114 of the 182 large efficiency regressions.
Repeated verification was the biggest group, with 67 cases. Heavy implementation pipelines added another 30. A checklist or construction recipe that was useful in one setting became mandatory work in another.
This is a familiar automation problem. A rule designed to make the system careful can become wasteful when it ignores the size, risk or uncertainty of the task in front of it.
The authors argue for cost-aware skill loading and verification budgets. In plain terms: choose the instruction only when it is likely to help, and let the agent scale its process to the job rather than running the full ritual every time.
What is confirmed, found and still open
Confirmed: the paper was submitted to arXiv on 12 August 2026. Its controlled runs used OpenCode 1.15.1 with Claude Opus 4.6 across tasks drawn from SkillsBench and SWE-Skills-Bench.
The research finding: the authors retained 307 skill-induced problems under their paired method. Most functional failures came from task-implementation faults, while most large cost regressions came from excessive procedure rather than prompt length alone.
The authors' proposal: agent platforms should check skill-task compatibility, estimate the cost of loading a skill and adapt verification depth to the task and remaining budget.
Still open: whether the same distribution appears with other models, agent harnesses, skill stores and real work outside deterministic benchmarks. The final root-cause labels involved manual judgement, and the reference run is a useful comparison, not mathematical proof of causation.
A skill can make an agent more capable. This study is a reminder that adding instructions also adds another thing that needs testing.
Sources
- Dong et al. - Agent Skills Can Be HarmfulPrimary paper record submitted 12 August 2026. Source for authorship, affiliations, study scope and headline counts.
- Dong et al. - Full HTML manuscriptFull primary manuscript. Source for the paired methodology, benchmark expansion, taxonomies, numerical results and threats to validity.
- Dong et al. - Fixed PDF manuscriptFixed author manuscript used to verify tables, affiliations, experimental setup, results and limitations.
