给这个季度安装了三个技能，并且交付了自己引以为豪的工作的我们：这篇文章不会告诉我们这么做是错的。
Claude Code 的技能生态在 2025 年 10 月到 2026 年第一季度之间迅速成熟。Anthropic 在 2025 年 10 月推出 Agent Skills；到 12 月，一项开放标准 —— SKILL.md 格式 —— 已经成了 Claude Code、Cursor 和 Agent SDK 之间的跨平台惯例。 第三方索引如今列出的技能已经超过 85,000 个。Hacker News 上那条题为“Claude Skills are awesome, maybe a bigger deal than MCP”的帖子，在滚出首页之前拿到了 738 分和 370 条评论。 那些一边点头一边安装看起来有用的技能的读者，做的正是这个生态被设计来促成的事 —— 而且在多数情况下，它确实有效。
本文并不是在论证技能不好。它也不是在论证任何人都应该少用 AI，不是在说下载专业知识天然就是欺诈，也不是在加入那场已经排练到第四轮的初级开发者技能退化恐慌。这里不是那个问题。那些文章大多谈的是学习，其中的大部分担忧是关于一些初级开发者在还没有形成基础之前就伸手去拿工具。那是真问题。但不是本文的问题。
范围条件必须先说清楚。低风险、快反馈的领域，正是技能真正有帮助的地方，包括那些基础还薄的用户。如果技能产出了糟糕结果，反馈回路很快就能抓住它。失败是可见的。专业能力的民主化实际上就是这样发生的，而且它确实有效。本文关心的是它的补集：高风险、慢反馈的领域，在这些领域里，失败要到变得昂贵时才会显现；而用户在技能实际运作的那个具体领域中的评估能力，可能低于他们的自信所暗示的水平。
本文谈的是尾部情形 —— 在哪些条件下，技能不再放大我们的判断，而开始放大别的东西 —— 以及为什么一项近期发现，如果细读，会提示我们：这个尾部比多数人以为的更宽。
1. 主线确实有效，而且有数据这样说#
关于 AI 辅助软件工作的正面论证，比多数怀疑论文章承认的更强；也比我为了接下来要提出的论点而希望它强的程度更强。
在 Anthropic 主导的一项随机对照试验中，研究者考察开发者如何在 AI 辅助下学习一个新库。AI 辅助参与者中的一个子组 —— 那些把工具用于概念探询的人 —— 在理解测验中的得分约为 86%。 另一个采用代码生成与解释混合方法的组，得分约为 80%。 两个结果都达到或超过了手工编码控制组约 70.9% 的得分。 这不是模糊说法。这是论文自己的数据，并且按产生这些结果的行为拆开来看：当 AI 辅助作为思考工具而不是代码生成器运作时，它产生了比独自工作更好的理解结果。
在实验室之外，Anthropic 自己的企业案例也报告了足以支撑乐观解读的压缩幅度。Thomson Reuters 把法律研究从数小时压缩到数分钟。eSentire 把威胁分析从五小时压缩到七分钟，输出与资深专家判断的一致性最初测得为 95%，在持续生产中也确认高于 90%。 这些不是玩具任务。它们是在真实风险下运行的生产工作流。民主化叙事 —— 可下载的专业能力可以在资深判断稀缺的领域替代稀缺的资深判断 —— 在这些案例中确实有证据支持。
脚手架式辅助这个框架也站得住。当 AI 辅助被用于探询时 —— 当用户在追问、探查解释、建立模型，而不是接受一个结果时 —— 它就像学习研究者所说的“更有知识的他者”（More Knowledgeable Other）：一种扩展用户最近发展区的资源，而不是绕过它。风险不是在这里出现的。风险出现在另一种模式里。
产生那些 86% 理解分数的同一项研究，也产生了本文要转向的发现。
2. 真实、有记录，而且很可能也关于我们#
2025 年 7 月到 2026 年 2 月之间发表的三项发现，单独看都不算惊人，合在一起却指向同一个方向：AI 辅助开发者实际产出的东西，与他们以为自己产出的东西之间，存在差距。
第一项来自 Fernandes 等人，2026 年 2 月发表于 Computers in Human Behavior。 在两项研究中 —— N=246 和 N=452 —— 参与者在可以使用 ChatGPT-4o 的情况下解答 LSAT 逻辑推理题。AI 使用让任务表现比基线提高了约 3 分。它也让自我估计表现提高了 4 分。净结果是：即便把真实提升计入之后，仍然存在系统性的信心盈余。表现提高了；对表现的感觉提高得更快。这个差距在两项研究中都得到了复现。
这一发现与长期退化的可能性相一致，但不足以证明长期退化。Fernandes 的设计是单次会话 —— 一组题目、即时自评、没有后续追踪。作者自己也在局限性部分明确写道，他们的设计“无法回答过度自信是否会随着时间损害学习”。LSAT 语境也限制了跨领域推断的范围。这项研究所确立的 —— 并且通过复现确立的 —— 是一次性元认知差距真实存在且可以测量。
第二项发现同样来自 Fernandes，对本文的特定读者来说更令人不舒服：在两项研究中，参与者在 SNAIL AI 素养量表上的得分，与元认知偏差呈正相关。 技术上最了解 AI 的用户，反而犯下最大的自我评估错误。相关性并不大 —— r = 0.21，约解释 4% 的方差 —— 但在两项研究中都有统计显著性。 这是一个方向上反直觉的信号，不是戏剧性的发现。可以把它当成一个探针式问题，而不是一条已经确立的定律：如果我们在一次 LSAT 会话中看到的模式，只是某种会复利增长的东西的尖端呢？
第三个数据点是 METR 2025 年 7 月的随机对照试验。 16 名经验丰富的开源开发者，在自己的代码库上完成了 246 个真实任务，条件随机分配为允许使用 AI 或禁止使用 AI。使用 AI 的开发者 —— Cursor Pro 搭配 Claude 3.5 和 3.7 Sonnet —— 完成任务所用时间反而多了 19%。他们相信自己快了 20%。在研究开始前，他们预期会提速 24%。
限定条件必须说明，因为成熟读者已经知道：N=16 很薄。代码库设置 —— 开发者在自己 100 万行级别的仓库上工作 —— 最大化了 AI 的上下文劣势。工具也是 2025 年初那一代。METR 自己在 2026 年 2 月的后续文章中明确说，他们的新实验设计“给出了一个关于当前 AI 工具生产力影响的不可靠信号”，并补充说“我们的数据只是关于这种提升规模的非常弱证据”。 METR 没有撤回 2025 年 7 月的数字，但他们已经降低了自己对这些数字能够说明当前 AI 工具什么的信心。请把这个数据点当作趋同模式中的一个信号，而不是决定性证据。
模式依然趋同。Stack Overflow 2025 开发者调查 —— 最大的年度开发者调查，受访者超过 60,000 人 —— 发现 84% 的开发者正在使用或计划使用 AI 工具。 最主要的挫败感，由 66% 的开发者提到，是“AI 方案差一点就对，但就是不完全对”。 这就是元认知差距的自我报告版本：工具产出的东西足够像真的，于是日常摩擦不是不存在的 API 这类幻觉，而是“差不多对”和“对”之间的间隔。
这三项研究各自都没有解释，差距在哪些用户身上最大 —— 答案来自那篇人人都浏览过却没有细读的研究。
3. 转折：Anthropic 那篇论文真正说了什么#
我们多数人记住的 Anthropic 数字 —— AI 辅助组理解测验 55.6%，手工组 70.9%，相差 17 个百分点 —— 是真的，但它也是错误的分析单位。
这个数字隐藏了什么？Shen 和 Tamkin 分析了 AI 条件下 26 名参与者中 25 人的屏幕录像（一份录像不可用），识别出六种不同的使用模式。作者明确把这标记为“探索性数据分析（非预注册）” —— 这些是基于屏幕录像的定性分析事后得出的类别，每个单元格的样本量在 2 到 7 名之间。单元格很小，发现具有方向性，但没有在子组层面得到统计验证。请把这些限制放在眼前。
六种模式分成两个层级，二者没有重叠：
高分模式（测验得分 65-86%，达到或高于 70.9% 的手工控制组）：
- 概念探询（Conceptual Inquiry）—— n=7，约 86%：参与者“只问概念性问题，并依靠自己提升后的理解来完成任务”
- 代码-解释混合（Hybrid Code-Explanation）—— n=3，约 80%：在请求生成代码的同时请求解释
- 先生成、后理解（Generation-Then-Comprehension）—— n=2，约 75%：先生成代码，再追问后续问题
低分模式（测验得分 24-39%）：
- AI 委托（AI Delegation）—— n=4，约 24%：“完全依赖 AI 编写代码并完成任务；这一组完成任务最快，过程中很少遇到错误或没有遇到错误”
- 渐进式 AI 依赖（Progressive AI Reliance）—— n=4，约 28%：逐步转向完全委托
- 迭代式 AI 调试（Iterative AI Debugging）—— n=4，约 39%：依赖 AI 解决错误，而不是澄清理解
图：按 AI 使用模式划分的理解测验得分 —— Shen & Tamkin（Anthropic），arXiv:2601.20245。水平线标出手工编码控制组得分（70.9%）。
论文自己的表述是：“高分模式通常涉及更多认知努力和更少 AI 依赖。”
17 个百分点的组级差距，完全是低分层的属性。高分层并没有比控制组低 17 分。它大约高出低分层天花板 15 分，并且达到或超过控制组。把委托子组抽出来，组级发现就翻转了。
这当然仍然是探索性的、事后的。每个单元格的 n 值 —— 每类 2 到 7 人 —— 小到足以让点估计很不精确。模式描述是定性的。没有预注册分析验证类别边界。严谨读者应当相应地给它权重。
可是：这个分裂方向上的差距宽达 30 到 45 个百分点；在每个层级的每个单元格中方向都一致；而且它与每个实际工作的开发者已经直觉知道的 AI 助手两种用法相吻合。我们可以要求它解释一个概念，直到我们理解。也可以要求它写代码，然后交付输出。
AI 辅助组并没有低于控制组。AI 辅助组的一半低于控制组 30 到 45 分。另一半持平或超过。区分它们的变量不是工具，而是模式。
4. 桥：从形成到使用，从模式到制品#
下一步是本文推论性最强的一步，所以我想把它明白标出来，而不是偷偷带过。
Anthropic 的研究测量的是技能形成：从未使用过 Trio 库的参与者用它工作 35 分钟，然后参加理解测验。没有被测量的是：一个已经熟悉自己领域的从业者，为生产用途安装一个 Claude Skill 时会发生什么。论文作者自己也指出，这是“本研究没有解决的一个重要问题”。 从形成阶段到生产阶段的迁移没有被测量；把 Shen 和 Tamkin 的课堂连接到读者的工作仓库，是我的论证，不是他们的论证。
这个论证依赖机制：一个从业者在相邻于其核心专长的领域使用技能时，相对于那个领域仍处在持续形成阶段，即便他在别处已经是资深专家。Anthropic 研究的形成阶段发现，适用于任何用户对技能所属领域的理解正在实时形成的场景 —— 而这正是相邻领域专家这个案例所描述的场景。
SKILL.md 格式的设计目的，是以结构化指令的形式给 Claude 提供它先前没有的能力，让从业者安装并运行。 Anthropic 自己的文档把 Skills 描述为能够让 Claude 代表用户“调用工具或执行代码”。 标准的技能安装工作流 —— 找到技能、安装技能、使用技能 —— 没有任何摩擦点会询问用户：我现在处于探询模式，还是委托模式？格式中没有任何东西要求在执行之前先进行概念参与；生态经济也没有补上这一点：技能的营销和排名依据的是压缩 —— “数小时到数分钟”，“5 小时到 7 分钟” —— 而不是有助于理解的脚手架。 这个制品针对的是委托默认值，因为安装时的奖励函数是速度，而不是理解。它最终以探询模式还是委托模式抵达用户，完全取决于用户 —— 而用户并不会被提示去选择。
图：两种反馈回路。探询模式（左）保留理解摩擦 —— 用户的评估能力在每次迭代中都被调动。委托模式（右）移除这种摩擦 —— 任务完成了，但理解从未被测试。Anthropic 子组发现（第 3 节）就是哪条回路产生哪种结果的实证记录。
是的，这些正是我刚才用来证明主线有效的数字。重点就在这里：这个生态中最可见的成功，也是这个生态最清晰的激励 —— 被奖励的是压缩，而不是让压缩变得安全的理解脚手架。
有一个限定。Anthropic 的探询模式子组并没有任何技能文件摩擦；他们使用的是普通聊天界面，并且选择了概念性参与。因此，本文的主张比“技能导致委托”更窄。主张是：在自由聊天语境中已经只是可变存在的、通向探询的默认摩擦，在技能封装的语境中更低；而生态的奖励函数并没有做任何事来重新引入这种摩擦。机制是激励，不是决定论。
这就是 Shen 和 Tamkin 的课堂与我们今早打开的仓库之间的联系 —— 而且说清楚，这是我正在提出的联系，不是他们测量过的联系。
5. 缺失的追缴保证金机制，现在给出定义#
追缴保证金通知不是警告，也不是可选项。
根据 FINRA 规则 4210 和 SEC Regulation T，保证金账户必须满足强制性的 25% 维持保证金要求。 当亏损使账户权益跌破该阈值时，经纪商有权在不提前通知账户持有人的情况下平仓。投资者不能决定止损是否触发。经纪商可以根据自己的裁量执行清算，不受投资者对该头寸看法的影响。止损是结构性的、法律强制的，不能被持有杠杆头寸的人覆盖。
图：金融杠杆会在账户权益低于监管阈值时触发强制止损（FINRA Rule 4210）。技能杠杆没有等价机制。缺乏技能所在领域评估能力的从业者无法发现错误 —— 而这正是技能一开始显得有用的那个条件。
追缴保证金通知不是财务谨慎的隐喻。它是基础设施。金融杠杆危险，于是金融系统的回应，是把这种危险写进架构：一个在亏损超过杠杆方事先同意的阈值时自动启动的断路器。
技能杠杆因类似原因而危险 —— 它会放大从业者已经持有的任何立场，无论正确还是错误 —— 但这里没有类似基础设施。没有阈值触发。没有外部方监控从业者判断力的“权益”，并在它跌破维持要求时强制平仓。从业者可以继续以工具速度持有一个自信但错误的立场，以技能所允许的速度产出貌似合理的输出，而没有任何自动停止。
金融系统的断路器有效，是因为相关变量 —— 账户权益 —— 可以实时测量。技能系统中的类比变量 —— 从业者在技能运作的具体领域中的判断准确性 —— 不能实时测量，而这恰恰是从业者安装技能的原因。波兰尼在这里的观察非常精确：“我们知道的多于我们能够说出的。” 专家表现依赖默会知识 —— 模式识别、情境判断、知道什么时候不要套用规则 —— 这些都无法编码进一个结构化 Markdown 文件，无论文件写得多好。技能继承的是专家作者的显性流程。它没有继承作者的迟疑 —— 资深安全工程师看到某个认证模式并放慢速度去检查威胁模型的那一刻。它没有继承那种让优秀从业者在错误语境中暂停的领域特异性不确定感。技能是一个被截肢的专家：它有动作，但没有本能。
Anthropic 子组发现描述的正是这个机制。探询模式用户达到或超过手工控制组，是因为探询模式就是止损 —— 它是一种让从业者评估能力保持参与的摩擦，迫使执行之前先形成理解。委托模式用户得分 24-39%，是因为委托模式完全移除了这种摩擦。任务完成了。输出看起来正确。理解从未形成。
Shen 和 Tamkin 的探询模式，是技能生态中最接近追缴保证金机制的东西；而它恰恰是一个设计良好的技能文件被建造出来要移除的摩擦。
6. 单一化风险，以及诚实的连续性#
在这一节提出技能特有的主张之前，我想先承认那个显而易见的主张 —— 软件依赖中的单一生态风险，已经是一个解决到足以让多数人对 Dependabot PR 有意见的问题。
Log4j、left-pad、XZ。软件供应链攻击已有记录、已有研究，也已有部分缓解。这个领域有软件物料清单（美国行政令 14028）、SLSA 供应链证明、Snyk 漏洞数据库，以及签名提交。SBOM 要求之所以存在，是因为行业早就明白：一个广泛使用的组件，会携带广泛分布的风险。技能注册表是一种依赖生态。单一化风险不是类别上全新的东西。
有两个属性把技能依赖风险与现有依赖风险文献所处理的任何东西区分开来。
第一：当软件依赖包含 bug 时，bug 影响代码行为。当技能包含一个有缺陷的假设时，缺陷影响从业者的思考。输出不只是错误代码；而是一个被训练成 —— 跨会话、规模化地 —— 从有缺陷模型出发推理的从业者。一个把某类漏洞视为范围之外的安全审查技能，不只是会在一次审查中漏掉那类漏洞；它还会训练从业者停止寻找它。依赖污染制品；技能污染评估制品的认知过程。
第二：发现有 bug 的依赖，需要运行代码并观察失败。发现有缺陷的技能，需要在该技能运作的具体领域中拥有专业知识。有这种专业知识的人，大概并不需要这个技能。没有这种专业知识的人，面对自信但错误的输出，无法把它和正确输出区分开来。检测机制要求的，正是安装该技能的用户所没有的东西。
这两个属性合在一起意味着，技能供应链的失败模式不是普通依赖问题。它是一种失败发生在认知层面、会自我强化、并且对最可能受影响的人不可见的依赖问题。
测量已经存在，而且很具体。Snyk 安全研究团队审计了截至 2026 年 2 月来自 ClawHub 和 skills.sh 的 3,984 个技能，方法结合了八项专门安全政策与人工参与验证。 发现是：534 个技能 —— 占全部审计对象的 13.4% —— 至少包含一个严重级安全问题。91% 已确认的恶意技能同时采用提示注入技术和传统恶意软件代码。 三个主要攻击向量分别是：外部恶意软件载荷、通过混淆命令抓取 AWS 密钥等凭据的数据外泄，以及安全功能禁用。发布时，仍有 8 个恶意技能公开存在于 clawhub.ai。
一项针对面向 agentic coding assistants 的提示注入攻击的结构化综述，覆盖 Claude Code、GitHub Copilot、Cursor 以及相关平台，共 78 篇主要来源，发现 85% 或更多被识别攻击成功攻陷至少一个主要 agentic coding 平台，而自适应攻击绕过了 90% 以上已发表防御。 这些数字来自具名基准的综合，不是独立复现；85% 和 90%+ 这些数字来自包括 MCPSecBench 在内的底层研究。应把它们视作方向性而非决定性数字。
十年前，在一个类似生态中，Fraunhofer AISEC 的一个团队分析了 Google Play 上 130 万个 Android 应用，发现其中 15.4% 包含可追溯到 Stack Overflow 答案的安全相关代码片段。 在这些应用中，97.9% 至少包含一个不安全片段。 机制相同：开发者从一个足够可信的来源复制看起来能工作的代码，恰好是在自己缺乏专业能力来评估它的领域。Fischer 等人在 2017 年把这作为警告发表。警告没有阻止模式继续；最终跟上的，是工具链。
刚安装的技能有八分之一概率包含严重缺陷，这不是理论风险；这是 Snyk 对本季度注册表中可用制品的计数。
7. 这篇文章写给谁：相邻领域用户#
第 2 节的三项发现合在一起，预测了最高风险用户的一种具体形状。文献中还没有任何研究直接检验这个预测。
它不是纯新手。纯新手怀疑自己的输出。他们问更多问题。他们更可能处在探询模式，因为他们明确知道自己不知道。
被预测为高风险的形状，是相邻领域专家：一个在主领域确有资深能力、AI 素养高、正在其核心专长相邻的领域安装并使用技能的从业者 —— 安全、法律、监管、医疗。推论链来自已经测得的发现：更高的 AI 素养与更低的元认知准确性相关（Fernandes，r=0.21）。 经验丰富的从业者会高估自己的 AI 辅助表现（METR）。 委托模式会以高任务完成速度产出自信但错误的输出（Anthropic）。 符合这一画像的从业者 —— 主领域资深、AI 素养高、在相邻领域使用技能 —— 按照这三项发现描述的模式，会有足够信心安装并采纳一个安全审计技能，有足够技术熟练度把输出解释为合理，又存在足以让他们抓不住安全专家能抓住的错误的领域缺口。这个组合就是被预测的失败模式 —— 不是已测量，而是从趋同模式中预测出来的。
GPS 能力萎缩发现，是文献中最接近直接测量工具替代性萎缩的东西 —— 在一个相近领域，不是同一领域。Dahmani 和 Bohbot（2020）发现，终身 GPS 使用越多，空间记忆表现越差；在约 3.2 年的纵向追踪中，更多 GPS 使用与依赖海马体的空间记忆更陡峭下降相关，其中最强纵向相关达到 r=-0.68。 准确说明这项证据是什么、不是什么：它测量的是 GPS 替代之下海马体空间记忆的萎缩；它没有测量 AI 技能使用之下的领域专业能力。我们是在跨领域外推，不是在同一现象内外推。这项研究是观察性的，纵向样本很小（N=13），因果方向是被论证的，不是最终确立的。但机制很具体：替代某项能力的工具，可能让它所替代的能力萎缩；时间尺度就是常规替代持续发生的时间尺度。 把安全审查委托给技能的从业者，不只是今天拿到一个错误答案；他们还可能在技能运行的每一次会话中，收窄“我知道什么”和“我以为我知道什么”之间的差距。
如果这个素描像某个熟悉形象，诚实的问题是：那个形象有多少时候就是我们。
8. 把追缴保证金机制重新设计进去#
我不会在结尾解决缺失的追缴保证金机制 —— 处方比诊断更难，任何声称并非如此的人都是在卖技能。
Anthropic 自己的安全文档已经命名了这个风险类别：“我们强烈建议只使用来自可信来源的 Skills：你自己创建的，或从 Anthropic 获得的。Skills 通过指令和代码为 Claude 提供新能力，这让它们强大，但也意味着恶意 Skill 可以指示 Claude 以不符合该 Skill 声称目的的方式调用工具或执行代码。” 平台所有者正在警告用户警惕平台自己的生态。设计议程由此展开：明确声明适用条件的技能；促使用户进入探询模式而不是静默执行的技能；显露不确定性，而不是把作者的信心投射进用户语境的技能。“行动前验证”不是技能设计 —— 它是责任转嫁。
这些是设计问题，不是答案。它们要求技能知道何时说：我不是这个语境中的正确工具；而这要求技能拥有一个它们目前并不具备的语境模型。金融追缴保证金机制之所以有效，是因为账户权益可以测量。技能中的类比物，需要测量某种难得多的东西。
在设计工作完成之前，真正有效的测试，是 Shen 和 Tamkin 意外发表出来的那个：我是在问这个技能为什么，还是在接受它给我的东西？每次运行它时，答案都摆在我们面前。
参考文献#
Skills Are Leverage, Not Substitute
Those of us who installed three skills this quarter and shipped work we are proud of: this essay is not going to tell us we were wrong to do it.
The Claude Code skill ecosystem matured rapidly between October 2025 and Q1 2026. Anthropic launched Agent Skills in October 2025, and by December an open standard — the SKILL.md format — had become the cross-platform convention across Claude Code, Cursor, and the Agent SDK. Third-party indexes now list upward of 85,000 skills. The Hacker News thread "Claude Skills are awesome, maybe a bigger deal than MCP" collected 738 points and 370 comments before it scrolled off the front page. Readers in that thread, nodding along and installing the skills that looked useful, were doing exactly what the ecosystem was designed for — and in most cases, it worked.
This essay is not making the argument that skills are bad. It is not making the argument that anyone should use AI less, or that downloading expertise is inherently fraudulent, or that the junior-developer deskilling panic — already in its fourth rehearsal — applies here. It does not. Most of those pieces are about learning, and most of the fear in them is about juniors who never developed their foundation before reaching for a tool. That is a real concern. It is not this one.
The scope condition comes first. Low-stakes, fast-feedback domains are where skills genuinely help, including users whose foundation is thin. If the skill produces bad output, the feedback loop catches it quickly. The failure is visible. This is how democratization of expertise actually works, and it works. The concern in this essay is the complement: high-stakes, slow-feedback domains where the failure is invisible until it is expensive, and where the user's evaluation capacity in the specific domain where the skill operates may be lower than their confidence suggests.
This essay is about the tail — the conditions under which skills stop amplifying our judgment and start amplifying something else — and about why a recent finding, read carefully, suggests the tail is wider than most of us think.
1. The Mainline Works, and Here Is the Data That Says So#
The positive case for AI assistance in software work is stronger than most skeptical essays concede, and it is stronger than I would like it to be for the argument I am about to make.
In an Anthropic-run RCT studying how developers learn a new library with AI assistance, one subgroup of AI-assisted participants — those who used the tool for conceptual inquiry — scored approximately 86% on a comprehension quiz. A second group using a hybrid code-explanation approach scored around 80%. Both results match or exceed the manual-coding control group's score of approximately 70.9%. This is not a hedge. This is the paper's own data, disaggregated by the behavior that generated it: when AI assistance operated as a thinking tool rather than a code generator, it produced better comprehension outcomes than working alone.
Outside the laboratory, Anthropic's own enterprise case studies report the kind of compression that justifies the bullish reading. Thomson Reuters compressed legal research from hours to minutes. eSentire compressed threat analysis from five hours to seven minutes, with output alignment to senior expert judgment initially measured at 95% and confirmed at above 90% in sustained production. These are not toy tasks. They are production workflows running against real stakes. The democratization narrative — that downloadable expertise can substitute for scarce senior judgment in domains where access to that judgment is constrained — has real evidential backing in these cases.
The scaffolding framing also holds up. When AI assistance is used with inquiry — when the user is asking follow-up questions, probing explanations, building a model rather than accepting a result — it functions as what learning researchers call a More Knowledgeable Other: a resource that extends the user's zone of proximal development rather than bypassing it. This is not how the risk shows up. The risk shows up in the other mode.
The same study that produced those 86-percent comprehension scores also produced the finding this essay turns on.
2. What Is Real, Documented, and Probably About Us#
Three findings published between July 2025 and February 2026 — separately unremarkable, together pointing the same direction — describe a gap between what AI-assisted developers produce and what they believe they produced.
The first is from Fernandes et al., published in February 2026 in Computers in Human Behavior. Across two studies — N=246 and N=452 — participants solved LSAT logical reasoning problems with ChatGPT-4o access. AI use improved task performance by approximately three points over baseline. It also improved self-estimated performance by four points. The net result: a systematic confidence surplus even after the real improvement is accounted for. Performance went up; the sense of performance went up faster. The gap was replicated across both studies.
That finding is consistent with, though not sufficient to prove, longer-term degradation. Fernandes is a single-session design — one problem set, immediate self-assessment, no follow-up. The authors themselves explicitly state in the limitations section that their design "cannot address whether overconfidence impairs learning over time." The LSAT context also limits the scope of the inference across domains. What the study establishes — and establishes with a replication — is that the one-shot metacognitive gap is real and measurable.
The second finding, also from Fernandes, is the more uncomfortable one for this specific audience: across both studies, participants' score on the SNAIL AI literacy scale correlated positively with metacognitive bias. The most technically knowledgeable AI users made the largest self-assessment errors. The correlation is modest — r = 0.21, which explains roughly 4% of variance — and statistically significant in both studies. This is a directionally counterintuitive signal, not a dramatic finding. Think of it as a probing question rather than an established law: what if the pattern we're seeing in one LSAT session is the tip of something that compounds?
The third data point is METR's July 2025 RCT. Sixteen experienced open-source developers completed 246 real tasks on their own codebases, under randomly assigned AI-allowed or AI-forbidden conditions. The developers using AI — Cursor Pro with Claude 3.5 and 3.7 Sonnet — took 19% longer to complete tasks. They believed they were working 20% faster. Before the study, they had expected a 24% speedup.
The caveats are mandatory because a sophisticated reader already knows them: N=16 is thin. The codebase setting — developers on their own 1M-line repositories — maximizes AI's contextual disadvantage. The tools were early-2025 vintage. METR's own February 2026 follow-up states explicitly that their new experiment design "gives us an unreliable signal of the current productivity effect of AI tools," adding that "our data is only very weak evidence for the size of this increase." METR has not retracted the July 2025 numbers, but they have walked back their confidence in what those numbers tell us about current AI tools. Use this data point as one signal in a convergent pattern, not as the decisive case.
The pattern converges anyway. Stack Overflow's 2025 Developer Survey — the largest annual developer survey, with 60,000+ respondents — found that 84% of developers use or plan to use AI tools. The top frustration, cited by 66% of developers, is "AI solutions that are almost right, but not quite." That is the metacognitive gap in self-report form: the tool produced something plausible enough that the gap between "almost right" and "right" is the daily friction, not hallucinations about nonexistent APIs.
What none of these three studies explains, on its own, is which users the gap is largest for — and the answer came from the one study everyone skimmed.
3. The Turn: What the Anthropic Paper Actually Said#
The Anthropic number most of us remember — 55.6 percent on the comprehension quiz for AI-assisted, 70.9 for manual, a 17-point gap — is true, and it is also the wrong unit of analysis.
Here is what that number hides. Shen and Tamkin analyzed screen recordings from 25 of the 26 AI-condition participants (one recording was unavailable) and identified six distinct usage patterns. The authors explicitly flag this as "exploratory data analysis (not pre-registered)" — these are post-hoc categories derived from qualitative analysis of screen recordings, with per-cell sample sizes between two and seven participants. Small cells, directional finding, not statistically validated across the subgroups. Keep those limits in view.
The six patterns split into two tiers with no overlap:
High-scoring patterns (quiz scores 65-86%, at or above the 70.9% manual control):
- Conceptual Inquiry — n=7, approximately 86%: participants "only asked conceptual questions and relied on their improved understanding to complete the task"
- Hybrid Code-Explanation — n=3, approximately 80%: requested explanations alongside generated code
- Generation-Then-Comprehension — n=2, approximately 75%: generated code, then asked follow-up questions
Low-scoring patterns (quiz scores 24-39%):
- AI Delegation — n=4, approximately 24%: "wholly relied on AI to write code and complete the task; this group completed the task the fastest and encountered few or no errors in the process"
- Progressive AI Reliance — n=4, approximately 28%: shifted progressively to complete delegation
- Iterative AI Debugging — n=4, approximately 39%: relied on AI to solve errors rather than clarify understanding
Figure: Comprehension quiz scores by AI usage pattern — Shen & Tamkin (Anthropic), arXiv:2601.20245. The horizontal line marks the manual-coding control group's score (70.9%).
The paper's own framing: "High-scoring patterns generally involve more cognitive effort and less AI reliance."
The 17-point group-level gap is entirely a property of the low-scoring tier. The high-scoring tier did not close 17 points below the control. It closed approximately 15 points above the low-scoring tier's ceiling, and at or above the control. Extract the delegation subgroup, and the group-level finding flips.
This is, again, exploratory and post-hoc. The per-cell n values — two to seven participants each — are small enough that the point estimates are imprecise. The pattern description is qualitative. No pre-registered analysis validates the category boundaries. A rigorous reader weights this accordingly.
And yet: the direction of the split is 30 to 45 percentage points wide, it is directionally consistent across every cell in each tier, and it aligns with what every working developer already knows intuitively about the two ways to use an AI assistant. We can ask it to explain a concept until we understand it. Or we can ask it to write the code and ship the output.
The AI-assisted group did not underperform the control. One half of the AI-assisted group underperformed the control by 30 to 45 points. The other half matched or beat it. The variable that separated them was not tooling. It was mode.
This next step is the essay's most inferential move, and I want to mark it plainly, not smuggle it past the reader.
The Anthropic study measured skill formation: participants who had never used the Trio library worked with it for 35 minutes, then took a comprehension quiz. The unmeasured case is what happens when a practitioner who already knows their domain installs a Claude Skill for production use. The paper's own authors note this is "an important question this study does not resolve." The formation-to-production transfer is not measured; the argument connecting Shen and Tamkin's classroom to the reader's working repository is mine, not theirs.
The argument runs on mechanism: a practitioner using a skill in a domain adjacent to their core expertise is in an ongoing formation phase relative to that domain, even if they are a senior expert elsewhere. The Anthropic study's formation-stage findings apply wherever the user's comprehension of the skill's domain is being built in real time — which is precisely the scenario the adjacent-domain-expert case describes.
The SKILL.md format is designed to give Claude capability it did not previously have, delivered as structured instructions the practitioner installs and runs. Anthropic's own documentation describes Skills as enabling Claude to "invoke tools or execute code" on the user's behalf. The canonical skill-install workflow — find skill, install skill, use skill — has no friction point that asks the user: am I in inquiry mode or delegation mode right now? There is nothing in the format that demands conceptual engagement before execution, and the ecosystem economics do not add one: skills are marketed and ranked on compression — "hours to minutes," "5 hours to 7 minutes" — not on comprehension-supporting affordances. The artifact is optimized for the delegation default because the reward function at install time is speed, not understanding. Whether it reaches the user in inquiry mode or delegation mode is entirely up to the user — and the user is not prompted to choose.
Figure: The two feedback loops. Inquiry mode (left) maintains comprehension friction — the user's evaluation capacity engages at each iteration. Delegation mode (right) removes that friction — the task completes, but comprehension is never tested. The Anthropic subgroup finding (Section 3) is the empirical record of which loop produces which outcome.
Yes, these are the same numbers I cited as evidence the mainline case works. That is the point: the ecosystem's most visible successes are the ecosystem's clearest incentives — and what gets rewarded is compression, not the comprehension scaffold that makes compression safe.
One qualifier. Anthropic's inquiry-mode subgroup reached inquiry without any skill-file friction at play — they were using a plain chat interface and chose to engage conceptually. The article's claim is therefore narrower than "skills cause delegation." It is that the default friction toward inquiry, already variably present in a free-form chat context, is lower still inside a skill-packaged one — and that the ecosystem's reward function does nothing to re-introduce it. The mechanism is incentive, not determinism.
That is the connection between Shen and Tamkin's classroom and the repository we opened this morning — and it is, to be clear, a connection I am arguing for, not one they measured.
5. The Missing Margin Call, Now Defined#
A margin call is not a warning and it is not discretionary.
Under FINRA Rule 4210 and SEC Regulation T, margin accounts are subject to a mandatory 25% maintenance requirement. When losses reduce account equity below that threshold, the broker is authorized to liquidate positions without advance notice to the account holder. The investor does not get to decide whether the stop-loss fires. The broker may execute the liquidation at their discretion, independent of the investor's view of the position. The stop-loss is structural, legally mandated, and cannot be overridden by the person holding the leveraged position.
Figure: Financial leverage triggers a mandatory stop-loss when account equity falls below a regulatory threshold (FINRA Rule 4210). Skill leverage has no equivalent mechanism. The practitioner who lacks evaluation capacity in the skill's domain cannot detect the error — the same condition that made the skill useful in the first place.
The margin call is not a metaphor for financial prudence. It is infrastructure. Financial leverage is dangerous, and the financial system's response was to build the danger into the architecture: a circuit breaker that activates when losses exceed a threshold the leveraged party agreed to in advance.
Skill leverage is dangerous for the analogous reason — it amplifies whatever position the practitioner is already holding, correct or incorrect — but there is no analogous infrastructure. No threshold fires. No external party monitors the equity of the practitioner's judgment and liquidates when it falls below the maintenance requirement. The practitioner can continue to hold a confidently-wrong position at tool velocity, producing plausible output at the speed the skill enables, with no automatic stop.
The financial system's circuit breaker works because the relevant variable — account equity — is measurable in real time. The skill system's analogous variable — accuracy of the practitioner's judgment in the specific domain the skill operates in — is not measurable in real time, which is precisely why the practitioner installed the skill. Polanyi's observation is precise here: "we can know more than we can tell." Expert performance depends on tacit knowledge — pattern recognition, contextual judgment, knowing when not to apply a rule — that cannot be encoded in a structured markdown file, however well-authored. The skill inherits the expert author's explicit procedures. It does not inherit the author's hesitation — the moment a senior security engineer sees an authentication pattern and slows down to check the threat model. It does not inherit the domain-specific uncertainty that makes a good practitioner pause in the wrong context. The skill is an amputated expert: it has the moves but not the instincts.
This is the mechanism the Anthropic subgroup finding describes. The inquiry-mode users scored at or above the manual control because inquiry mode is the stop-loss — it is the friction that keeps the practitioner's evaluation capacity engaged, forcing comprehension before execution. The delegation-mode users scored 24-39% because delegation mode removes that friction entirely. The task completes. The output looks correct. The comprehension never forms.
Shen and Tamkin's inquiry mode is the closest thing the skill ecosystem has to a margin call, and it is exactly the friction that a well-designed skill file is built to remove.
6. Monoculture, With Honest Continuity#
Before I make the skill-specific claim in this section, I want to concede the obvious one — monoculture risk in software dependencies is a solved-enough problem that most of us have opinions about Dependabot PRs.
Log4j, left-pad, XZ. Software supply-chain attacks are documented, studied, and partially mitigated. The field has Software Bills of Materials (US Executive Order 14028), SLSA supply-chain attestation, Snyk's vulnerability database, and signed commits. The SBOM mandate exists because the industry already understood that a widely-used component carries a widely-distributed risk. A skill registry is a dependency ecosystem. The monoculture risk is not categorically new.
Two properties distinguish skill dependency risk from anything the existing dependency-risk literature addresses.
First: when a software dependency contains a bug, the bug affects code behavior. When a skill contains a flawed assumption, the flaw affects the practitioner's thinking. The output is not just wrong code; it is a practitioner who has been trained — at scale, across sessions — to reason from a flawed model. A security-review skill that treats a certain vulnerability class as out-of-scope doesn't just miss that class in a single review; it trains the practitioner to stop looking for it. The dependency corrupts the artifact; the skill corrupts the cognitive process that evaluates the artifact.
Second: detecting a buggy dependency requires running the code and observing the failure. Detecting a flawed skill requires domain expertise in the specific domain the skill operates in. Those with that expertise probably don't need the skill. Those without it face confident-wrong output indistinguishable, to them, from correct output. The detection mechanism requires exactly what the user who installed the skill doesn't have.
These two properties together mean that the skill supply-chain failure mode is not a garden-variety dependency problem. It is a dependency problem where the failure is cognitive, self-reinforcing, and invisible to the person most likely to be affected.
The measurement exists and is specific. Snyk's security research team audited 3,984 skills from ClawHub and skills.sh as of February 2026, using a methodology combining eight specialized security policies with human-in-the-loop validation. The findings: 534 skills — 13.4% of all audited — contain at least one critical-level security issue. 91% of confirmed malicious skills simultaneously employ prompt injection techniques alongside traditional malware code. Three primary attack vectors: external malware payloads, data exfiltration via obfuscated commands capturing credentials like AWS keys, and security disablement. At publication, eight malicious skills remained publicly available on clawhub.ai.
A structured review of 78 primary sources on prompt injection attacks targeting agentic coding assistants — covering Claude Code, GitHub Copilot, Cursor, and related platforms — found that 85% or more of identified attacks successfully compromise at least one major agentic coding platform, and that adaptive attacks bypass more than 90% of published defenses. These figures are synthesized from named benchmarks, not independently replicated; the 85% and 90%+ numbers come from underlying studies including MCPSecBench. Weight them as directional rather than definitive.
A decade ago in the analogous ecosystem, a team at Fraunhofer AISEC analyzed 1.3 million Android applications on Google Play and found that 15.4% contained security-related code snippets traceable to Stack Overflow answers. Of those apps, 97.9% contained at least one insecure snippet. The mechanism was the same: developers copied functional-looking code from a trusted-enough source, in a domain where they lacked the expertise to evaluate it. Fischer et al. published this as a warning in 2017. The warning did not stop the pattern; the tooling eventually caught up.
A 1-in-8 chance that the skill just installed contains a critical flaw is not a theoretical risk; it is Snyk's count of the artifacts available on the registry this quarter.
7. Who This Is For: The Adjacent-Domain User#
Read together, the three findings in Section 2 predict a specific shape for the highest-risk user. No study in the literature has directly tested this prediction.
It is not the raw beginner. The raw beginner doubts their output. They ask more questions. They are more likely to be in inquiry mode because they know, consciously, that they do not know.
The predicted high-risk shape is the adjacent-domain expert: a practitioner with genuine seniority in their primary domain, with high AI literacy, installing and using a skill in a domain adjacent to their core expertise — security, legal, regulatory, medical. The inferential chain runs from the measured findings: higher AI literacy correlates with lower metacognitive accuracy (Fernandes, r=0.21). Experienced practitioners overestimate their AI-aided performance (METR). The delegation mode produces confident-wrong output with high task-completion speed (Anthropic). A practitioner who fits this profile — senior in a primary domain, AI-literate, using a skill in an adjacent one — would, on the pattern these three findings describe, have the confidence to install and act on a security audit skill, the technical sophistication to interpret the output as plausible, and the domain gap that makes them unable to catch the errors a security expert would catch. That combination is the predicted failure mode — not measured, predicted from the convergent pattern.
The GPS atrophy finding is the closest thing the literature offers to a direct measurement of tool-substitution atrophy — in a cognate domain, not the same one. Dahmani and Bohbot (2020) found that greater lifetime GPS use correlated with worse spatial memory performance, and that in a longitudinal follow-up over approximately 3.2 years, greater GPS use correlated with steeper decline in hippocampal-dependent spatial memory — the strongest longitudinal correlation reaching r=−0.68. To be precise about what this evidence is and isn't: it measures atrophy in hippocampal spatial memory under GPS substitution; it does not measure domain expertise under AI skill use. We are extrapolating across domains, not across phenomena. The study is observational, the longitudinal sample is small (N=13), and the direction of effect is argued, not definitively established. But the mechanism is concrete: the tool that substitutes for a capability can atrophy the capability it replaced, over the timescale that regular substitution runs. The practitioner who delegates security review to a skill is not just getting a wrong answer today; they are potentially narrowing the gap between "what I know" and "what I think I know" in that domain, over every session the skill runs.
If this sketch resembles a familiar figure, the honest question is how often that figure is us.
8. Designing the Margin Call Back In#
I will not close by solving the missing margin call — the prescription is a harder problem than the diagnosis, and anyone who claims otherwise is selling a skill.
Anthropic's own security documentation already names the risk category: "We strongly recommend using Skills only from trusted sources: those you created yourself or obtained from Anthropic. Skills provide Claude with new capabilities through instructions and code, and while this makes them powerful, it also means a malicious Skill can direct Claude to invoke tools or execute code in ways that don't match the Skill's stated purpose." The platform owner is warning its users about the platform's own ecosystem. The design agenda follows from there: skills that declare their applicability conditions explicitly; skills that prompt inquiry-mode engagement rather than silently executing; skills that surface uncertainty rather than projecting the author's confidence into the user's context. "Verify before acting" is not a skill design — it is a responsibility offload.
These are design questions, not answers. They require skills that know when to say I am not the right tool for this context, which requires skills to have a model of context they currently do not have. The financial margin call works because account equity is measurable. The skill analogue requires measuring something considerably harder.
The test that works, until the design work is done, is the one Shen and Tamkin published by accident: Am I asking this skill why, or am I accepting what it gave me? The answer is available to us every time we run it.
References#