LangChain shares a practical guide for evaluating 'skills' — dynamically loaded instructions that improve coding agent performance in specialized domains. The post covers a 4-step pipeline: setting up a clean Docker-based testing environment, defining constrained tasks with clear metrics, structuring skills using AGENTS.md/CLAUDE.md files and modular XML sections, and comparing performance with/without skills using LangSmith. Key findings include that Claude Code with skills completed tasks 82% of the time vs. 9% without, skill invocation reliability is a real challenge, and balancing skill granularity (too many similar skills causes wrong invocations) requires testing. LangSmith tracing was used to observe Claude Code's trajectory and iterate on skill content.
Table of contents
What are Skills?The Basic Evaluation PipelineStep 1: Set Up a Clean Testing EnvironmentStep 2: Define the TasksStep 3: Define the SkillsStep 4: Run and Compare PerformanceConclusionSort: