The post introduces two metrics, Characters per Opportunity (CPO) and Percentage Code Written (PCW), to measure the capabilities and value of AI code assistants. CPO is a benchmark metric that cannot be gamed and correlates with real tool value. PCW measures the value driven by a code assistant to the end developer. Existing
Table of contents
Why Do We Care About Autocomplete?Why Existing Benchmark Metrics Cannot be TrustedCPO: Characters per OpportunityWhy CPO Measures Real Value and Cannot be GamedCodeium’s Current CPOHow We Have Developed Against CPOReal vs Perceived ValuePercentage Code WrittenConclusionSort: