The post introduces two metrics, Characters per Opportunity (CPO) and Percentage Code Written (PCW), to measure the capabilities and value of AI code assistants. CPO is a benchmark metric that cannot be gamed and correlates with real tool value. PCW measures the value driven by a code assistant to the end developer. Existing benchmark metrics cannot be trusted due to gaming and incentivizing the wrong product decisions.
Table of contents
Why Do We Care About Autocomplete?Why Existing Benchmark Metrics Cannot be TrustedCPO: Characters per OpportunityWhy CPO Measures Real Value and Cannot be GamedCodeium’s Current CPOHow We Have Developed Against CPOReal vs Perceived ValuePercentage Code WrittenConclusionSort: