What’s one of the biggest challenges with using AI for QA? It’s the lack of programmability. We set out to find a solution by pairing Playwright MCP with several different LLM models. Find out where we landed.

Thoughtbot's platform is a central hub for software developers and product designers, offering insights into product development, design thinking, and agile methodologies. Through articles, case studies, and design sprints, Thoughtbot offers insights into user research, prototyping, and usability testing. Readers can learn about lean product development, iterative design processes, and cross-functional collaboration to create user-centric digital products and services.

thoughbot

Thoughtbot explored building programmable AI-powered QA automation by moving beyond ChatGPT's manual interface. They tested Browser-Use (slower, less accurate) before discovering Microsoft's Playwright MCP, which uses accessibility trees instead of screenshots for faster, more reliable testing. Different LLM models produced varying results: GPT-4 missed issues, GPT-5 was thorough but slow, and Claude struggled with consistency until they had it generate its own QA prompt. This meta-approach of letting the AI design its detection strategy proved most effective at identifying widget issues across client websites.

Building programmable QA with AI

If you enjoyed this post, you might also like: