When a bug is fixed, how can we make sure it will not be re-introduced in the future?

Of course, through a proper regression test, i.e., a test that fails in the presence of the bug and passes in its absence.

However, regression tests are frequently ommited by the developers, who tend to submit the patch alone.

How can AI automation help developers it this scenario?

We explore this question in depth by introducing our open-source GitHub bot that is triggered when a pull request is opened, and proposes a test that is guaranteed to fail in the presence of the bug and pass in its absence.

We use a combination of LLMs and traditional, search-based testing techniques to achieve this goal.

The bot is currently deployed in three Python repositories (Javascript and Rust will be soon also supported) of the Mozilla corporation, where it has already helped developers with their regression tests. We share interesting early insights and feedback from this deployment and invite more open-source developers to adopt the bot and provide constructive feedback!

Devoxx

A PhD researcher presents an open-source tool that automatically generates regression tests when a bug fix is submitted as a pull request. The tool uses a hybrid LLM and search-based software testing approach (genetic algorithms) to produce 'fail-to-pass' tests — tests that fail before the patch and pass after. It only posts a suggested test as a PR comment when it has verified the test actually exhibits this behavior, reducing false positives. Evaluated on 426 Python PRs across 12 repositories, it succeeded ~30% of the time. It was also deployed at Mozilla, yielding 8 confirmed useful tests. The tool supports Python primarily, with beta support for JavaScript and Rust, and requires a Docker setup, a lightweight server, and an LLM API key.

Bug fixed, now what? by Konstantinos Kitsios