When Two Databases Become One: How DuckDB Saved Our Trading Operations from Manual Reconciliation A story of how we transformed a painful manual process into an automated solution using DuckDB’s …

The Groww Engineering Blog offers insights, case studies, and technical articles on building scalable and reliable financial technology platforms. Covering topics such as backend architecture, microservices, and fintech innovations, the Groww Engineering Blog provides resources for software engineers and fintech professionals seeking to build robust and secure financial products and services.

Groww Engineering

A trading platform faced recurring position-order mismatches across two separate MySQL databases, requiring 20-30 minutes of manual reconciliation by two engineers. By leveraging DuckDB's MySQL scanner extension to perform cross-database joins, they automated the entire process into a 2-3 minute operation running every 15 minutes. The solution eliminated manual intervention, improved accuracy from 85% to 99.9%, and enabled proactive monitoring instead of reactive fixes during market hours.

When Two Databases Become One: How DuckDB Saved Our Trading Operations from Manual Reconciliation

“At our trading platform, we had a classic distributed systems problem: our order data lived in one MySQL database, and our position data lived in another”
You are fired… end of story

You were doing all these manual operations on prod? Isn’t it weird? You are handling people’s money.

Sounds to me this isnt the actual issue buddy…

This is a perfect example of the hidden cost of data integrity incidents. 20-30 minutes of manual reconciliation twice a day isn’t just “operational overhead” - it’s your team’s incident response being done by hand.
The real value here beyond DuckDB is thinking operationally: how many of your “manual processes” are actually incident response procedures that should be automated? Most trading/fintech orgs have dozens of these reconciliation jobs hiding in Airflow DAGs or cron jobs.
When you do automate them (like Groww did), the next question is: how do you know it’s working? Reconciliation jobs fail silently - the query runs but returns 0 mismatches (false negative). You need monitoring that detects when reconciliation patterns change, when mismatch counts spike unexpectedly, or when the job latency increases. That’s where incident intelligence becomes critical.