It’s interesting how we approach problems at different stages of the software development life cycle.
A few months back at Medoc (where I am currently working as a backend engineer, and since it’s a growing startup, a lot of things are being handled by me), I asked my teammate (a fresher) to write an endpoint for fetching today’s appointments and those that are not completed for a doctor.
It was quite simple because we have a separate collection for appointments with an eventDate field, so he just needed to query it based on the current time. But here was the catch: the frontend also required patient details along with the appointments. The patient data is stored in a separate database.
So it became a case of dual database calls, where the first call fetches the appointments and the second call fetches the associated patient details, which is a typical N+1 query problem.
What my teammate did, like most people would, was:
<ul>
<li>fetch appointments</li>
<li>loop over each appointment</li>
<li>fetch patient details one by one</li>
</ul>
And it worked.
Manual testing passed, it was deployed to dev for QA testing, and that passed too.
Soon it was deployed to production and worked fine for over two months. Everything seemed sorted.
Then a few days back, I got a complaint from the frontend team that their home screen was getting stuck, and that too for only one doctor, which was strange. A frontend friend and I started debugging it, but here we made a mistake. We were testing in the dev environment and were not able to reproduce the issue.
We spent almost an hour trying to fix and optimize the UI. Our initial thought was that since it was a home screen, it was hitting too many endpoints while loading the initial data. But we couldn’t find the real culprit.
Then we checked the production data for that doctor. Once we accessed the user account and opened the appointments view, we found the issue.
This particular doctor had 6,000+ appointments, which is a lot for a startup, and he had marked only a very small number as completed, so the incoming data was huge.
I immediately started pinpointing the API responsible for fetching these appointments. As soon as I looked at its implementation, the problem became clear.
It was making 6,000+ database calls to retrieve patient data. That’s why the response time was around 4 seconds, which was never caught during initial testing because we never tested with this much data.
I immediately refactored the code using in-memory maps (feel free to comment if you have a better approach).
The implementation was simple:
<ul>
<li>first, fetch all required appointments</li>
<li>then create an array of patient IDs</li>
<li>fetch all patient details in a single database call</li>
<li>create a map with patient ID as the key and patient details as the value</li>
<li>loop over the appointments again to construct the DTO</li>
</ul>
I also optimized a few things. This route was over-fetching data, so I added proper projections and fetched only the required fields. This reduced the payload size as well.
After this, the latency came down to 500–600 ms, which was a huge difference.
Now comes the real lesson.
The fault was not just the developer’s. It was also:
<ul>
<li>me, for merging the PR without a deep review and not clarifying constraints</li>
<li>the tester, who tested only in a controlled environment</li>
<li>the developer, who didn’t think about edge cases or ask for clarity</li>
</ul>
But I believe these kinds of unexpected production issues are what make developers grow and face the real horror of production.
While writing code, always expect the worst and hope for the best.
— Nikhil

Developer's Journey is a place where we share our stories, career growth, technical adventures, and personal milestones, no matter where we are in our journey.

Developer's Journey

A backend engineer shares a production incident where an appointment-fetching endpoint worked fine in dev and QA but caused 4-second response times in production. The issue was an N+1 query problem: the code made 6,000+ individual database calls to fetch patient details. The solution involved batching patient data retrieval into a single query using in-memory maps and adding proper projections, reducing latency to 500-600ms. The incident highlights the importance of testing with realistic data volumes, thorough code reviews, and anticipating edge cases during development.

It Worked in Dev. It Worked in QA. Then Production Happened.

I think Pagination and Indexing in DB can do this even better

These are the experiences that truly shape developers. Great post 👏

Let’s apply this situation to a 5-second window.
In your latest solution, imagine you have 50k patients and 1k doctors. At a given time, all 1k doctors call the API to retrieve incomplete appointments. Assume that 1 KB of memory is required to load one patient’s details.
In parallel, for each doctor, the API fetches and holds (50k × 1 KB) = 50 MB of data for further processing. For 1k doctors, this results in approximately 50 GB of memory usage.
In reality, the memory usage would likely be even higher due to overhead from data structures, object metadata, and buffering.

Great post. This is real life for developers. This experience is common for every dev. Unfortunately, users always surprise us by using the system in unexpected ways. I’ve been through this many times, but the good thing is that we always come out better devs.