Processing hundreds of thousands of items one-by-one with individual database queries is a common performance anti-pattern. A real-world batch job that took nearly 24 hours was reduced to ~15 minutes by bulk-loading data grouped by day instead of querying per item. The post generalizes this to other I/O-heavy scenarios (HTTP APIs, messaging, file I/O), warns against chatty I/O patterns like ORM lazy loading in web requests, and discusses why parallelization or moving logic into the database are less effective than batching. It also cautions against over-estimating scale and stresses understanding your tools, using Entity Framework Core's change tracker and lazy loading proxies as an example of hidden memory overhead.

7m read timeFrom blog.codingmilitia.com
Post cover image
Table of contents
Intro #Practical example #That’s an actual real world example #Generalizing #Some quick fire bits #Outro #
2 Comments

Sort: