Processing hundreds of thousands of items one-by-one with individual database queries is a common performance anti-pattern. A real-world batch job that took nearly 24 hours was reduced to ~15 minutes by bulk-loading data grouped by day instead of querying per item. The post generalizes this to other I/O-heavy scenarios (HTTP APIs, messaging, file I/O), warns against chatty I/O patterns like ORM lazy loading in web requests, and discusses why parallelization or moving logic into the database are less effective than batching. It also cautions against over-estimating scale and stresses understanding your tools, using Entity Framework Core's change tracker and lazy loading proxies as an example of hidden memory overhead.
Table of contents
Intro #Practical example #That’s an actual real world example #Generalizing #Some quick fire bits #Outro #2 Comments
Sort: