<a href="https://jojoldu.tistory.com/872" target="_blank" rel="noopener nofollow">👉🏻original post</a>
<h2>“Let’s add recommendations to our service. Can’t we just hand that to an LLM?”</h2>
In meetings with CTOs—or at meetups—I hear this constantly. CEOs, PMs, POs, business leads: everyone talks as if an LLM already “knows” all of our service data and can act as a universal recommendation engine.
Under that expectation sits a single misunderstanding: the idea that an LLM stores data like a database and retrieves it when asked.
That misunderstanding is simple to explain: from the outside, the two look the same.
You query a database with SQL and get an answer.
You prompt an LLM and get an answer.
The experience—“ask a question, get an answer”—feels identical, so people stop caring what happens under the hood.
But under the hood they are fundamentally different.
A database answers by looking up what is already stored.
An LLM answers by reasoning from (1) general knowledge learned during training and (2) whatever information we put in the prompt.
Your live service data is not sitting inside the model unless you explicitly provide it.
Retrieving from storage vs. inferring from handed context are not the same job.
<h2>Bottom line: an LLM is not a database. It’s a giant function.</h2>
Databases store data.
They hold user profiles, purchase history, product catalogs, click logs, and return exact results for queries.
Ask for “products user A bought in the last 30 days,” and you get that list, bounded by what was stored and the query conditions.
An LLM does not hold your service data in a reliably queryable form like a DB.
Of course the model isn’t “empty”: it learned patterns from public text on the internet, so it can answer general-knowledge questions (“What is Baemin?”). That’s closer to having read an encyclopedia than to having your business facts.
A large set of parameters implements a huge mathematical function: input (prompt) → most plausible output (response).
So if you ask, “Recommend products for user A,” the model effectively responds with:
“Who is user A? What products exist? What did they buy?”
The LLM doesn’t know your service data because it isn’t connected to your database by default.
<h2>Stateless vs stateful</h2>
Default LLM behavior is stateless.
A function takes input and returns output: f(x)=y_f_(x)=y, where x_x_ is the prompt and y_y_ is the response.
“Stateless” means it doesn’t remember what it processed last time. Each call is independent. Even if you asked about user A yesterday, it won’t “remember” yesterday’s chat unless you bring that context back in.
Chat feels continuous because we keep re-injecting prior turns into the input (or attach external memory). The base model still isn’t “remembering” on its own.
Databases are stateful storage.
Once written, data persists until explicitly deleted. Yesterday’s purchase row for user A is still there tomorrow—and next year. Query it, and you get that exact row.
<h2>“Then we’ll just train it on our data.”</h2>
Training is not the same as remembering.
Even with fine-tuning to absorb some domain patterns, using a model as a durable, query-accurate store of fresh internal data is the wrong mental model. You don’t get a model that reliably recalls “the exact list of items user A bought on March 15” like a DB would. It may internalize patterns, not records. Memorization-like effects exist, but they’re not queryable storage of business truth.
Your data changes constantly.
Items go out of stock and come back; clickstreams grow every hour. What you fine-tuned a week ago is already stale. Fine-tuning costs time and money, and cadence varies by model size and approach—you can’t realistically re-train on every inventory swing and every hour of logs.
Fine-tuning is also often tied to a base model; upgrades can force re-training or re-evaluation.
So “make the LLM memorize our data” doesn’t scale. Fetch data when needed and pass it in. That fetch path is what databases are for.
<blockquote>
“So LLM recommendations are impossible?” / “Doesn’t AI pull DB data automatically now?”
</blockquote>
LLM-assisted recommendation is absolutely possible—but the architecture is not “just ask the LLM to recommend.”
Modern stacks use tool/function calling and integration patterns so the system behaves like it queries a database. Peel it back and it’s still:
LLM decides to call a tool → the tool queries the DB → results are returned to the LLM (typically as context in the prompt).
The LLM isn’t magically querying Postgres by itself; middleware fetches rows and feeds them in.
For that to work, you still need a database and clean data. Data infrastructure comes first—same conclusion as before.
<h2>Two sane patterns</h2>
1) Feed the LLM outputs from a traditional recommender
Use a managed reco product (e.g., cloud recommender APIs) or collaborative filtering to generate a broad candidate set, then let the LLM re-rank for context or explain recommendations in natural language.
2) Use a vector database as a retrieval stage
Embed your catalog/content as vectors, retrieve top‑K semantically relevant items for the user’s question or interests, then pass that small candidate set to the LLM for final packaging.
Either way, the core idea is identical:
Don’t dump the whole dataset into the prompt. Shrink it to a tractable bundle the model can reason over.
Too much context often hurts accuracy.
Quality of LLM-assisted recommendation is mostly bounded by the quality of what you pass in, not by “how smart the model feels” in isolation.
If a classical recommender already works well, there’s no reason to replace it with an LLM.
<h2>Why classical recommenders still matter</h2>
They are built to ingest behavioral signals at scale: dwell time, repeated product views, searches, cart adds/removes, partial video watches, etc.—often millions of events, encoded into models.
To replicate that with an LLM-only approach, you’d conceptually need to stuff all of a user’s events, all users’ purchases, the full catalog, reviews, orders—into the prompt. And you’d need to do it again on every request, because the model doesn’t persist state.
That’s practically impossible: token limits, cost, and the sheer absurdity of serializing tens of millions of log lines into natural language every time. LLMs are strong at language/context reasoning, not at being a drop-in replacement for massive numeric behavioral modeling in real time.
Years of investment in behavioral modeling aren’t “replaced” by an LLM. The right move is: summarize/retrieve structured signals, then ask the LLM to compose or explain on top.
<h2>Latency and cost</h2>
Classical recommenders often respond in milliseconds at near-zero marginal cost. LLM calls can be orders of magnitude slower and more expensive depending on model and prompt size.
At millions of QPS, “LLM on every recommendation” is usually the wrong default. You don’t need an LLM in the hot path for every impression.
<h2>Closing mental model</h2>
An LLM is a giant function, not a giant datastore.
Separate inference from storage. The LLM reasons; the database holds truth. You layer the LLM on top—you don’t pretend it replaces your data plane.
Wanting to adopt AI is a good direction—but ask first:
<ul>
<li>Are we actually logging the behaviors we need?</li>
<li>Have we defined what data matters for recommendations?</li>
<li>Can we retrieve, filter, and shape that data into something an LLM can use reliably?</li>
</ul>
If you can’t answer those, the work before the LLM is still the work: build the pipeline that delivers right-sized, trustworthy context—then let the model reason on top of it.
Based on “[LLM은 데이터베이스가 아니다]” by [<a href="https://github.com/jojoldu" target="_blank" rel="noopener nofollow">Dong Uk Lee (aka 향로, jojoldu)</a>] · [<a href="https://www.tistory.com/" target="_blank" rel="noopener nofollow">tistory</a>] · [2026.03.21] · Translated by [SeyoungCho].

SeyoungCho

LLMs are stateless mathematical functions, not databases — they don't store or retrieve your service data by default. The post debunks the common misconception that you can simply 'ask an LLM' to power recommendations without any data infrastructure. It explains the fundamental difference between database storage (stateful, exact retrieval) and LLM inference (stateless, context-dependent reasoning), why fine-tuning doesn't solve the problem, and presents two practical architectures: using a traditional recommender to generate candidates that the LLM re-ranks, or using a vector database for semantic retrieval before passing a small candidate set to the LLM. Classical recommenders still matter for behavioral signal modeling at scale, and LLM calls are orders of magnitude slower and costlier than traditional systems at high QPS.

[Translated] LLM is not a database