#engineering

Beyond the chatbot: a multi-tenant AI platform that scales, costs what it should and measures its own answers

For a technology services company we built a platform of AI assistants offered to many customers: capacity follows the load, no release interrupts a job in progress, expensive model work is paid for once, and every retrieval technique, up to the knowledge graph, is switched on per customer and measured with evaluations.

Artificial intelligence · RAG · Knowledge graph · AWS · PostgreSQL

TL;DR: we built a platform on which many organizations query their own documents through AI assistants, each one isolated from the others. Scalability, cost and reliability live in the infrastructure: capacity grows with the job queue, no release interrupts a job in progress, and expensive model work is paid for only once. Every AI choice, from the model to the retrieval technique, is instead a configuration switched on per customer: several techniques coexist, from hybrid search to the knowledge graph, and which one works best is decided by evaluations, not impressions.

Building a chatbot that answers from a few PDFs takes an afternoon. Building a platform on which many organizations upload large amounts of documents and query them from the web app, from a widget on their own site, from a messaging app and through an API is a different job. The hard questions change: what happens when many customers upload documents at the same time, how much each answer costs as usage grows, what happens to a long indexing job when a new version is released. And above all, a question traditional systems do not have: how do you build something stable on a technology that changes every month, where today’s best model is overtaken in six months and the most promising retrieval technique has to be proven, not assumed.

For a technology services company we designed and built a platform of this kind, which the client offers to its own customers as a service. The answer we gave to all these questions is a single one: scalability, cost and reliability must be properties of the infrastructure, while every AI choice must be a configuration that is switched on per customer and measured.

A new customer is a configuration, not a new deployment

The first choice that decides whether a platform scales is the multi-tenant model. The fastest way to serve the first customer is a dedicated installation, but every additional customer becomes another environment to maintain, update and pay for. Instead we built a single platform in which every organization works in one or more knowledge spaces, each with its own documents, permissions and configuration: which models it uses, with which prompts, how documents are split into chunks and enriched, which retrieval techniques are active.

Platform management and the AI side are separate services, because they run at opposite paces: short, transactional requests on one side; connections held open for the whole stream and slow calls to model providers on the other. That way each one scales on its own, but the source of truth for identity and permissions stays single, whatever channel the user comes from.

Isolation between customers is a property of the infrastructure too. The document records, the chunks and the vectors live in the same PostgreSQL database as the application data, and the boundary between customers is a constraint of every query, in the same transaction and with the same guarantees as the rest of the data, instead of a filter to remember in a second system. A new customer is an organization and a few spaces, not a project.

Capacity follows the queue, and switches off when it is not needed

The jobs of an AI platform are not all alike. A message from a conversational channel must be answered within seconds; indexing a long document or extracting a knowledge graph can take many minutes. With a single queue, one customer’s bulk upload would delay the answer to another customer’s user. That is why background work is split into lanes, each with its own queue and its own capacity: an interactive one, a processing one and one for the longest jobs.

In the ordinary lanes every worker takes one job at a time. We give up some density in exchange for predictable behaviour, and that choice makes the next one possible: scaling on the right metric. The CPU of a worker waiting for a model’s answer says nothing about the load; the number of queued jobs per active worker says exactly how much capacity is missing. A burst of uploads becomes more workers within a few minutes, and in quiet periods the processing lane scales down to zero. There is a detail that is easy to overlook: when a service is at zero, the active-worker metric has no data and target tracking alone often cannot wake it up. We added a dedicated alarm that forces a restart when messages are waiting and no worker is running.

The longest jobs, which can last hours, do not go through the ordinary lanes: each one starts a dedicated task that lives exactly as long as the job and then disappears. For the ordinary lanes the project has a rule: a job that keeps a worker busy for too long is a design defect, not a configuration problem. Multi-step processing is split into stages that save their results to the database: a new attempt reuses the work already completed instead of starting over.

No job interrupted, not even during a release

Autoscaling and releases share a blind spot: to reduce capacity or replace a version, the container platform stops processes without knowing whether a worker is halfway through an indexing job. The most common fix is to raise the stop timeout until it covers the longest job, but on AWS Fargate the limit is two minutes, and a shutdown time sized for the worst case would slow down every release anyway. Accepting the interruption and requeuing instead means redoing work that includes model calls already paid for.

We reversed the responsibility: the worker tells the infrastructure when it cannot be stopped. A supervisor next to each worker knows the jobs in progress and their maximum deadline, and keeps scale-in protection active as long as there is work to finish. When it notices that a new version has been released, it asks the worker to finish the current job, take no more and shut down. A release or a capacity reduction waits for the indexing in progress to finish, instead of interrupting it.

The same principle applies to the user. In the web app every conversation turn is saved before the model starts answering, and the answer is completed and saved even if the connection drops. If the user stops it or a provider fails, the turn stays recorded as interrupted and can be regenerated in place, without losing the thread of the conversation.

Cost is governed at every step, not at the end of the month

In an AI product every operation has a variable cost, and the surest way to lose control of it is to find out from the invoice. We treated cost as a design requirement, step by step.

When documents come in, the vision model is paid for only where it is needed: only PDFs that contain images, such as diagrams or scans, go through a model able to read them, while all the others are extracted as text. The most expensive work, such as knowledge graph extraction, is saved together with its provenance (content, prompt, model, software version): work already paid for is not paid for again, neither on a retry nor when the graph has to be rebuilt, which happens from the saved data without calling any model.

The platform works with models from several providers, through Azure OpenAI and Amazon Bedrock, and every step of the pipeline uses the model suited to its task, not the most powerful one available: rewriting a question or extracting metadata does not need the model that writes the final answer. Adding a model means registering it with its capabilities, without changing the pipeline. Conversation memory has a ceiling too: beyond a share of the context window, the oldest turns are summarized by a dedicated model, and a request that would not fit anyway is compressed or stopped before it is sent. Every model call is recorded with step, model, customer and space: what a customer costs is read from the data, not estimated.

Several techniques for the same problem, switched on space by space

In RAG there is no absolute right technique. An archive of contracts, a collection of product sheets and a set of internal procedures raise different questions, and a technique that excels on one can be superfluous or expensive on another. That is why we did not pick a technique: we built a platform in which several techniques coexist and are switched on per space.

Problem Technique
A chunk extracted from a long document loses its context contextual retrieval: every chunk is enriched with the document’s title and summary
Semantic search dilutes codes and identifiers hybrid search, semantic and keyword-based, with a single final reranking
The answer depends on attributes of the document automatically extracted metadata, used as a hard filter or as a preference in the ranking
The answer requires connecting scattered entities knowledge graph, with every piece of evidence re-authorized before it reaches the model

The knowledge graph deserves a story of its own, because it tackles the best-known limit of classic RAG. Similarity search is good at finding the passages that answer a local question, but it struggles when the answer requires connecting entities scattered across different documents: which components recur across several projects, which procedures depend on one another. In the spaces where it is active, a language model analyses every chunk and extracts entities, facts and relations from it, guided by entity categories configured for the space. At query time, semantic search finds the starting points and the graph connects them across documents. Two choices make it fit for a multi-tenant platform: on the graph the boundary is the space itself, always derived from the one already authorized and never from the request, and every piece of evidence gathered in the graph is re-authorized against the relational database, with the same rules as classic search, before it reaches the model. And the expensive part, the extraction, is paid for only once, as we have seen.

None of these techniques required touching the others. Vectors from different embedding models coexist in the same table, and a space can move to a better model without migrating the others; a new technique inherits authentication, isolation, tracing and cost accounting, and is tried on one space before it becomes the default choice.

Evaluations decide, not impressions

Having many techniques is an advantage only if you know which one works best, and in an AI system “it seems to answer well” is not a measurement. The quality of an answer depends on documents, model, prompt and retrieval technique together, and changing one of these elements can improve ten answers and worsen ten others without anyone noticing. That is why every change can be checked with repeatable evaluations.

At the core is an LLM-as-judge system that scores every answer on three independent axes: whether the retrieved documents are relevant to the question, whether the answer is correct against a reference answer, and whether it is grounded in what was retrieved rather than in the model’s own knowledge. Evaluation cases belong to each space and run on its real document base, with its models and its prompts. Every result keeps a full snapshot of the configuration at that moment, together with the software version: comparing two models, two prompts or two retrieval configurations on the same cases, or checking that a release has not made anything worse, is a query, not a reconstruction.

The same rigour applies to the most delicate components. The knowledge graph has automated acceptance tests that check the properties that matter: a single embedding computation and a single model call for the answer per question, isolation between spaces even with colliding identifiers, idempotent rebuilding of the graph. Prompts follow the same path: they are tried in one space, validated, and only then become the default. Underneath it all, a test suite of more than five thousand tests makes it safe to change the code continuously.

What the client gets

The client offers the platform as a service, and growing it does not require rethinking it. A new customer is an organization and a few spaces; a burst of uploads is a few more workers for a few minutes; a new model is registered with its capabilities; a new retrieval technique is switched on in one space, measured and then extended. And the cost of every customer can be read step by step.

This is how we approach AI products. Models and techniques are the part that changes fastest, and precisely for that reason they cannot be where the guarantees live: scalability, cost and reliability live in the infrastructure, while every AI choice becomes a configuration that is switched on, compared and measured. So when the next promising technique arrives, the question is not whether it can be adopted, but what the numbers say.