The bug report you never want to read
Every multi-tenant security incident you never want to read about starts with the same sentence: *"Customer A can see Customer B's data."* And almost every one traces back to a single line of code that *should* have scoped a query to the right customer — and did not. A new database query someone added in a hurry. A reporting feature bolted on late. A shortcut that looked harmless at the time. The system was one forgotten filter away from a breach the entire time, and nobody knew until it leaked.
If your business runs on a shared platform — a SaaS product, an internal system serving multiple departments, or, increasingly, an AI platform serving multiple customers — this is the risk that should keep you up at night. Not a sophisticated attacker. A teammate, on a normal Tuesday, who forgot one line.
We build and run a multi-tenant platform for AI agents, and we lived with exactly this exposure. Isolation between customers rested on the application *remembering* to scope every single query correctly. That is a fragile foundation, because "remember every time, forever, across every developer who ever touches the code" is not a security control. It is a hope. Here is how we replaced the hope with something a future developer cannot accidentally switch off.
Why one line is too thin a defence
The conventional way to keep tenants apart is to filter every query by the customer it belongs to — "only return rows where this data belongs to *this* customer." It works, right up until someone writes a query that forgets to.
The problem is structural, not a matter of discipline. The filter lives in application code, which means it lives in exactly the place a busy developer can omit without anything visibly breaking. The query runs. It returns data. The tests pass. Everything looks fine — until the day the missing filter happens to sit on an endpoint that returns *everyone's* data. There is no error to catch, because nothing is technically broken. The system is simply doing precisely what the forgotten line failed to stop it from doing.
You cannot solve a "someone might forget" problem by asking people to remember harder. You have to move the boundary somewhere it *cannot* be forgotten.
Move the boundary where it cannot be skipped
We rebuilt isolation as three independent layers, so that a miss at one is caught by the next:
1. The front door identifies which customer a request belongs to, the moment it arrives.
2. The application layer filters queries by that customer — the conventional defence, still useful.
3. The database itself refuses to return rows that do not belong to the active customer — even to a query that never heard of the rules.
The first two layers are application code, which means they are the layers a developer can forget. The third cannot be forgotten, because the enforcement no longer depends on the code asking nicely. The database itself says no. It will withhold another customer's rows from a hand-written query, from a reporting tool, from a piece of code written years later by someone who never knew the original rules existed.
This is the principle worth taking away even if you never touch a database: the layer that enforces a critical security boundary should be the one a future developer cannot accidentally skip. Application filters are necessary, but they can never be sufficient on their own, because they rely on perfect memory across everyone who will ever touch the system.
Fail closed: a bug that returns nothing, loudly
The single most valuable design choice was deciding what happens when something goes wrong. We made the system fail closed.
If the platform cannot confidently determine which customer a request belongs to, it does not guess, and it does not fall back to "show everything." It returns *nothing*. An unresolved request gets zero rows — a result that is loud, obvious, and safe, rather than quiet and catastrophic.
This inverts the entire risk profile. Under the old model, a bug leaked data silently — the worst possible failure, because no alarm sounds when a system hands out more than it should. Under fail-closed, the same class of bug produces an empty screen that someone notices and reports in minutes. We turned a category of silent data leaks into a category of obvious, harmless malfunctions. A safe failure that is visible beats a dangerous one that is silent, every time.
The subtler cousin: when the guard rejects the right people
There is a second, sneakier failure mode in multi-tenant systems, and it is the one most likely to bite a business slowly rather than all at once.
We had a security check that compared two identifiers to confirm a request belonged to the customer it claimed. The check was a literal, character-by-character comparison — and it started rejecting *legitimate* requests. Same customer, same user, same everything: refused.
The cause was that our identity system represented the same customer in two different formats depending on the path the request travelled — two spellings of one identity. The security guard was comparing the two spellings, seeing different characters, and correctly enforcing its rule against an identity it had simply failed to *recognise as itself*.
This is not a hole — nothing unauthorised gets through. But it is the equally dangerous inverse: a security control that fires on false alarms and blocks the very people it is supposed to admit. And here is why it is genuinely dangerous rather than merely annoying: the pressure to "just make it work" pushes someone toward *loosening* the check. That is exactly how a real hole gets introduced later — a frustrated fix to a false alarm becomes the gap an attacker walks through.
The correct fix is a discipline, not a patch: convert both sides to one canonical form before you compare them — and do it the same way in every place that makes the comparison. Identity is not the string you happened to receive; it is the thing that string refers to. If two parts of a system normalise identity differently, you have built a new vulnerability in the seam between them.
Establish identity once, at the edge
The deeper lesson behind both stories is about *where* trust is established. The answer that held up for us: establish who the caller is exactly once, at the outermost edge of the system, and pass that verified identity inward. Internal components trust the verified identity rather than each re-deriving it.
Re-checking identity at every internal hop sounds safer but is the opposite — every place that re-derives identity is another place that can do it slightly differently, which is precisely the bug above, multiplied. Decide once, at the boundary, and let the canonical answer flow inward.
Maturity is documenting what you left out
One last decision separates a mature security posture from a box-ticking one: we wrote down, explicitly, what we *deliberately did not* protect this way, and why.
Not every piece of data needs the same boundary, and forcing it where it does not belong creates breakage that masquerades as security. Some internal records are accessed by unguessable identifiers in contexts where the customer scope genuinely does not apply; locking them down the same way would break legitimate functionality without closing any real hole. We documented that reasoning rather than leaving the next engineer to find the gap, assume it was a mistake, and either "fix" it into a malfunction or quietly distrust the whole design.
We were equally explicit about the limit that *remains*. These controls shrink the blast radius from "any forgotten filter leaks everything" to a far narrower, well-understood case. That is a real improvement, not a cure — and saying so plainly is what makes a security claim trustworthy. An undocumented gap reads as a bug. A documented one reads as a decision.
What to ask your vendors
If you buy or build software that holds more than one customer's, client's, or department's data in a shared system — and almost every modern SaaS and AI platform does — these are the questions that reveal whether isolation is real or merely hoped for:
- Where is the boundary enforced? If the only answer is "our developers filter every query," that is a hope, not a control. Ask what enforces isolation when a developer forgets.
- What happens when something goes wrong — does it fail open or closed? The right answer is that an unresolved request returns nothing, not everything.
- How is customer identity established, and how many places re-derive it? "Once, at the edge" is a far healthier answer than "everywhere, independently."
- What did you deliberately leave out, and why? A vendor who can answer this precisely has actually thought it through. A vendor who says "everything is fully isolated" with no nuance probably has not.
None of the engineering here is exotic. But the principles behind it — put the boundary where it cannot be forgotten, fail closed, establish identity once, and document what you left out — are the difference between a platform that *says* your data is isolated and one that can *prove* it.