0%

How twenty companies actually run generative AI in production

Every tile below is one shipped deployment: what goes in, what comes out, who checks it, and how long it has been running. Tap one.

Reading time about an hour if you operate the figures. Snapshot: August 2026. Sources at the foot.

The work arrives whether or not anyone has time for it

A doctor at Kaiser Permanente finishes a twenty minute appointment. The patient leaves. What stays behind is a note that has to be written: what was said, what was found, what was decided, what happens next. It goes into the medical record, and other clinicians will act on it. Writing it takes real minutes, and those minutes come out of the same day that the next patient is waiting in.

Now multiply. Kaiser runs 40 hospitals and more than 600 medical offices across eight states and the District of Columbia. Every appointment in that system produces one of these notes. Nobody has to decide whether the work exists. It exists.

That is the shape of almost every deployment in this piece, so it is worth drawing before anything else. Not a technology, not a model: a queue of identical jobs that keeps arriving. One patient visit. One customer message. One product photograph. One contract. One page of a regulatory filing. Each is a unit of work, and the only question any company is really asking is what happens to each one.

picture it Think of a tray on a desk that never empties. Overnight, someone slides another hundred slips into it. Each slip is one job, and each job has to end up somewhere: done by a machine, done by a person, done by a machine and checked by a person, or bounced because nobody could handle it. Everything in this piece is a claim about where the slips go. The models change, the slips do not.

Four things that can happen to one job

Figure 1. Click any tile to cycle it through the four states. The counter underneath re-tallies as you go.

Those four states are the whole vocabulary. A raw tile is a job that has arrived and had nothing done to it. A machine tile was produced by a model and went out as it was. A checked tile was drafted by a model and read by a person before it counted. An escalated tile was handed to a human, either because the machine declined it or because the machine got it wrong.

Hold on to the difference between machine and checked, because it is the single most consequential design decision in every deployment here, and it is the one that press releases are vaguest about. At Kaiser the ambient tool drafts the note and the clinician reviews it before it enters the record; the organisation says plainly that its clinicians are the medical decision makers. Every tile in that deployment is a checked tile, by policy. In other deployments the model's output goes straight to a customer with nobody in between.

Kaiser's tool has supported more than four million encounters. That number only means something once you know which of the four states those four million tiles are in. So the first real question is not how good the model is. It is what the model is even being handed.

The model has never read your refund policy

Try this one in your head. A passenger types to an airline: can I bring my Labrador, Max, in the cabin?

To answer, you need two completely different kinds of knowing. First, that a Labrador is a dog and that a named pet is still a pet. That is ordinary world knowledge, and a language model trained on the open internet has plenty of it. Second, you need that airline's live pet policy: the weight limit, the carrier dimensions, which routes allow it, what it costs. A model trained on the open internet does not have that, and it has no way to tell that it does not have it.

This is the failure that every enterprise deployment starts from, and it is worth being precise about the mechanism rather than calling it hallucination and moving on. A language model produces the next piece of text that is plausible given the text so far. Plausible is not the same as true, and it is definitely not the same as true at this company today. Asked about a pet policy it has never seen, the model will still produce a pet policy, because a fluent pet policy is exactly what plausibly follows the question.

picture it You have hired someone brilliant, well read, and completely new. On day one they can write, summarise, translate and argue. Ask them where the fire extinguisher is and they will confidently point at a wall. They are not lying. They have simply never been in the building, and nothing about being well read tells them that. The fix is never a better hire. It is showing them the floor plan before they answer.

Showing them the floor plan has a name. Everything a model can see when it answers, the question plus whatever text you have placed alongside it, is called the context. The question by itself is a thin context. The question with the airline's actual pet policy pasted in beside it is a fat one, and the answer changes completely.

Same model, same question, two contexts

Figure 2. Click the two buttons to move the policy text in and out of the context. Watch which parts of the answer survive.

Air India shipped exactly this, and shipped it early. Its assistant AI.g went live in May 2023, which makes it one of the oldest generative deployments in this piece. It now takes about 40,000 questions a day across more than 1,300 topics, and has resolved more than 13 million conversations. The airline reports that 97% are handled without a person and 3% go to a contact centre agent. Call volumes stayed roughly flat while passenger traffic doubled.

The Labrador question is Air India's own example, and the airline's point about it is precise: nobody wrote a rule for Labradors. The model knew that a Labrador is a dog. The pet policy, fetched and placed in the context, supplied the rest.

The intelligence was general. The correctness was fetched.

Which raises the obvious question. If the answer depends on pasting the right document into the context, and a large company has hundreds of thousands of documents, how does anything know which one to paste?

Retrieval is a paste job, and that is the whole trick

Morgan Stanley's advisors sit on top of a research archive of roughly 350,000 documents. Decades of analyst notes, market commentary, product rules. Before 2023, finding the paragraph that answered a client's question meant knowing it existed and remembering where. Advisors describe searches that used to run half an hour.

The archive was never the problem. The archive was an asset. The problem was that a question in English and a document in a folder had no way of finding each other.

The technique that connects them is called retrieval augmented generation, which everyone shortens to RAG, and which is less exotic than the name suggests. Three moves, done in order, every single time somebody asks something.

Cut the archive into pieces

A 90 page research note is not a useful thing to fetch, because most of it is irrelevant to any one question and the context has a size limit. So every document is split into passages of a few hundred words. Each passage is called a chunk. Morgan Stanley's 350,000 documents become millions of chunks. This is done once, in advance, not while the advisor waits.

Give every chunk a set of coordinates

Each chunk is run through a model that turns text into a list of numbers, typically several hundred of them. That list is called an embedding, and its useful property is that passages about similar things land near each other, even when they share no words. A chunk about "cabin pets" sits near a question about "my Labrador" because the numbers encode meaning rather than spelling. The whole set of coordinates is the index.

Turn the question into the same kind of coordinates

When the advisor types a question, the question goes through that same embedding model. Now the question is a point in the same space as every chunk, and "which passages are relevant" becomes an arithmetic question: which points are nearest. That is the move that makes the archive searchable by meaning.

Paste the nearest few in, and only then ask the model

The top handful of chunks, usually between three and ten, are pasted into the context above the question. The model reads them and answers. It is not recalling the archive. It is reading photocopies that were handed to it a quarter second earlier, which is why it can also print where each claim came from.

The citation is the audit trail

Because the system knows which chunks it pasted, it can attach them to the answer. This is what makes retrieval acceptable in regulated work. An advisor can click through to the source note. At Novo Nordisk the same mechanism runs in reverse: once a human expert approves a definition, it is stored and reused verbatim in every future document on that topic, so the approved words are retrieved rather than regenerated.

picture it A librarian who has not read your books but is extremely fast. You ask a question at the desk. She does not answer from memory. She walks the stacks, pulls four pages, drops them in front of you, and only then talks you through what they say, pointing at each page as she goes. If the pages are wrong, her answer is wrong, and she will still sound certain. Retrieval moves the accuracy problem out of the model and into the filing. That is a much better place for it, because filing is something a company can inspect and fix.

Retrieval also explains something that otherwise looks strange, which is how fast some of these systems were built. Moderna's first internal assistant, mChat, went from nothing to running in about two weeks in early 2023. It reached more than 80% internal adoption. Nobody trained a model. The company already had its documents; retrieval is mostly plumbing between an index and an existing model.

It also explains the shape of the biggest reported win in this whole set. Novo Nordisk's clinical study reports run to 300 pages and summarise a drug trial for regulators. Staff writers averaged 2.3 of them a year. The company built NovoScribe on retrieval plus approved text, and reports that drafting fell from ten to fifteen weeks to about ten minutes, with review cycles roughly halved. More than fifty writers on a report became three working with the model. The company has said its annual spend on the model is less than one writer's salary.

Ten minutes is not a claim that a machine wrote a regulatory submission unsupervised. It is a claim about the drafting step, with the review step still there. Pfizer's version is smaller and easier to believe: scientists face roughly 20,000 documents per drug programme, and its VOX platform lets them ask in plain language or by voice, which the company estimates saves up to 16,000 hours of searching a year.

Everything so far has assumed the work arrives as text. Most of it does not.

A conversation is not text until something turns it into text

Go back to the Kaiser exam room. The job that arrives is not a document. It is two people talking, with interruptions, an accent, a rustle of paper, a name said once. Before any language model can touch that, the sound has to become words.

That conversion has a name and a separate model. Morgan Stanley's meeting tool, Debrief, runs Zoom audio through Whisper, a speech recognition model, and then hands the transcript to a language model that writes the client note and the follow up actions, which land in the CRM. Two models, in a line, doing two different jobs. Kaiser's ambient tool does the same thing in an exam room, in more than fourteen languages and across more than fifty specialties.

The general word for the kind of thing going in or coming out is modality: text, speech, images, video, code, structured data. Every deployment in this piece is a path from one modality to another, and the path is usually the most concrete thing you can know about it.

picture it A translator who only reads. Whatever you want translated has to reach her as writing, and whatever she produces comes back as writing. If you arrive with a cassette, someone types it up first. If you need it spoken aloud at the other end, someone reads it out. Those two someones are separate hires with separate skills. Most systems that look like one clever machine are three plain ones standing in a row, and knowing which row a company built tells you more than knowing which model they licensed.

The same pipeline, six different pairs of ends

Figure 3. Click a path to redraw it. The middle stays a language model in every case; only the ends change.

Two of those paths are worth sitting with, because they behave very differently from the text ones.

Text to image, at Zalando. The European fashion retailer generates editorial campaign imagery rather than photographing it. By the fourth quarter of 2024 roughly 70% of its editorial campaign images were generated. Its head of content solutions told Reuters that production fell from six to eight weeks down to three or four days, with costs down as much as 90%. The reason he gives is not cost, though. It is speed: a trend surfaces on social media on a Tuesday and is dead in a fortnight, and a six week shoot cannot answer it. When "brat summer" broke, Zalando could ship imagery inside days.

Text to video, at Netflix and Coca-Cola. Netflix's co-chief executive confirmed on an earnings call that a building collapse sequence in the Argentine series El Eternauta was the first generated final footage in a Netflix original, made with its in-house Eyeline Studios roughly ten times faster than conventional visual effects. His argument was budget rather than ambition: at that show's scale the shot was otherwise unaffordable. Coca-Cola went further and made whole holiday commercials this way in 2024 and again in 2025, with about a hundred people on the project, five of them AI specialists generating tens of thousands of clips, and a production calendar that fell from about a year to about a month across roughly 140 countries.

Notice how the review gate moves. A generated research summary is checked by the advisor who asked for it. A generated commercial is checked by the entire internet, at once, and both Coca-Cola spots drew heavy public criticism. Same technique, wildly different last mile.

So far each system has had one door and one kind of thing coming through it. Two of the largest deployments here do not.

One model with several senses is not the same as several models behind one door

Two phrases get used as if they were interchangeable, and they describe opposite architectures. It is worth pulling them apart with a concrete question in hand.

A shopper photographs a broken bracket under a sink and types: what is this and will it fit my tap?

There are two honest ways to build a system that answers. In the first, a single model accepts the photograph and the sentence together, in the same context, and reasons across both. It is multimodal: one model, several senses. In the second, an image model describes the photograph, a text model reasons about the description, and a piece of routing code decides which model sees what. That is multi-model: several specialists, one front door, and a router in the middle choosing between them.

picture it Multimodal is one doctor who looks at you, listens to you, and reads your chart before speaking, holding all three at once. Multi-model is a clinic: a receptionist decides whether you need the radiologist, the dermatologist or the pharmacist, and each of those only ever sees their own slice. The clinic is cheaper to run and easier to swap staff in and out of. The single doctor notices things that fall between the specialties. Real deployments run both, at different layers.

Two architectures answering the same question

Figure 4. Switch between the two. The layout is deliberately identical, so the only thing that moves is what each box can see.

JPMorgan Chase built the clinic. Its internal platform, LLM Suite, went live in the summer of 2024 as a front door that routes employees' requests to several outside frontier models while the data stays inside the bank's own environment. The bank's chief analytics officer has described an early contrarian bet: that the models themselves would become a commodity, and that the connective tissue around them was the defensible part. The platform went from zero to 200,000 onboarded users in eight months, reached roughly 250,000 with access, and the bank reports that more than 60% of employees use it. Its retrieval layer is on its fourth generation and now handles several modalities. Roughly 450 production use cases are running, with a stated target of 1,000 by the end of 2026.

Amazon built the other thing, in the place where the difference is worth the most money. Rufus, its shopping assistant, combines retrieval with a knowledge graph called COSMO that encodes context: who buys a thing, for what, when. Amazon reported that more than 300 million customers used Rufus during 2025, with monthly users up 149% and interactions up 210%, that shoppers who use it are about 60% more likely to complete a purchase, and that it drove close to 12 billion dollars in incremental annualised sales.

Both companies routed a lot of work through models without ever training one. That distinction matters more than it sounds. Almost nothing in this piece involved building a model. It involved building the pipes.

And once the pipes exist, a much less comfortable idea becomes available: the model does not have to stop at answering.

The model stopped answering and started doing

Everything so far ends with text appearing on somebody's screen. A person reads it and then does something. Take that person out of the middle and you have the architecture that every vendor in 2026 is calling an agent, and which is easier to understand as a loop than as a noun.

Give the model a short list of things it is allowed to call: look up an order, check stock, issue a refund up to fifty pounds, write a detection rule. Each of those is a tool. Then let the model, after each result comes back, decide whether it is finished or needs to call another one. That decide, call, read, decide again cycle is the agent loop. There is no new kind of model involved. The change is that the output is wired to something that acts.

picture it The difference between an assistant who writes you a note saying "you should cancel the 4pm and rebook the flight" and an assistant who has the calendar password. The second one is not smarter. She has been given hands, and the interesting question immediately stops being how good her judgement is and becomes how much she is allowed to touch before someone signs off. An agent is a permission list attached to a language model.

Something arrives that is not a question

Commonwealth Bank's fraud systems watch more than 80 million signals a day across roughly 20 million payments. A cluster of transactions starts to look unusual in a way that no existing rule describes. Under the old arrangement this became a ticket for a human analyst, and the queue of tickets was the bottleneck.

The model calls a tool instead of writing a paragraph

The agent queries the transaction data itself. This is the move that separates an agent from a chatbot: the model emitted a structured call rather than prose, something executed it, and the result came back into the context. The model now knows something it did not know a second ago, and that something came from the bank's live systems.

It reads the result and decides it is not finished

Severity assessed, context pulled, the loop goes round again. Each turn is cheap. A hundred turns is not, which is why real deployments cap the loop and why the cap is a design decision rather than a technicality.

It produces something executable, not something readable

The output is a proposed detection rule. CommBank reports that its agent has contributed to developing or updating three quarters of the bank's card fraud rules, and that its in-house teams built the system in three months.

A person still signs the rule

The bank is explicit that its AI models sit under its risk frameworks with clear human accountability. This is the review gate again, in the highest stakes position it occupies anywhere in this piece: the agent drafts the rule, a human approves it, and the approved rule then runs unsupervised over millions of payments. Fraud losses fell by more than 20% in the first half of the 2026 financial year against the same period the year before.

The consumer versions are less dramatic and more revealing about where this is going. Amazon has given Rufus the ability to buy: it can shop other stores directly, and it will hold a target price and purchase when the price drops. Walmart's Sparky launched in June 2025 as a shopping agent and is now wired into fulfilment, and Walmart reports the value of goods sold through Sparky rising 150% in a quarter, baskets 35% larger than non-Sparky baskets, and weekly active users up over 110% quarter on quarter.

Both of those numbers deserve a raised eyebrow, and it is the same eyebrow you should raise at every figure in this piece: the customers who choose to use a shopping assistant were probably already the ones intending to buy. Amazon's own framing, "60% more likely to complete a purchase", compares people who used the tool against people who did not, which is not the same as what the tool caused.

Once a model has hands, the question that has been implicit since the first tile becomes the only question that matters, and it turns out to have a dial attached to it.

One dial decides how much of the queue the machine keeps

Here is the decision, stripped of everything else. A job arrives. The system forms some estimate of how well it can handle this particular one: call it a confidence score, which in practice is assembled from retrieval quality, the customer's wording, sentiment, how unusual the request is, and how expensive a mistake would be. Then it compares that score against a line somebody drew. Above the line, the machine keeps the job. Below it, a human gets it.

That line is the escalation threshold, and where a company puts it is the single most consequential number in its deployment. Move it down and the machine keeps more work, which is cheaper and faster. Move it down far enough and the machine starts keeping work it should not have kept, and those mistakes reach customers.

OpenTable built a version of this you can actually see the guts of. Working with Salesforce, its team scores every conversation live: asking for help adds 5, asking for a representative adds 10, typing in capitals and clear frustration takes it to 20. Cross the threshold and the conversation goes to a person. The threshold is adjustable, and they move it: raise it when staffing is tight, lower it at holidays.

picture it A triage nurse at the door of a busy clinic, with one instruction: send through anyone you are less than X percent sure about. Set X at 99 and almost everyone goes to a doctor, the queue stretches round the block, and the nurse was pointless. Set X at 40 and nobody waits, until the morning someone with chest pain is sent home with paracetamol. There is no correct X. There is only which mistake you would rather make, and that is a business decision wearing a technical costume.

Drag the threshold and watch both costs move

Figure 5. One hundred jobs, each with its own difficulty. Drag the slider to move the line. Tiles above it are kept by the machine, tiles below go to a person, and the red ones are jobs the machine kept and got wrong.

The presets are the argument. Klarna set the dial low and told everyone. In February 2024 the Swedish payments company announced that its OpenAI-powered assistant had handled 2.3 million conversations in its first month, roughly two thirds of all customer service chats, across 23 markets and more than 35 languages. Resolution time fell from 11 minutes to under 2. Repeat contacts fell 25%. The company put the work at the equivalent of 700 full time agents and estimated 40 million dollars of profit improvement for the year, later revising to about 60 million and 853 agent equivalents by the third quarter of 2025.

Air India sits further left still, containing 97% of 40,000 daily queries, and has run there for years without the reversal Klarna had. The difference is not the model. Airline questions are narrow, authenticated and highly repetitive, and 1,300 topics covers most of the world. Klarna's queue contains disputes, refunds and people who are upset about money.

Kaiser sits at the other end and never moved: every draft note is reviewed by the clinician before it enters the record. Four million encounters, zero unreviewed tiles. That is not timidity. It is a correct reading of what a mistake costs when the artefact is a medical record.

Nobody in this piece found the right setting. They found the setting their queue could survive.

Which brings us to the more awkward part, because the numbers those companies publish are not measuring the same thing as each other. Sometimes not even close.

Ninety-seven percent of what, exactly

Take one hundred customers who open a chat window. Sixty get an answer from the model and close the window without saying anything. Fifteen get an answer, are unsatisfied, and give up. Twenty ask for a person. Five come back the next day with the same problem.

Answer quickly: what is the automation rate?

You can defend 60%. You can defend 80%, if you count everyone who did not reach a human, and giving up counts. You can defend 75% if you count the twenty escalations as failures but not the abandonments. And if the five repeats are logged as new conversations, the denominator changes underneath all of it. Every one of those is a number a company could publish honestly, and they differ by twenty points.

This is not a hypothetical. Salesforce's own published customer results for Agentforce span 25% to 95%, because the underlying definitions differ: 1-800Accountant reported 90% case deflection during tax week, OpenTable around 70% of enquiries resolved, Pandora 60% deflection, Reddit 46% case deflection, GE Appliances 25%. Salesforce's own benchmark, CRMArena-Pro, put an out-of-the-box agent at roughly 35% accuracy before customisation, which tells you how much of those customer numbers is configuration rather than model.

picture it A shop that measures success by how few people ask an assistant for help. Empty the shelves and the number goes to a hundred percent. Deflection counts the absence of a human, not the presence of a solved problem, and those two things come apart precisely where the queue is hardest, which is the part any buyer most wants to know about.

The same hundred conversations, counted three ways

Figure 6. Click each definition. Nothing about the conversations changes; only the rule for what counts as a success. The headline number moves by 22 points.

A second, quieter version of the same problem runs through the internal deployments. Morgan Stanley reports that 98% of financial advisor teams have adopted its assistant. JPMorgan reports more than 60% of employees using LLM Suite. Moderna reports its legal department at 100% and an average of about 120 conversations per user per week. Goldman Sachs put its assistant in front of all 46,500 employees in June 2025 after a 10,000 person pilot.

Adoption is a real signal and a genuinely hard thing to achieve. It is also not output. A tool that everyone opens once a week and a tool that removes a week of work per person produce identical adoption charts. Moderna's per-user conversation count is the more informative number precisely because it is harder to reach by mandate, and Goldman's is more informative still, because it reports an outcome: developers describing roughly 20% higher productivity and 15% fewer post-release bugs.

None of which tells you the thing a company deciding whether to start most wants to know, which is how long any of this took.

The demo takes a fortnight and the rollout takes two years

Moderna built mChat, an internal assistant on OpenAI's API, in about two weeks in early 2023. It worked, and more than 80% of the company used it.

Zalando's move to 70% generated editorial imagery is estimated to have taken eighteen to twenty-four months. Both statements are true, and the gap between them is the actual subject of enterprise AI.

What takes the two years is never the model. It is the index nobody had built, the permissions nobody had mapped, the review step nobody had staffed, the legal sign-off on training data, the connectors into the CRM, and the several months during which the tool exists and nobody uses it. Harvey, the legal platform deployed at A&O Shearman and PwC, publishes a deployment cadence of six to nine months per firm and runs a forward-deployed engineering team that sits inside firms mapping partner-by-partner workflows and rebuilding knowledge bases. That is the real work, and it is not AI work.

picture it Plumbing a house. Choosing the taps takes an afternoon and it is the only part anyone enjoys. Then someone has to find where the mains comes in, discover the previous owner's improvisations behind the plaster, and get an inspector to sign it. The tap is the model. Everything else is eighteen months. Companies that report fast rollouts almost always had the pipes already, which is why Moderna's two weeks came after years of standardised data infrastructure.

Eleven deployments, from first pilot to reported scale

Figure 7. Click any bar to read what happened in that gap. The dark segment is pilot, the filled segment is the scaled rollout.

Two patterns fall out of that chart and both are slightly deflating.

The first is that the earliest movers are ordinary companies with unglamorous problems. Air India shipped in May 2023, before most of the technology industry had a strategy. It shipped a support assistant, which is the least fashionable use case available, and it is still running.

The second is that almost every deployment here started as a narrow, boring, high-volume workflow with a metric the company already tracked. Note-taking. Research lookup. Order taking. Campaign imagery. Nobody transformed anything. They picked a queue, measured it, and pointed a model at it.

The ones that went wrong went wrong in the same few ways, and the failures are more instructive than any of the wins.

Every one of these has a scar

Klarna is the important one, because Klarna went first and said so loudly, which means the correction is public in a way most corrections are not.

In May 2025 its chief executive said the company had let cost dominate the decision and that the result was lower quality, and Klarna began recruiting human agents again, in a flexible pool, with a commitment that a customer can always reach a person. It is worth being accurate about what changed: the assistant still handles roughly two thirds of chats. The company did not reverse the deployment. It moved the threshold back up for the cases where machine parity had not actually held, and it discovered which those were by shipping.

Then there is what happens when the input is hostile. Taco Bell's parent, Yum! Brands, expanded voice ordering from a hundred-odd US drive-thrus to around 500 restaurants with Nvidia in March 2025. By August it had slowed the rollout: customers complained, and people worked out that the system could be goaded, including one order for 18,000 cups of water. A drive-thru microphone is an open text field pointed at the public, and nothing in a retrieval pipeline anticipates a person who is there to break it.

picture it A restaurant kitchen that has never had a customer walk in drunk, never had a supplier send the wrong crate, and never had a food critic in the room. It runs beautifully for a year. None of the three things that eventually close it are about the cooking. A system that has only been tested on the traffic it expected has not been tested. Every failure below entered through a door nobody was watching.

Coca-Cola's is a third kind of failure, and the one with no technical fix. The 2024 and 2025 holiday spots were competently made and widely disliked. For 2025 the company switched from generated humans to animals specifically to dodge the uncanny valley, and its global head of generative AI said the craftsmanship was ten times better. The complaint was never craftsmanship. It was that the audience found out.

Six ways a working system still fails

Figure 8. Click a failure to see which stage of the pipeline it enters at, and what the fix was.

Salesforce is the uncomfortable one to end on, because it is the vendor. Marc Benioff said in September 2025 that the company had taken its customer support headcount from around 9,000 to around 5,000, with AI handling roughly half of customer interactions and support costs down 17%. Whatever else that is, it is a company selling a product and demonstrating the product on itself in the most legible possible way.

The honest summary of the failure list is that none of these are model failures. Klarna's was a threshold set by finance. Taco Bell's was an unguarded input. Coca-Cola's was a misread of the audience. The stale index and the drifting metric are operations problems that predate language models by decades.

The models mostly worked. The systems around them are where everything happened.

Which means it is worth drawing that system once, properly, since all twenty are built from the same six layers.

Twenty deployments, six layers, one shape

Strip the branding off every system in this piece and the same stack is underneath. Not because anyone standardised it, but because each layer solves a problem you hit in a fixed order once you try to put a model in front of real work.

Morgan Stanley's evaluation practice is the layer people skip, and it is the one that separates the deployments that lasted from the ones that got quietly retired. Before Debrief shipped, the team built evaluation datasets covering different meeting types and tested specifically whether the model captured the action items without inventing any. That is not a benchmark score. It is a set of examples from the actual queue, with correct answers written by people who know the work, run again every time anything changes.

picture it A restaurant. The chef is the model, and hiring a better chef is the part everyone fixates on. But the kitchen also needs a supplier who delivers the right ingredients (retrieval), a pass where dishes are checked before they leave (the review gate), a rule that certain orders always go to the head chef (escalation), someone tasting the same three dishes every week to catch drift (evaluation), and a policy about what the kitchen refuses to cook (guardrails). Swap the chef and the restaurant runs. Remove the pass and it closes.

The stack every one of them converged on

Figure 9. Click a layer to expand it and see which companies are doing what at that level.

The layer that gets built last and should be built first is the review gate, and you can predict a deployment's trouble almost perfectly from where it sits. Kaiser put it on every single tile and has run four million encounters without a public incident. CommBank put it on the rule, not the transaction, so a human approves the pattern and the machine then applies it millions of times unsupervised. A&O Shearman's guidance to its lawyers, from the beginning, was that everything coming out of the system must be validated. Coca-Cola's gate was the general public.

Guardrails are the other layer worth naming, because they are the reason a drive-thru microphone is harder than a research archive. A guardrail is a rule applied outside the model: caps on quantities, refusal of certain topics, no free-text into a payment field. Air India reports no inappropriate responses over more than eighteen months of operation, which is a claim about guardrails and topic scoping rather than about the model's manners.

What twenty roll-outs will tell you if you let them

Read the roster below in one pass and a small number of things are true of nearly all of it.

Almost nobody trained a model. They bought access to one, or several, and spent the effort on an index, a router, a review step and an evaluation set. The workflows chosen were narrow and high-volume, with a number the company was already reporting to somebody before the model arrived. Every deployment that survived contact with the public has a place where a human signs, and the companies that had trouble are the ones that moved that place to save money rather than because the work had earned it.

picture it Twenty restaurants that all turned out to have bought the same oven. Nobody coordinated it. They each had a queue of orders, each tried a few things, and each arrived at the same arrangement because the arrangement is what the problem demands. The convergence is evidence about the problem, not about the vendors. If you have a queue with a number attached, the shape of what you should build is already largely decided.

And the numbers, all of them, come from the companies that built the things. Every figure in this piece is sourced, and almost none of it is audited.

The roster

Figure 10. Filter by what the system does, then click any tile for the full record: modality, architecture, live date, scale, published numbers and the caveat.

Pick the queue first. The model is the last decision you make, and the least interesting one.

The thing worth taking from all of this is unglamorous and slightly annoying: none of these companies is doing anything you could not describe to a colleague in four sentences. Fetch the right documents. Put them next to the question. Let the model draft. Decide who checks it. The gap between the companies that got value and the ones that did not is almost entirely in the fourth sentence.

Now set the dial yourself

picture it A mixing desk with four faders, where every combination sounds like something and only a few sound good. Push the threshold down and the room gets loud and cheap. Push review coverage up and it gets quiet and expensive. The setting you want is the one where the errors you still make are errors you could live with explaining. Try to find it before reading anyone else's press release.

The argument is over. This is the same simulation from the threshold figure with everything unlocked: the queue's difficulty, how expensive an error is, whether a review gate exists, and where the line sits. The presets rebuild three of the deployments you have just read about. Break them.

The routing board

Figure 11. Four controls, one queue of two hundred jobs. The readout underneath is the only honest scoreboard: what shipped, what a person touched, and what went out wrong.