ACTO

The client runs the patient support line for a health system. Most of what came in was the same questions rephrased a thousand ways — booking, coverage, visit prep, lab results — and every one went to a frontier model at full price.

Paying the top rate was only half the problem. The other half was that one general model is the wrong place for most of these questions. Some need something done, like moving an appointment, which is a job for something with access to the booking system. Others need an answer from the client’s own medical documents, which is a job for a model that has read them.

That second kind is where the stakes are. A health question has to reach a model that answers from the client’s documents. Sending it somewhere with no medical knowledge is the worst thing this system can do, and most of the design is arranged around not doing it.

Our team built ACTO to sit in front of that traffic. It answers repeats out of Redis without calling a model, classifies whatever’s left, and sends it to one of two kinds of destination: an admin agent with tools that act on the client’s site, for booking, coverage and visit prep; or one of the RAG models on our platform, each backed by the LLM the client chose — Opus, GPT, whatever they prefer — and answering from the client’s own documents.

Clients set up the RAG side themselves, through a frontend service we built. They pick the LLM behind each model, upload documents that get indexed automatically, add example questions so the router knows what belongs there, and can remove any of it. That last part matters for everything below: the client’s example questions are what the router matches against.

My part: I owned the routing side — the embedding, the cache, the classifier, the routing, and the eval harness. My teammate built the admin agent and the RAG pipelines. Every response comes back with the answer plus the label, the destination, whether it came from cache, whether the LLM classified it, and how long it took.

The query goes through bge-large-en-v1.5, which turns text into an embedding: 1,024 numbers standing in for what the text means, so two ways of asking the same thing land near each other. It’s self-hosted inside the virtual private cloud, because patient queries carry PHI and that content isn’t leaving the private network to be embedded by somebody else’s API.

One vector gets made per query, and it gets reused three times: by the cache check, by the classifier, and overnight by the job that clusters the day’s log without re-embedding anything.

The cache holds a small, curated set of question-and-answer pairs. A new query is compared against them by cosine similarity, and above about 0.92 the stored answer comes back without a model being called at all.

What makes it semantic is that the embedding places text by meaning rather than wording, so two questions that mean the same thing land close together even if they share no words. “How do I cancel?” and “cancel my appointment” score high. “How do I cancel?” and “how do I book?” share a word and score lower. Nobody decides by hand what counts as the same question; the threshold does.

How we got 0.92: empirically. We built two sets of pairs: paraphrases that should hit, same question in different words, and near-misses that must not, similar wording with a different answer, like cancel versus reschedule. We scored every pair, looked at where the two groups landed, and picked the value that kept the near-misses below it.

It’s high on purpose, because the two mistakes don’t cost the same. A miss costs one model call. A false hit serves someone a confidently wrong answer. We’d rather pay.

Keeping the set small and curated does the other half of the work: a hit returns a vetted answer, not an arbitrary model generation from three weeks ago. And nothing user-specific or action-based ever lives there. A booking can’t come from cache, because it isn’t an answer, it’s an action on someone’s record.

One number that doesn’t travel: 0.92 belongs to this embedding model. Every model spreads meaning differently, and when I rebuilt the system on MiniLM, paraphrases landed near 0.75, so 0.92 would have missed almost everything. Swap the model, redo the pairs.

On a miss, the same vector is compared against the client’s example questions — one matrix multiply, no model call — and the best-matching example proposes a label. The classifier only takes it if two things hold: the top match clears a high acceptance bar, and it clearly beats the second-best label by a margin. A high match that’s nearly tied with another label isn’t trusted, and neither is a query that doesn’t look much like any example.

Anything that fails either check goes to an LLM classifier. It gets a prompt with the query, a description of each destination, and example questions for each, and picks one at temperature 0.

That split is the design. The embeddings settle the clear cases for free, and the LLM only runs on the uncertain slice. Both thresholds, the acceptance bar and the margin, were picked the same way as the cache’s: from measured pairs, not by feel.

The label maps to a destination through the client’s config. Admin labels go to the agent; health labels go to the RAG model configured for that topic. When a client adds or removes a model in the frontend, routing picks up the change without a deploy.

Once the destination answers, the query, its vector and the answer are appended to the day’s log, and that’s all. Nothing on the request path writes to the cache.

Overnight, a job clusters the day’s log using the vectors already stored with each query, so nothing is re-embedded. Counting exact strings would undercount the paraphrases, and paraphrases are the whole point: twenty ways of asking about parking are one intent worth caching, not twenty one-offs. The largest groups get promoted into the cache, and the previous night’s harvest is flushed. The hand-curated entries are never touched.

The clustering threshold is 0.40, and it isn’t a guess either. Queries within one intent scored 0.49 and above, queries from different intents scored 0.29 and below, and nothing landed between. With separation that clean, the cut goes in the gap.

The harness runs a frozen ground-truth set through the full classifier: embeddings, acceptance bar, margin and LLM classifier. It never touches the endpoint, Redis or the destinations. The embedding stage is free and deterministic, and only the uncertain queries call the LLM, at temperature 0, so a run is cheap and close to repeatable. That’s what let it run on every change.

The test set was generated with an LLM and approved by the client, so every label was signed off by someone who knows the domain. Each query carries its correct label, so grading is automatic: compare the predicted label with the stored one.

A variant is a different version of the prompt the LLM classifier gets, and that prompt has two parts that matter: a description of each destination, what the admin agent and each RAG model handles, and example questions for each. Everything else stays fixed — the embedding stage, the thresholds, and the test set, with its queries embedded once and reused — so any change in the score comes from the prompt alone. And each variant changes one thing, so it’s clear what moved it.

The output is overall accuracy, per-label precision and recall, and the most frequent confusion pairs. The confusion pairs are what drive the next prompt change.

The examples. Variant A used only the client’s real queries, and some labels were too thin to cover how people actually phrase things. Variant B added LLM-generated queries modelled on the client’s. What worked best was the client’s real queries plus general health questions from the field, which covered phrasings the client’s own data didn’t.

The destination descriptions. Early prompts described each destination in a line or two, so when two sounded similar, the LLM had little to go on. We rewrote them to say what each one covers, what it doesn’t, and how it differs from its closest neighbour. That gave the LLM a real reason to pick one over the other, and it went straight at the top confusion pairs.

The loop was the same throughout: run the eval, find the worst confusion pair, change one thing in the prompt, and keep it only if the score held. Misclassification went from 28% to 15%.

The score the harness leads with weights recall on health labels above everything else. A health question sent to the admin agent gets an answer from something with no medical knowledge. That’s a safety failure, not a cost one, and it doesn’t belong on the same scale as a booking question landing in coverage.

So a variant can come out slightly worse on raw accuracy and still be the right one to ship, and the weighted score says so where accuracy alone wouldn’t.

What the harness doesn’t measure is answer quality. It checks whether the label was right, not whether the destination answered well. That needs its own set.

ACTO’s main website is built on Laravel, and every request moves through the same four steps. A route maps a URL and method, like POST /appointments, to a controller method. The controller reads the input, calls the business logic, and returns a response; it’s meant to be a thin traffic director, not where the logic lives. Validation checks the input is what you expect before anything uses it. And the response is either the result or an error saying what was wrong.

The twelve-plus endpoints that take user input were the ones that mattered:

Before, each controller method validated its own input at the top. The same rule, like “email must be valid”, was written in several places, and each endpoint returned errors in its own shape. The frontend had to handle failures differently per endpoint, and QA had to recheck bad-input handling on every one.

A Form Request is a class Laravel gives you for the validation step. The rules, and optionally who’s allowed to make the request, go in that class, which is then type-hinted in the controller method. Laravel runs it automatically before the controller code, and if the input is bad, it sends back an error response on its own. The controller never sees bad input.

After, a request hits an endpoint and Laravel runs that endpoint’s Form Request first. If validation fails, it returns a standard 422 with the errors listed per field, and the request never reaches the controller. If it passes, the controller gets input that’s already been validated and only handles the actual work.

Three things came out of that. Checks happen up front, so controllers stay thin and focused. Each endpoint’s rules live in one named class that’s easy to find and review. And the errors are consistent by default — enforced by Laravel itself, not by a team convention someone can forget.

The result: the frontend handles every failure the same way, and QA reviews the rules in the Form Request classes and trusts the error behaviour, instead of poking at every endpoint by hand. That’s the five hours a week.

In the routing. The thresholds are global, when some labels should be much harder to hit than others. The nightly clustering is greedy and order-dependent, so the same log in a different order groups slightly differently. Confidence is a margin, not a calibrated probability. New labels are only as good as the client’s example questions. And nothing learns from a user rejecting an answer.

In the eval. The test phrasing is cleaner than real traffic. Temperature 0 is near-deterministic, not guaranteed. Prompt changes only reach the uncertain slice, so if the embeddings confidently pick the wrong label, no prompt fixes it. And when a client adds a model, its description and examples need to be good, and it needs test queries too, or the eval won’t notice it’s broken.

In the validation. More files to maintain, and rules shared across endpoints can still end up duplicated between Form Requests unless they’re pulled into shared rule classes.

Work out which mistake is expensive before tuning anything. A miss against a false hit, a cost failure against a safety failure. Once those are written down, the thresholds and the scoring weights follow from them instead of from taste.

Spend the expensive thing only where it’s needed. The embeddings do the clear cases for free, and the LLM only sees the uncertain ones. The same idea runs through the whole layer: one vector, made once, reused three times.

A measurement cheap enough to run on every change gets run. Keeping the harness off the live system and pinned at temperature 0 mattered more than making it exhaustive, because it meant it ran on every change rather than before releases.

Scope is worth stating plainly. Our team built ACTO end to end. I owned the layer that decides where each query goes, and saying that reads better than implying I built the whole thing.