Courtyard AI

Courtyard makes small businesses discoverable inside AI assistants. A business connects its website, Courtyard extracts a knowledge base from it, publishes that as a live ai.{hostname} site, registers it with search engines and AI answer engines, and gives the owner an agent, Eve, they can talk to about their own business.

I worked on the backend of the main product, and my term split into two halves. The first was discovery and content: getting published sites indexed by Google, Bing, IndexNow and Brave, warming AI answer-engine caches, and building the image pipeline that fills a customer’s media library from their own website. Along the way I built the eval suite for Eve.

The second was the SMS growth loop. A stranger texts our number, gets a free analysis of how visible their site is to AI assistants, signs up for an account entirely over text, and pays over text too. No browser at any point.

When a customer’s site went live, nothing told Google, Bing or IndexNow it existed. Pages published later were never resubmitted, and there was no way to know whether a submission had worked.

Going live is messier than a flag flipping. DNS has to propagate, and the site’s TLS certificate on CloudFront is issued asynchronously. Until that certificate is applied, the site isn’t reachable over HTTPS, even when DNS already points at it.

So registration is one job with a fixed order. It polls the site for liveness: four attempts ninety seconds apart, each fetch with a ten-second timeout. If the certificate issues during that window, the job applies it itself, because nothing else in the system does. Once the site answers, it fans out to seven providers with Promise.allSettled, and it saves every provider’s result before deciding whether to fail, so even a run where everything failed leaves a record of what was tried and why.

The first version hooked the wrong place. I put the trigger where I happened to see the site’s active flag flip. A QA review found three structural problems: duplicate jobs from race conditions, four other code paths that flip the same flag without ever creating the job, and a permanently stuck state if job creation crashed after the flip. The “fires once on go-live” guarantee wasn’t enforced anywhere.

I couldn’t fix the four-paths problem without understanding how certificates, CloudFront tenants and the two DNS setup flows actually worked, so I stopped and read. That surfaced two things I hadn’t known.

Certificate issuance isn’t certificate application. The certificate authority issues it, but HTTPS doesn’t work until it’s applied, and after the flag flipped nothing reliably applied it. A job running ninety seconds after DNS verification would have failed in production for most users.

Failed jobs are terminal. The queue has no automatic retry. If the handler throws, the job is dead.

Those two facts together are why the final design is one job with a wide window that applies the certificate itself, rather than a short job assuming something else did. I also moved the trigger to the one point where both DNS setup flows converge, so a single insertion point covers every path. The window grew across revisions too, from three attempts thirty seconds apart: once a job only gets one shot, it has to cover the worst case of issuance plus propagation, not the typical one.

The advisory lock bug, the one I’d lead with. To keep registration idempotent, I wrote a helper backed by a Postgres advisory lock that deduplicated against jobs in a started state. Separately, as defence in depth, I removed a guard so certificate activation kept running on every publish as a fallback. Each change was correct on its own. Together they were a bug: the helper now ran on every publish, and a job that had already succeeded no longer counted, so every future publish created a fresh registration job and hit all four provider APIs again.

The fix was small. The lesson is the shape of the mistake: neither change was wrong, and the interaction only existed because I’d reasoned about them in separate sittings.

The Bing tag, and infrastructure disagreeing with code. Bing verification needs a meta tag live on the site. My first version patched it straight into the stored index page in S3, which worked once and was then wiped by the next publish. The fix was to inject Bing’s token into the page’s frontmatter alongside Google’s, so the template renders it on every publish.

Verification still failed. The tag was provably in storage, but CloudFront’s edge cache was still serving the old HTML when my check ran thirty seconds later. Waiting sixty seconds fixed it. It’s the clearest example I have of the code being correct and the deployed system still being wrong.

A miss worth being honest about. Weeks after the first version shipped, a fifth activation path was added that I had no visibility into, and registration never fired for it. The original four were something I found and fixed. The fifth one found me.

Search submission originally lived inside the knowledge publish job, so retrying a failed indexing run meant re-running the entire publish pipeline, with nothing to monitor or debug on its own. I pulled it out into its own job with its own lifecycle, logging and retry.

The first version was a hardcoded orchestrator. I rewrote it behind a provider contract, and later three more providers were added in three pull requests with zero orchestrator changes, which is the proof the abstraction earned its keep.

Changed pages travel from the publish job on a new JSONB metadata column, so the job submits only what changed, and all nine places that trigger a publish got that behaviour without being touched. The handler treats the metadata as untrusted: it reads the hostname from the business record, parses the metadata behind a strict type guard, and falls back to every visible page if it’s missing or malformed.

Deleted pages are tracked separately from changed ones, because providers want different things. IndexNow uses a 410 tombstone to de-index, Google relies on the sitemap, and Bing recrawls:

The redactor, and why order matters. Provider errors are built from raw HTTP responses and written somewhere an admin reads them, so I wrote a six-pattern redactor, with errors capped at 500 characters. Bearer tokens have to be stripped first. Otherwise the generic key-and-value rule matches “authorization:” as the key, consumes the word Bearer as its value, and leaves the real token exposed.

Customer knowledge pages had no imagery, and scraping a small-business site naively returns hundreds of spacer GIFs, nav icons, third-party insurance logos, stock photos, and the same photo at six different URLs.

The answer is a five-stage, in-memory pipeline, plus an owner-facing media library, a publishable media page, on-demand recrawl and a rolling cron. It’s also the first thing I planned before building: a parent ticket with sub-tickets, a design doc, and throwaway scripts run against real sites before any of it went into the app. That was a direct correction of an earlier ticket, where I started deleting code before I understood the pipeline it belonged to.

The extractor runs a headless browser render and a plain fetch, and merges both. That isn’t redundancy. Some hosts, like GoDaddy, serve pages a renderer sees as empty, while other builders serve stub HTML specifically to automated browsers. Whichever one a site defeats, the other still returns something. It crawls up to 40 pages, down from an earlier 200 so heavy sites still fit the time budget, and runs the free filters first: skip URLs already in the library, and drop obvious stock photography by filename.

Every one of those decisions was forced by a real site. A plain HTML scan, the way logo extraction already worked, missed a restaurant’s hero photo because it was a CSS background, and found three of a grooming site’s eight or nine images because the rest lazy-load on scroll. Rendering only when the scan came up short didn’t work either: you can’t tell a site with few images from one whose images haven’t loaded yet.

The first render kept anything served from the site’s image CDN, and that CDN serves everything. Filtering on what each file actually is cut the result to five, because the render jumped to the bottom in one motion and every middle section’s lazy loads were skipped. Scrolling one viewport at a time, forty steps with a 600ms pause, brought it to sixteen.

The rest came from sites picked deliberately to break it. A Next.js portfolio’s image optimiser buried real URLs in a query string, and unwrapping them both rescued the images and collapsed size variants of one photo. Tracking pixels and site-builder chrome are dropped by domain. Cart, checkout and account pages are skipped outright, and every page is isolated, so one failure is logged rather than losing the whole run.

GPT-4o-mini gives each image a keep-or-drop verdict at temperature 0 and low detail, with structured outputs. There are two ways to be dropped: junk, meaning spacers, icons, logos and decorative chrome, and not theirs, meaning a third party’s imagery embedded in the customer’s site.

The relevance check uses the business’s context, and when that context is missing it falls back to junk-only rather than guessing at relevance with no basis.

A perceptual hash catches the same photo at a different size. It structurally can’t catch the same scene at a different crop, because cropping changes the low-frequency structure the hash is built from. So dedup is a two-pass cascade: a cheap, strict pHash first, then DINOv2-small embeddings on whatever survives, clustered with union-find.

Pixels go first, not a model, because comparing every image against every other with a model grows quadratically in cost, while hashing has no per-call price at all. Hashes are compared by Hamming distance. In the earlier separate-scripts version, a single image was effectively fetched four times, once per stage, which was slow, hammered the customer’s own site, and tripped hotlink protection.

Two details in the hash matter. The image is squashed to a square with a fill resize, so the whole image contributes; centre-cropping would throw away the edges that tell photos apart and merge unrelated shots with a similar middle. And the DC term of the DCT is dropped, with the other 63 coefficients hashed against their own median, because the DC term would otherwise be a bit that never changes.

Merging on embeddings needs two conditions: the closest variant pair must reach 0.92 cosine similarity, and the whole images must reach 0.82.

Memory needed care too. Keeper bytes are held across stages so describing and uploading never re-fetch someone else’s URL, while each collapsed duplicate’s buffer, up to 25 MB, is freed straight away. Fetching goes through the shared hardened fetch rather than my earlier hostname check, because it’s the DNS layer that actually blocks SSRF tricks like a hostname that quietly resolves to localhost. A last check removes images the business already has stored.

A vision model describes images six at a time, again at temperature 0, given both the business’s context and the site author’s alt text, so descriptions are grounded rather than generic. They lead with the subject, like “A Saint Bernard with a black-and-white coat…”, instead of narrating “the image,” because the text is context for another AI, not a caption for a person. The same call assigns a category and tags. Bytes are sent inline so the model doesn’t re-download an image we already hold, and formats it can’t read are converted first rather than failing silently.

Storing strips EXIF metadata, re-encodes to WebP at up to 2048 pixels and quality 82, uploads to S3 and upserts the image row. Each image is keyed by its content fingerprint, so a re-run overwrites the same object instead of piling up copies, and storage errors are reduced to their message before logging, because the SDK attaches credentials to its own error objects.

Trigger. The job runs as a DAG edge after onboarding ingestion, but only when storage is configured. A job that runs as a no-op still counts as a run, and would permanently poison the dedup baseline for that business.

Publishing is deliberately not a DAG edge. The job queues a republish only if images were actually stored and the media page is already live, and it records that media changed so a retry landing between storing and queueing doesn’t lose the publish. A recrawl that finds nothing new must not fan out republishes across every locale.

Fingerprints stored from day one. Every row keeps its exact-file hash, perceptual hash and embedding, with first-seen and last-seen times, before the recurring recrawl existed. The job already computes all three, so storing them was nearly free, and it turned the future cron into a database comparison instead of a backfill that re-downloads every image ever stored. Pixel dimensions are kept for the same reason, so the gallery can size every tile without downloading a hundred full-size images to measure them.

Recrawls. A weekly cron picks up to 50 businesses per run that have just crossed a 90-day mark, skipping any with a run in flight, so the work rolls smoothly across weeks. Businesses from before the feature were backfilled by a script that dry-runs by default, refuses production without explicit confirmation, skips anything already crawled, and ran in small daily batches, partly for cost and partly so their 90-day dates don’t all fall due at once.

Images from anywhere else, like an emailed photo or a manual upload, come in through the same single entry point and the same sanitisation. Owners get the media library, a publishable media page, a recrawl button, and edit fields with limits.

An eval is a test suite for a nondeterministic system: you can’t assert exact output, so you assert properties of it. Eve’s suite has three layers, from strictest to fuzziest.

The first layer checks the agent’s event stream, every tool call in order. Tool ordering, like reading a page before editing it, becomes a hard assertion rather than a rubric’s opinion, and any run that returns the canned no-answer fallback fails, because saying nothing otherwise passes every rubric by saying nothing wrong. The second layer is LLM-as-judge with closedQA rubrics. The third is factuality grading against the business’s own stored pages.

There are nine buckets of cases, run against two real test businesses, with the agent given exactly the nine tools production has. I built on the eve framework’s own runner, which already ships evals, tool-call assertions and a judge, reading an old unmerged prototype for patterns and then starting fresh.

The setup was most of the actual time. Getting one case running locally took far longer than writing all of them, because of three configuration problems hidden behind the same vague error: an environment variable set to an empty string that the runtime read as “running on Vercel”; a test-only auth hook that disabled itself because pulling environment variables makes a laptop look like Vercel; and the app reading one database while the test businesses lived in another. None were logic bugs. A framework hiding its underlying error turned a twenty-minute problem into a multi-day one.

Answering questions from the open web instead of the knowledge base is a straight hallucination risk on a customer-facing product. Once the default tools were turned off, the failing cases went green.

It’s worth walking, because the two runs are indistinguishable from the reply alone. Same question, same business, same prompt:

Almost all of that is input. The system prompt and tool schemas are re-sent every turn, and output is tiny, so cost scales with total turns rather than with the number of cases. That token math is what settled how the suite should run.

A stranger texts the product’s number. They have no account and no business record, and every tenant-scoped tool assumes both. The goal is to give them a free analysis of how visible their site is to AI assistants, then sign them up, all over SMS.

The analysis runs on Cloro, a paid third party that asks AI assistants questions about the business. So the first tool deliberately spends nothing, and its order is the cost control:

A second tool is where money actually moves: vet the questions, claim the slots, submit the run.

The cost control the whole loop rests on. Without it, one unauthenticated message could trigger thirteen questions across five assistants: dozens of paid tasks. The guard is a single condition in the write’s WHERE clause: slots already used, plus the slots being requested, can’t exceed the allowance. However many requests race, that condition can never push the total past the allowance. It’s enforced in code as policy, never requested politely in a prompt.

Claiming first has a cost: if something fails before submission, slots leak. A failure now carries an explicit safe-to-release flag, so slots can never be released once money might have moved. A failed run also keeps its ID instead of clearing it, because a cleared ID makes “spent but failed” look identical to “never ran,” and a retry would pay twice.

Closing an idempotency hole. Cloro recognises repeat requests by an idempotency key built partly from a prompt ID, and the normal customer path mints a new prompt ID every time, so a replay is billed as new work. For prospects I derive that ID from a SHA-256 hash of the run and the question text, so the same question always produces the same key. It’s hashed rather than used raw because the question comes from a stranger, and raw text could smuggle in the key’s delimiter. Unique indexes back it up in the database.

The webhook that reports finished runs learned that a prospect’s run is a separate reporting path from a customer’s, with two reapers: one for stale runs, and one for results that finished but were never texted back. Whether to invite someone to sign up at all is a pure function with a fixed set of allowed moments, a six-hour cooldown, and a permanently honoured “no.” It’s code rather than prompt text because prompt-only rules leaked: the model writes the words, it doesn’t set the sales cadence.

A tool can’t wait between “here’s my email” and “here’s the code.” It runs and exits. So all signup state lives on the phone’s row, and deliberately not in the field that means “this number has an account,” because during signup a user already exists in Clerk before their email is proven.

The first text makes a conditional claim on that row, and one condition is subtle: the row must not already point at a signup’s user, because that pointer is the only way back to the Clerk user holding the email. Losing it would lock the address forever. The user is recorded before the code is sent, so a failed send retries by resending the code rather than creating a second user.

That last distinction is the whole point. Clerk’s verified flag describes the email address, not anything this phone number did, so the flow keeps its own record of when the code was actually entered. That’s the difference between a retry-safe flow and an auth bypass.

On the second text the code is checked, and if our record already shows it verified, Clerk is skipped entirely: a verification attempt is single-use, so a retry would fail on a code that was already correct.

Provisioning creates or reuses the organisation, checks whether the site already has an owner while excluding the caller’s own organisation, then upserts the user, business and membership. That exclusion is the retry fix: without it, a second attempt finds the organisation it just created and decides the site is taken.

Finishing signup is a single transaction that binds the phone row only if it still belongs to this exact signup, then writes the account and clears every signup field. Only a completed free analysis carries over to the new business. One still running stays the prospect’s, so its results are still texted back instead of vanishing into a business record they haven’t seen.

The texter is waiting on “your account is ready,” so account writes happen inside the request, and the knowledge base build runs afterwards with waitUntil. A build failure doesn’t roll back the account; it texts them instead. That build runs through an onboarding function I split out of the browser flow, since SMS has no browser session to read, and the browser version is now a thin wrapper around it, which makes it load-bearing for both.

A reaper deletes abandoned signups’ reserved Clerk users, since otherwise a typed-and-abandoned email stays reserved forever and that person can never sign up. For connecting the site, a live DNS lookup is trusted over a stored flag, setup instructions can be emailed to whoever manages the website, and a stalled setup gets exactly one follow-up text to the owner and one email to their web person after three days.

Paying through settings and Stripe still worked in the browser, but a texter never sees one. The agent now has exactly one way to send a billing link, and it only works in a text session, for a caller with write permission, on the owner’s own reply turn so a link never goes out on a proactive message, and for an account that isn’t comped.

The link is a stateless token signed with HMAC, expiring after 24 hours, with no database row behind it. Opening it checks the shape, then the signature, then the expiry, in that order: an expired but genuine link is a different fact from a forged one, and nothing inside the payload is trusted before the signature checks out. Stripe Checkout is created when the link is tapped, not when it’s sent, so an owner who subscribed in the meantime lands in the billing portal instead of a second checkout.

Interactions are where bugs live. The advisory lock bug came from two individually correct changes I’d reasoned about in separate sittings.

Correct code isn’t a correct system. The Bing tag was provably stored and still not served, and the ONNX engine worked everywhere except the deployed bundle.

Plan before building. The image pipeline was the first thing I designed up front, with a doc and throwaway scripts against real sites, after learning what deleting code I didn’t understand costs.

Put limits where money moves. Claiming before spending bounds the spend. Claiming afterwards only bounds a counter.

Hidden errors are the expensive ones. Every eval setup problem was configuration, and a framework swallowing its real error is what made them take days.