Engineering notes · 28 August 2026

Inside the market engine: a week of stress-testing

Agentery’s core promise is that when we say two AI products compete, they really do — the markets, price benchmarks and search results all hang off one semantic engine. This week we put that engine through a systematic evaluation: three embedding models, four text-composition templates, and two judge models, all measured against labelled ground truth. The outcome was deliberate inaction: we changed nothing, and now have the numbers proving that’s the right answer.

How the engine works: two vectors per listing

Every listing carries two short machine-written profiles, each embedded into its own 512-dimension vector.

The job vector answers “what job does a buyer hire this for?” — the primary job, a concise description, and the target user (only when it’s specific; generic values like “developers” are dropped). This vector alone drives search, market membership and duplicate detection, because search is fundamentally about intent.

The functional vector answers “what can it actually do?” — core actions, inputs, outputs, the systems it touches, and its access mode.

The two combine in exactly one place: the substitutability gate. Two listings count as commercial substitutes only when the minimum of their job similarity and functional similarity clears a calibrated threshold. In plain terms: they must solve the same problem and work in a comparable way. That two-key rule is what stops a browser-automation API and a no-code scraping app — same goal, different kind of product — from being wrongly merged into one market.

Three embedding models, one verdict

Against the incumbent (OpenAI text-embedding-3-small), we tested Voyage voyage-4-lite and Qwen3-Embedding-8B at identical dimensions on identical texts, with decision rules written down before any number was computed.

Voyage tied the incumbent on substitute discrimination (AUC 0.941 vs 0.937 — statistically indistinguishable on 357 labelled pairs), genuinely beat it on search retrieval (recall 0.846 vs 0.740 on 127 real buyer queries), but left 1.8× more listings with zero neighbours — which would thin out the long tail of market membership. Qwen3-8B, at half the price, was measurably worse at the judgement that matters most (AUC 0.905). Both verdicts: keep the incumbent. A cheaper or partially better model doesn’t justify changing a central engine.

Four templates, all null — and the real finding

An external reviewer observed that our two profiles are less independent than they look (both repeat the job and description) and that our text cleaner occasionally produced mangled sentences. So we expanded the labelled ground truth from 485 to 1,973 judged pairs and ran a four-arm test with the model held fixed: the production template as control, an “orthogonal” functional profile with no repeated content, a fixed cleaner alone, and both together.

Every variant came out statistically null against production. Even feeding the embedding model grammatically mangled text made no measurable difference. The redundancy turns out to act as a stabiliser for listings with sparse functional evidence.

The headline number from the whole programme: simple concatenated text scores AUC 0.937; our structured profile template scores 0.974. The template design is worth roughly ten times more than the choice of embedding vendor. The moat is the architecture, not the model.

Checking the referee

The ground-truth pairs are judged by a model from a different lab than the one that writes the profiles, so the ground truth stays independent of the thing being measured. We verified the judge itself: a second frontier model re-judged a 300-pair sample and agreed 94.5% of the time on the calls that matter (kappa 0.886). Every hard disagreement was a genuine boundary case — complement-versus-substitute calls like “A triages bugs; B fixes them”.

Also this week

Pricing coverage self-heal. A buyer pointed out that 11x had public pricing we showed as unavailable — their prices live on per-product pages our scanner couldn’t see. We measured the whole class: of ~3,600 own-domain listings marked “no public price”, about 90 had real pricing pages. After an automated verification stage rejected look-alikes (platform-generic pricing, competitor-comparison pages), 56 now show live prices. A weekly sweep keeps finding and pinning such pages.

Honest timing. The “Updated / Next update” line on the index now reads a stamp written the moment the day’s scan results actually load, with the next update expected 24 hours later — no more guessed countdowns.

More discovery sources. The listing pipeline gained a GitHub pass that catches established projects adding MCP/agent capability (first run surfaced n8n and gemini-cli), Smithery as an explicit newest-listings source, Hugging Face Spaces, and expanded npm signals — all feeding the same screening funnel. Products that buyers searched for and we then researched and listed are now tracked as their own demand-driven acquisition channel.

The Agentery Economic Price Index and market data live at agentery.com/aepi.