Evaluating an Anonymous Model: What You Can Actually Know

Evaluating an anonymous AI model is two different jobs that most teams collapse into one. The first job — measuring what the model does — works fine without a name: latency, tool-call reliability, refusal behavior, and output stability are all observable from your side of the API. The second job — establishing who stands behind the model — does not work at all, because there is no one to ask.

The distinction stopped being academic the moment stealth listings became a normal way to ship a frontier-class model. When ox-alpha first appeared on OpenRouter, teams started routing real work through an endpoint whose maker will not identify itself. Any theory about which lab built it is community fingerprinting, not a fact.

This guide is the due-diligence half of the eval problem. Our standard metric toolkit covers how to measure quality; this post covers what an identity buys you that no benchmark can — and how to run honest empirical tests on the parts that remain measurable. The spine is an explicit inventory of the unknowables: training-data provenance, retention, deprecation risk, license, and liability.

Every model evaluation you can run produces evidence about behavior: how the model answers your prompts, today, under your conditions. That evidence is genuinely useful and the second half of this post is about collecting it honestly. But behavior is also the one thing a motivated vendor can shape — anything that has ever appeared in a public benchmark could have been trained toward, which is precisely why an anonymous model arriving with strong self-reported numbers proves so little.

An identity produces a different kind of evidence entirely: commitments. A named vendor publishes a retention window, a training-use default, and a deprecation policy — and attaches a legal entity to each one. When the model misbehaves, that entity owns a changelog, a status page, and a support queue. When its output infringes someone's copyright, that entity is the counterparty your contract points at. None of this is observable in a response payload. It exists only in published terms, signed by someone.

That is the frame for everything below. The question is never "is the anonymous model good?" — you can test that. The question is "what am I giving up by not knowing who made it?" — and the honest answer is a specific, enumerable list, most of which cannot be recovered by any amount of clever testing.

The best thing about OpenRouter's Stealth Program is that its rules are written down. The Stealth Program EULA (last updated July 6, 2026) defines "Stealth Provider," "Stealth Model," and "User Content" as formal terms, and then says three things every team should read before sending a single production prompt to an anonymous endpoint.

The contrast with the rest of the platform is instructive. OpenRouter's own account settings expose prompt-logging, chat-logging, zero-data-retention, and model-training toggles at the organization level — these are user-controllable dimensions for named providers generally. A stealth listing's provider-side policy is not controllable the same way. And the platform's main Terms of Service (updated July 29, 2026) add two disclaimers that matter here: OpenRouter states it "strives to accurately represent the status of prompt logging and training for each Model on our Site" while disclaiming liability for errors in Model Terms — so even the listing page's description of a stealth model's data policy is not warranted — and it is "not responsible for any incorrect location reporting to the Models," which undercuts any residency assumption you might be tempted to make about an anonymous endpoint.

To see the shape of the hole an anonymous listing leaves, look at what three named API vendors publish — OpenAI's enterprise privacy commitments, Anthropic's API data-retention documentation, and Google's Gemini API data-logging policy. This is not a tutorial on their data practices — it's a contrast instrument. Each of the commitments below is a published document with a legal entity's name on it, which is exactly the artifact a stealth listing cannot produce.

The structural point is what these documents provide: a training-use default attached to a named legal entity — and, for OpenAI and Anthropic, a published retention window. Neither of those attributes exists for an anonymous provider: no named retention SLA, and no auditable training default beyond what the aggregator chooses to disclose on the listing's behalf.

The table below is this post's core deliverable — the inventory, not a checklist. Each row is one thing procurement or engineering eventually asks about a model. The third column is the honest kicker: for most rows, there is no test you can run.

None of the above means empirical evaluation is pointless — it means being precise about what it produces. Four dimensions are fully measurable client-side, against any endpoint, using nothing but your own request/response logs and repeated calls. No cooperation from the provider is required, and no identity is revealed by the results.

The tool-call case deserves the extra attention because it's concrete: the issue filed against a third-party agent framework documents stealth/ox-alpha returning silent empty, zero-token responses on certain tool-call payloads — a dated, reproducible divergence from a named model's expected tool-call contract, with no vendor changelog or postmortem possible because there is no named vendor to publish one. If your workload is agentic, build the harness before you route traffic; a full agent-eval pipeline is the systematic version of this.

And hold on to the boundary: none of these tests require — or produce — any information about training-data provenance, retention practice, or legal liability. They describe what the model does, not what was done to build it or who stands behind it. A team that runs all four beautifully has still learned nothing from the Identity Ledger's rows.

The standard defense against benchmark gaming is a private task suite — items the model's builders have never seen. Against an anonymous model this is not optional: since you cannot know the training cutoff, the lab, or the data pipeline, a public benchmark score is unfalsifiable as evidence of generalization. Three construction patterns from the current literature apply.

The held-out twin. Build a second benchmark matching an existing one's distribution, difficulty, and scoring rubric — but composed of items that did not exist when a candidate model's training data could have been frozen. Comparing scores on the public split versus the twin is the practical contamination-detection technique named across this literature.

Disposable synthetic suites. Work in the S3Eval strand generates large numbers of synthetic tasks specifically because they postdate any given model's training cutoff and are cheap to keep regenerating. The design goal is disposability — a suite you can throw away and remint — rather than a one-time held-out set that ages into the next training crawl.

Contamination-resistant design. A May 2026 arXiv position paper ("LLM Benchmark Datasets Should Be Contamination-Resistant", Al-Lawati, Lucas, Lee, Wang) argues public benchmark datasets should be engineered to be contamination-resistant — unlearnable while still supporting inference. Note the framing: this is a still-open research direction, not a shipped methodology you can adopt today.

Suite construction is half the job; run discipline is the other half, and it's where single-run scores quietly lie. Applied eval-engineering guidance from Braintrust (Jess Wang, March 6, 2026) recommends starting a private suite with as few as five representative examples chosen to hit different layout and behavior challenges rather than attempting comprehensive coverage up front — the cited case study then ran 70+ experiments across 11 prompt versions over two days.

The same case study measured concrete rerun variance: at temperature 0.7, three identical runs of one task showed a 10% swing in one scored dimension and 13% in another; at temperature 0.0, output was deterministic. The stated practical rule: run a candidate configuration three times before trusting a score, and take the worst repeated result as the real number — "If v10 scored 50% once and 40% twice, the real performance is 40%."

For an anonymous endpoint this discipline does double duty — repeated runs are also how you detect behavior shifts on a model that will never announce them. And because reruns cost real tokens, score against cost-per-successful-task rather than raw accuracy — the metric that survives a model being cheap, chatty, and wrong.

Model disappearance is routine platform behavior, not an edge case. OpenRouter's own tutorial content states that more than 70 models have been pulled or deprecated by providers "in the last few years" — an unscoped count with no stated population or fixed window, so resist turning it into a rate — and recommends fallback-chain engineering as the mitigation. That's for the platform at large, named models included. A stealth listing adds the EULA's explicit removal clause on top: "Stealth Models may be removed from our Stealth Program at any time upon request of the Stealth Provider or at OpenRouter's sole discretion, with or without notice to you."

Historically, stealth previews resolve in one of two ways: they graduate to a named, priced model, or they are withdrawn. The binary itself is the risk signal — "still listed today" carries no forward guarantee, and the record of how often stealth listings turn out to be who you'd guess is a separate question from whether yours will exist on Friday. Plan for the endpoint's disappearance the way you plan for a spot instance's: checkpoint your prompts, keep a fallback chain warm, and never let an anonymous listing become a single point of failure.

The forward projection follows from the incentives. Stealth listings are cheap distribution for labs that want frontier-scale feedback without frontier-scale accountability, and the EULA's train-by-default terms are what make free access economically rational — so expect more anonymous endpoints, not fewer. The teams that benefit will be the ones that treat them as what they are: a free trial with data-terms pricing, useful for scoped, non-sensitive workloads, and never a substitute for a counterparty. If you're formalizing that boundary for your own organization — which workloads may touch an anonymous endpoint, which require a named counterparty — our AI transformation engagements build exactly that kind of model-governance policy alongside the eval harness.