Arete-27B is a useful case study in changing an AI model's behaviour after its original training. Nous Research's downloadable Qwen3.8-27B derivative targets a particular political framing, which the developer describes as reinforcing Western values. Its published results justify examining the intervention. They do not establish that the model is neutral across subjects, languages or users. Arete model card
For a team buying AI for research or customer-facing work, the decision is more specific than choosing a model with the lowest bias score. Which behaviours matter in the intended task? Who defines a correct answer? Does a change improve factual accuracy while preserving useful disagreement and legitimate uncertainty? Those questions make the release relevant beyond the politics of its training objective.
- 01Keep the objective visibleA targeted post-training intervention can be evaluated without treating its preferred worldview as a universal standard.
- 02Separate dimensionsKeep factual accuracy, framing, refusal behaviour and agreement with the user as separate questions.
- 03Inspect the rubricRead the evaluation prompts and grading rules before interpreting a headline score.
- 04Match test conditionsCompare the original and adapted models under matched conditions, including language and reasoning settings.
- 05Add independent examplesUse independent task examples and human review before adopting the model for consequential work.
01 — Release evidenceWhat was released and what changed
The public repository's upload history places the weight files on November 6. Nous's announcement states November 2026 without a day, so the precise availability date comes from the repository rather than an inferred blog timestamp. Repository history
The model card identifies supervised fine-tuning, a 27-billion-parameter starting model, text-only post-training data and an Inkling teacher. It reports no reinforcement learning or architectural change. Under Nous's evaluation, the thinking-off comparison raises the factual score from 2.32 to 2.98 out of five and reduces answers classified as containing propaganda from 22% to 0.9%. Model and results
Those are developer-reported measurements under a developer-defined rubric. The cited pair uses thinking off and temperature zero; results with other settings should not be pooled into this comparison. The score names should travel with the results. Calling the second figure an error rate, or subtracting it from 100 to produce an accuracy claim, would change what was measured. Figures checked November 10.
The business relevance is the possibility of deliberately changing a model's response tendencies. Whether the particular change is desirable depends on the task and on an evaluation that can distinguish improved evidence handling from a different preferred answer.
02 — DefinitionsSeparate a wrong fact from a disputed frame
Factual accuracy and framing can interact, but they are not identical. An answer can state a verifiable event correctly while describing it through a loaded label. Another can use restrained language while omitting the fact the user actually requested. A single score may conceal that distinction.
For a procurement test, define the dimensions before comparing models. Factual checks should identify the relevant source and the date or context in which it applies. Framing checks should explain the language or treatment being assessed. Refusal checks should distinguish a legitimate boundary from avoidance of an otherwise answerable request.
| Dimension | Question for a reviewer | Why it needs its own record |
|---|---|---|
| Factual support | Does the answer accurately represent the evidence? | Pleasant wording can still contain false claims |
| Completeness | Does it address the important part of the request? | An omission can mislead without an explicit falsehood |
| Framing | Are labels and assumptions justified? | A preferred vocabulary is not automatically objective |
| Uncertainty | Does confidence match the available evidence? | A fluent answer can conceal unresolved questions |
| Refusal | Is declining the request appropriate here? | More answers are not always better answers |
03 — Prompt designInspect how the benchmark asks its questions
Nous publishes an 803-prompt evaluation organised around seven value categories. The dataset includes source references, required facts and markers used to assess framing; its prompts are in English. Nous describes using Grok 4.7 as a model judge, with the answer's model identity withheld and a rubric refined against a small manually graded pilot. Published evaluation
That openness makes a more useful assessment possible. Read a sample of the prompts before relying on the aggregate. Look for what the question presupposes, what evidence a successful answer must include, and whether a careful qualification would be rewarded or penalised. A benchmark can be well documented and still cover only a narrow slice of the behaviour a buyer needs.
One scoring detail changes how the headline should be read: an unanswered prompt receives both a fact score of zero and a propaganda score of zero, with all 803 prompts retained in the denominator. A model that says nothing can therefore look good on the framing measure while failing to inform the user. Inspect completion and factual performance alongside the propaganda rate. Scoring rules
In your own test, include neutral formulations as well as challenging ones. A question that strongly suggests the expected conclusion may measure compliance with the premise as much as independent reasoning. Pair it with a version that asks for the evidence without supplying the interpretation. Keep the relevant facts constant so the comparison has a clear purpose.
The public benchmark is a starting point for inspection and reproduction, not a replacement for that additional set. If the model was developed with a particular evaluation in view, success there should lead to a question about generalisation. Does the changed behaviour survive a different topic, wording, source or language?
Avoid reducing that question to whether the model agrees with the reviewer. Agreement is easy to observe; well-supported reasoning requires more work to evaluate.
04 — Judge reviewMake the judge answerable to people
Using a model to grade another model can make a large evaluation more manageable, but it also introduces another decision-maker. The grading instructions, evidence supplied and treatment of ambiguity all influence the result.
For an application-specific evaluation, ask the judge to identify the exact claim and source that support its decision. Keep the original answer alongside the grade. A result such as “poor neutrality” is difficult to audit; a record that identifies an unsupported label, a missing fact or a contradiction can be reviewed and corrected.
Use human review on a deliberately varied sample, including disagreements between the models and borderline cases. If people cannot reliably apply the rubric, refining the wording may be more valuable than increasing the number of automated judgements. A larger run does not remove ambiguity in the rule being applied.
Also examine errors by the judge. It might reward an answer that repeats expected terms while misunderstanding their meaning, or penalise an answer that expresses the right fact differently. Track those cases separately from errors by the model under evaluation. Otherwise, improvements to the grading process can be mistaken for improvements to the model.
The useful output is a reviewable chain: question, evidence, answer, grade and reason. It gives a buyer something more concrete than a leaderboard position and supports meaningful discussion when reasonable reviewers disagree about framing.
05 — ComparabilityCompare like with like
A before-and-after comparison should hold the evaluation conditions steady wherever possible. Use the same questions, sources, system instructions and output limits. Record the reasoning setting and sampling configuration. If the serving environment requires different settings, disclose the difference and treat the result accordingly.
The original and adapted models should also receive the same opportunity to answer. Giving one a source document and asking the other to rely on its stored knowledge tests a different question. So does allowing one model a long explanation while truncating the other before it reaches its qualification.
Keep the evaluation set separate from the examples used to develop prompts or tune the model. Repeatedly revising instructions against the same few failures can produce a convincing demonstration without establishing broader improvement. Reserve examples that have not guided those changes, and retain the failures instead of replacing them with easier cases.
A useful comparison record looks like this:
| Item to fix or record | Purpose |
|---|---|
| Exact checkpoint and serving configuration | Makes the evaluated system identifiable |
| Prompt and supplied evidence | Establishes what the model actually knew |
| Reasoning and sampling settings | Helps explain variation between runs |
| Grading rules and reviewer procedure | Makes the meaning of success explicit |
| Held-out examples and failure archive | Supports checking whether gains generalise |
No model calls were run to populate this worksheet. It is a proposed method for a buyer's evaluation. Our fine-tuning use-case guide provides the broader adaptation context; the extra requirement here is to make the intended behavioural change measurable before pursuing it.
06 — AudienceTest the work your audience will actually see
A publisher operating in several markets needs more than a single English aggregate. It needs to know whether the assistant represents sources accurately in the languages and formats its readers use. Translation alone may change what a question asks or which terms carry political meaning.
Build a small, independently reviewed set for each priority language rather than assuming a translated benchmark is equivalent. Where a term is contested, ask reviewers to document the dispute and evaluate whether the answer describes it accurately. A model should not have to pretend that every controversy has one agreed label to pass a well-designed test.
For a hypothetical supplier-research application, one case could ask the assistant to summarise an adverse finding and the supplier's response. Another could ask for a comparison using incomplete evidence. A third could present a claim from a source that later corrected it. The acceptance criteria concern faithful representation, scope and uncertainty, not whether the answer sounds favourable to a particular country or organisation.
Include ordinary tasks as well. A targeted intervention is only useful for the application if the model still performs the everyday work it is being bought to do. Assess instruction following, source citation, formatting and task completion alongside the sensitive-topic cases, using the same definitions across candidates.
Our guide to human review and escalation is relevant when disagreement cannot be resolved automatically. The model can surface evidence and uncertainties while a qualified person makes the final publication or procurement decision.
07 — DeploymentCheck the deployment decision separately
Model behaviour is one part of procurement. The ability to download weights, the right to use them for a particular purpose, and the resources needed to operate them are separate questions. A favourable behavioural evaluation does not settle any of those by itself.
The Arete repository identifies an Apache 2.0 licence. Confirm the exact checkpoint, included files and applicable terms for the planned deployment rather than treating a family name as a complete licence record. The open-weight licence audit explains why model access and usage rights deserve separate attention.
Avoid estimating a production hosting budget from the parameter count alone. The precision of stored weights, serving software, concurrent requests, context and supporting infrastructure all affect the implementation. Obtain a measured serving configuration or an appropriate provider quote before attaching a cost-per-task claim to the release.
Keep the evaluation environment close enough to production to expose relevant behaviour. If the production assistant will retrieve documents, use the same document permissions and source-selection rules in its trial. If outputs will enter a review queue, test what the reviewer actually receives, including citations and uncertainty.
This can reveal a less dramatic but useful conclusion: the application may need better retrieval, clearer prompts or a stronger review step more urgently than it needs a different checkpoint. A model change should solve the observed problem rather than stand in for defining it.
08 — RequirementsChoose on a documented behavioural requirement
An actionable requirement is narrower than “less biased.” It might ask the assistant to distinguish a source's claim from an established fact, represent material counterevidence, avoid unsupported certainty, or answer a legitimate question without adopting a premise that the evidence does not support.
Write that requirement in language a reviewer can apply. Then identify examples where the current system fails it and evaluate whether the proposed change improves those cases without creating unacceptable regressions elsewhere. Include the cases where the adapted model loses; they explain the trade-off the business is actually making.
Do not collapse the result into a single endorsement if the evidence is mixed. A model could be a suitable candidate for one language and task while remaining insufficiently evaluated for another. A limited deployment with review may be justified even when unrestricted use is not. Conversely, a small benchmark gain may be irrelevant if it does not address a real failure in the application.
Arete makes the question of deliberate behavioural adaptation concrete. The useful buying habit is to require the objective, evaluation and trade-offs to remain visible. A model's preferences do not disappear because a score improves; the test should show whether its behaviour better serves the work and the people affected by it.
09 — DecisionTurn the claim into an evaluation you can defend
Define the behaviour you need to change
Inspect the published rubric, preserve the test conditions and add independent examples from the intended task. Keep factual support distinct from viewpoint agreement. That gives a team a defensible reason to trial an adapted model, or a clear explanation for keeping its current one.
Digital Applied's AI transformation service helps teams define the behaviours an AI application must demonstrate, connect those requirements to evaluation, and plan the review needed before adoption.