#ai-evaluation

Update GPT-6 Astra skills, AGENTS.md and task prompts with a practical audit, clear completion criteria and a rollout plan for multiple projects.

Test whether an AI agent uncovers missing client requirements before building. Compare interview quality, code inspection, evidence and acceptance criteria.

Assess AI research claims with a practical evidence matrix. Separate formal proofs, measured results and demos, and record what each check establishes.

Evaluate a vision model on real interface screenshots, including missing text and ambiguous controls. Score reading, target location and unsupported claims.

DeepSeek V4.1 Flash adds vision and lower prices. Compare 19 official benchmarks, agent frameworks and API costs; V4 Pro now continues past September 14.

Compare AI models using the review work needed for an accepted result. Track inspection, corrections and rechecks before treating a lower bill as savings.

Two AI reviewers can repeat one mistake. Design reviews around separate evidence checks, clear rubrics and independent calculations instead of votes.

OSWorld measures desktop GUI control, Terminal-Bench measures CLI work. Same-named benchmarks differ by version and harness, so check before you compare.

How to read AI model leaderboards without being fooled by benchmark contamination, eval gaming, and cherry-picked MMLU, GPQA, and SWE-bench scores.

Stage 5 of the agentic AI pipeline — prototype brief, eval harness, success criteria, and the templates that get you from vendor to live demo.

200 agentic AI terms — agents, MCP, memory, planning, evaluation, governance. Examples, source links, cross-references. The reference glossary.

Eighty AI eval metrics — quality (BLEU, ROUGE, BERTScore), agentic (success rate, tool accuracy), safety, fairness, RAG (faithfulness). Defined.

Defining ASR for production agents — completion vs partial vs hallucinated, cost-adjusted scoring, and benchmark suites. Formal methodology + reference dataset.

We measured low/medium/high reasoning effort across 5 frontier models on math, code, and analysis. Quality lift, latency tax, and cost-per-correct-answer data.

Cross-model hallucination rates on factual recall, citation accuracy, and code reference. 5,000 prompts tested across 5 frontier models with confidence bands.

Why $/token is the wrong unit and $/successful-task is the right one. Formulas, worked examples across 6 task families, and a downloadable scoring template.