/ topics / evals
evals
10 stories we've shipped under this tag. Each is the actual link from the original issue — no rehashing.
The Pentagon wants $30 million to build an AI-powered lie detector
Engineers should scrutinize validation, false-positive rates, and adversarial robustness before treating biometric signals as reliable evidence.
policy
Artificial Analysis Intelligence Index v4.2
The index may inform model selection and benchmarking, but the supplied excerpt lacks methodology or results to assess its engineering value.
research
AI Engineer Notebooks – free, framework-free RAG/agents/evals on Colab
Useful as a low-friction reference for testing RAG and agent patterns without committing to a framework or local setup.
tools
How Claude Watermarks AI-Generated Text
Useful for evaluating provenance claims: it connects sampling mechanics to detector reliability and shows how watermark removal affects deployment assumptions.
safety
Asana cleared 5 years of engineering work in 2 weeks with Codex
The case study tests claims about AI coding productivity against a concrete migration: five years of estimated work, two weeks, and $12K.
industry
TutorMoments: Do AI tutors know when to help and when to hold back?
Useful for teams building educational agents: tutoring quality depends on intervention timing, not just answer accuracy, and requires evaluations for over-helping.
research
Microsoft Open Sources code-testing-generator: a Polyglot Unit-Test Agent That Hits 92.1% Task Completion Versus 78.9% for Stock Copilot
Its 92.1% completion rate versus 78.9% for stock Copilot suggests repository-aware planning and validation materially improve test-generation agents.
tools
AI financial advice is surprisingly good, especially if you ask right questions
Useful prompt-design signal, but the excerpt provides no study details, model comparisons, or evidence for assessing reliability in production.
research
The Eval Flywheel: Turning Every Production AI Failure Into a Regression Test
Moving beyond vibes-based development requires a systematic pipeline where edge cases found in production logs are automatically integrated into CI/CD eval suites.
tools
An AI coding agent, used to write code, needs to reduce your maintenance costs
Frames the right metric for AI coding ROI: maintenance burden, not velocity. Useful counterweight if you're justifying or scoping an AI coding tool adoption.
tools