RegLab collaborated with the Stanford Institute for Human-Centered AI (HAI) on a brief that presents a blueprint for how government agencies can effectively evaluate AI tools in high-stakes public services.
Key Takeaways
-
Common AI evaluation methods are insufficient to help governments evaluate AI tools in practice. Despite existing legal mandates, government agencies often struggle to implement effective AI evaluation processes.
-
Agencies already possess “hidden benchmarks” — administrative records of past task performance that can be used to train and evaluate AI models to meet public sector needs. If released, such benchmarks could both measure and drive AI progress.
-
Agencies should require third-party evaluations as a condition for procurement and conduct randomized AB tests to understand AI impacts as systems are deployed. Collaborating with universities and other research institutions can offer a cost-effective way to draw on independent outside expertise.
-
Effective post-monitoring deployment will require rethinking traditional quality assurance approaches to include human-AI collaboration on oversight.
Read the full brief here.