publications

Near-Exponential Savings for Mean Estimation with Active Learning

Accepted to the 39th Conference on Neural Information Processing Systems (NeurIPS 2025)
November 2025

Language models cannot reliably distinguish belief from knowledge and fact

Nature Machine Intelligence
November 2025

Not Your Typical Government Tipline: LLM-Assisted Routing of Environmental Protection Agency Citizen Tips

Association for Computational Linguistics, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
November 2025

The Inadequacy of Offline LLM Evaluations: A Need to Account for Personalization in Model Behavior

ArXiV
September 2025

Audits Under Resource, Data, and Access Constraints: Scaling Laws For Less Discriminatory Alternatives

ArXiV
September 2025
,

Fairness through Difference Awareness: Measuring Desired Group Discrimination in LLMs

ArXiV
August 2025

Extracting memorized pieces of (copyrighted) books from open-weight language models

ArXiV
July 2025

Do mandatory minimum penalties and penalty relief work? Evidence from California’s clean water program

PLOS Water
July 2025

Adverse Event Reporting for AI: Developing the Information Infrastructure Government Needs to Learn and Act Date

HAI Policy Brief
June 2025
,

Bias Delayed is Bias Denied? Assessing the Effect of Reporting Delays on Disparity Assessments

FAccT
June 2025
,

Cleaning Up Policy Sludge: An AI Statutory Research System

HAI Policy Brief
June 2025

The California Report on Frontier AI Policy

June 2025

What Is the Law? A System for Statutory Research (STARA) with Large Language Models

ICAIL
June 2025
,

AI for Statutory Simplification: A Comprehensive State Legal Corpus and Labor Benchmark

ICAIL
June 2025

BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems

ArXiV
May 2025

An automated framework for assessing how well LLMs cite relevant medical references

Nature Communications
April 2025

Enabling disaggregation of Asian American subgroups: a dataset of Wikidata names for disparity estimation

Nature Scientific Data
April 2025

Evaluation as Due Process: Civil Service in an Automated Age

Administrative Law Review
March 2025

Assessing the Implementation of Federal AI Leadership and Compliance Mandates

HAI White Paper
January 2025

Machine Unlearning Doesn’t Do What You Think: Lessons for Generative AI Policy and Research

December 2024

Belief in the Machine: Investigating Epistemological Blind Spots of Language Models

Arxiv
October 2024
,

AI for Scaling Legal Reform: Mapping and Redacting Racial Covenants in Santa Clara County

Journal of Legal Analysis
October 2024
,

Remote sensing and computer vision for marine aquaculture

Science Advances
October 2024

Considerations for governing open foundation models

Science
October 2024

A Path for Science‑ and Evidence‑based AI Policy

September 2024
,

Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risk of Language Models

Arxiv
August 2024

Statistical Uncertainty in Word Embeddings: GloVe-V

Empirical Methods in Natural Language Processing
June 2024

Drop a Line, Submit on Time? Randomized Tailored Reminders Improve Pollution Reporting Timeliness

Journal of the Association of Environmental and Resource Economists
June 2024
,

Not (Officially) in My Backyard: Characterizing Informal Accessory Dwelling Units and Informing Housing Policy With Remote Sensing

Journal of the American Planning Association
June 2024
,

Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools

ArXiv
May 2024
,

MultiLegalPile: A 689GB Multilingual Legal Corpus

ACL 2024
May 2024
,

Corpus Enigmas and Contradictory Linguistics: Tensions Between Empirical Semantic Meaning and Judicial Interpretation

Minnesota Journal of Law, Science & Technology
May 2024

Integrating Water Quality Data with a Bayesian Network Model to Improve Spatial and Temporal Phosphorus Attribution: Application to the Maumee River Basin

Journal of Environmental Management
May 2024

LawInstruct: A Resource for Studying Language Model Adaptation to the Legal Domain

Arxiv
April 2024

Quantifying the Uncertainty of Imputed Demographic Disparity Estimates: The Dual-Bootstrap

March 2024

Limitations of Reporting Requirements under California’s Livestock Antimicrobial Restriction Law

Environmental Health Perspectives
February 2024
,

On the Societal Impact of Open Foundation Models

February 2024
,

Mitigating allocative tradeoffs and harms in an environmental justice data tool

Nature Machine Intelligence
February 2024

Governing by Assignment

Penn Law Review
February 2024

Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models

Journal of Legal Analysis
January 2024

Potential for allocative harm in an environmental justice data tool

Nature Machine Intelligence
December 2023

Leveraging genomic sequencing data to evaluate disease surveillance strategies

iScience
December 2023

The AI Regulatory Alignment Problem

HAI Policy Brief
November 2023

Estimating and Implementing Conventional Fairness Metrics With Probabilistic Protected Features

Conference on Secure and Trustworthy Machine Learning
October 2023

Talent exchanges for state governments

SIEPR Policy Brief
September 2023

Toward Operationalizing a Pipeline-aware Approach to ML Fairness: A Research Agenda for Developing Practical Guidelines and Tools

ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization
September 2023

LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models

NeurIPS D&B
August 2023

The Bureaucratic Challenge to AI Governance: An Empirical Assessment of Implementation at U.S. Federal Agencies

AIES
August 2023

The Privacy-Bias Tradeoff: Data Minimization and Racial Disparity Assessments in U.S. Government

FAccT
June 2023

How Redundant are Redundant Encodings? Blindness in the Wild and Racial Disparity when Race is Unobserved

ACM FAccT
June 2023

Integrating social services with disease investigation: A randomized trial of COVID-19 high-touch contact tracing

PLOS ONE
May 2023

Estimating Racial Disparities When Race is Not Observed

Working Paper
March 2023

Automated vs. manual case investigation and contact tracing for pandemic surveillance: Evidence from a stepped wedge cluster randomized trial

Lancet: eClinicalMedicine
January 2023

Measuring and Mitigating Racial Disparities in Tax Audits

Working Paper
January 2023

Implementation Challenges to Three Pillars of America’s AI Strategy

HAI - RegLab White Paper
December 2022

Entropy Regularization for Population Estimation

ACM AAAI (2023)
August 2022

Detecting Environmental Violations with Satellite Imagery in Near Real Time: Land Application under the Clean Water Act

ACM CIKM
August 2022

Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance

ACM AI, Ethics & Society
August 2022

How to Build Academic-Public Health Partnerships: The Stanford – Santa Clara County Experience with COVID-19 Response

Chapter for Brookings' Build Me the Evidence, (Tamar Bauer ed)
July 2022

Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset

arXiv
July 2022

Algorithmic Fairness and Vertical Equity: Income Fairness with IRS Tax Audit Models

ACM FAccT 1479
June 2022

Integrating Reward Maximization and Population Estimation: Sequential Decision-Making for Internal Revenue Service Audit Selection

ACM AAAI (2023)
April 2022

Designing Accountable Health Care Algorithms: Lessons from Covid-19 Contact Tracing

NEJM Catalyst
March 2022

Science Translation During the COVID-19 Pandemic: An Academic-Public Health Partnership to Assess Capacity Limits in California

AJPH
February 2022

Artificial Intelligence for Adjudication: The Social Security Administration and AI Governance

Oxford Handbook on AI Governance
January 2022
,

Mapping industrial poultry operations at scale with deep learning and aerial imagery

arXiv
December 2021

Building a National AI Research Resource: A Blueprint for the National Research Cloud

HAI White Paper
October 2021

A Language-Matching Model to Improve Equity and Efficiency of COVID-19 Contact Tracing

118 PNAS e2109443118
September 2021

Evaluation of Allocation Schemes of COVID-19 Testing Resources in a Community-Based Door-to-Door Testing Program

JAMA Health Forum , 2, 8, e212260
August 2021
,

On the Opportunities and Risks of Foundation Models

arXiv
August 2021

Enhancing Environmental Enforcement with Near Real-Time Monitoring: Likelihood-Based Detection of Structural Expansion of Intensive Livestock Farms

International Journal of Applied Earth Observation and Geoinformation
July 2021

Context-Aware Legal Citation Recommendation Using Deep Learning

ICAIL 2021
April 2021

Executive Control of Agency Adjudication: Capacity, Selection and Precedential Rulemaking

Journal of Law, Economics, and Organization (2024, Forthcoming)
April 2021

When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset

ICAIL 2021
April 2021

How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals

Nature Medicine
April 2021
,

The Distributive Effects of Risk Prediction in Environmental Compliance: Algorithmic Design, Environmental Justice, and Public Policy

ACM FAccT 2021
March 2021

Leveraging Administrative Data for Bias Audits: Assessing Disparate Coverage with Mobility Data for COVID-19 Policy

ACM FAccT 2021
March 2021

How US Law Will Evaluate Artificial Intelligence for Covid-19

BMJ
March 2021
,

Temporal Cluster Matching for Change Detection of Structures from Satellite Imagery

ACM COMPASS
March 2021

Disparate Limbo: How Administrative Law Erased Antidiscrimination

Yale Law Journal
January 2021

Evaluating Facial Recognition Technology: A Protocol for Performance Assessment in New Domains

Stanford's Institute for Human-centered Artificial Intelligence White Paper
November 2020

Affirmative Algorithms: The Legal Grounds for Fairness as Awareness

U. Chicago Law Review Online
October 2020

AI’s Promise and Peril for the U.S. Government

HAI Policy Briefs
September 2020

Algorithmic Accountability in the Administrative State

Yale Journal on Regulation
August 2020
, ,

Feasible Policy Evaluation by Design: A Randomized Synthetic Stepped-Wedge Trial in King County

June 2020
,

Improving Scientific Judgments in Law and Government: A Field Experiment of Patent Peer Review

Journal of Empirical Legal Studies
June 2020
,

Menu Labeling, Calories, and Nutrient Density: Evidence from Chain Restaurants

PLOS ONE
May 2020

Artificially Intelligent Government: A Review and Agenda

Big Data Law (Roland Vogl, ed., 2021)
March 2020

What To Do About Artificially Intelligent Government

The Hill
February 2020

Government by Algorithm: Artificial Intelligence in Federal Administrative Agencies

Report to the Administrative Conference of the United States
February 2020

Innovations for Environmental Compliance: Emerging Evidence and Opportunities

SIEPR Policy Brief
February 2020

Due Process and Mass Adjudication: Crisis and Reform

Stanford Law Review
January 2020

Did Restaurant Hygiene Grading in Los Angeles Immediately Reduce Foodborne Illness by 20% Across All of Southern California? A Response to Jin & Leslie

SIEPR Working Paper
November 2019

New Evidence on Information Disclosure through Restaurant Hygiene Grading

American Economic Journal: Economic Policy
November 2019
,

Improving the Reliability of Food Safety Disclosure: Restaurant Grading in Seattle and King County, Washington

Journal of Environmental Health, 84, 2, 30-37
September 2019

Quality Review of Mass Adjudication: A Randomized Natural Experiment at the Board of Veterans Appeals, 2003-16

Journal of Law, Economics, & Organization
July 2019

Equity in the Bureaucracy

UC Irvine Law Review
June 2019
,

Is Yelp Actually Cleaning Up the Restaurant Industry? A Re-Analysis on the Relative Usefulness of Consumer Reviews

WWW Proceedings
May 2019

Deep Learning to Map Concentrated Animal Feeding Operations

Nature Sustainability
April 2019
, ,

Making Street-Level Bureaucracy Work: Safer Food in Seattle and King County

Evidence Works: Cases Where Evidence Meaningfully Informed Policy
January 2019

When Algorithms Import Private Bias into Public Enforcement: The Promise and Limitations of Statistical Debiasing Solutions

Journal of Institutional and Theoretical Economics
January 2019

Do Checklists Make a Difference? A Natural Experiment from Food Safety Enforcement

Journal of Empirical Legal Studies
June 2018

Judging Statistical Criticism

Observational Studies
January 2018

Managing Street-Level Arbitrariness: The Evidence Base for Public Sector Quality Improvement

Annual Review of Law and Social Science
October 2017

Does Peer Review Work? An Experiment of Experimentalism

Stanford Law Review
January 2017

Government Under Review

Boston Review
June 2016

Designing Information Disclosure

Administrative and Regulatory Law News
December 2012

Fudging the Nudge: Information Disclosure and Restaurant Grading

The Yale Law Journal
December 2012

Improve Restaurant Report Cards

New York Times
March 2012

featured projects

Tax Enforcement
See All projects