Doctoral Thesis Proposal - Wayne Chi
November 11, 2026 3:00PM—4:30PM
Location:
6115
-
Gates and Hillman Centers
Speaker:
WAYNE CHI,
Ph.D. Student, Computer Science Department, Carnegie Mellon University
https://waynechi.com/
Evaluation has long been a cornerstone of progress in artificial intelligence (AI), but as large language models (LLM) and agentic capabilities continue to improve, designing evaluations that are both challenging and realistic grows increasingly difficult. At the same time, the fundamentals of evaluation are changing: human performance no longer universally upper bounds model performance, and verification has arguably become more difficult than generation itself. Consequently, human verification and judgment have become more important than ever. Given these shifts, we argue that designing human-centric evaluations—evaluations grounded in how humans actually work with and judge AI systems—is critical to advancing AI.
In this proposal, we explore how to design human-centric evaluations by (1) grounding human preferences in real user contexts and (2) mirroring human verification processes in automated evaluations. First, we collect human preferences directly within developers' live IDE workflows through Copilot Arena and transform developers' instructions and contexts into a test-driven benchmark, EDIT-Bench. Second, we investigate how to mirror human verification in automated evaluations through GameDevBench and Interaction-as-a-Judge, showing that more human-like evaluation mechanisms—such as directly interacting with built applications through computer use rather than evaluating code alone—improve alignment with human preferences and downstream model performance. Finally, we bring these two ideas together by using Interaction-as-a-Judge to unlock in-the-wild player ratings from online games and train a code-free reward model to evaluate and improve open-ended game development.
Thesis Committee
Chris Donahue (Co-Chair)
Ameet Talwalkar (Co-Chair)
Graham Neubig
-Cho-Jui Hsieh (University of California Los Angeles)
In-person & Zoom
Contact
Matt Stewart