Year: 2024
Type: conference-paper
Abstract: Verification and validation of AI systems, particularly learning-enabled systems, is hard because often they lack formal specifications and rely instead on incomplete data and human subjective feedback. Aligning the behavior of such systems with the intended objectives and values of human designers and stakeholders is very challenging, and deploying AI systems that are misaligned can be risky. We propose to use both existing and new forms of explanations to improve the verification and validation of AI systems. Toward that goal, we preseant a framework, the agent explains its behavior and a critic signals whether the explanation passes a test. In cases where the explanation fails, the agent offers alternative explanations to gather feedback, which is then used to improve the system's alignment. We discuss examples of this approach that proved to be effective, and how to extend the scope of explanations and minimize human effort involved in this process.
Institutions: University of Massachusetts Amherst, Oregon State University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
Funder: National Science Foundation
Awards: FW-HTF-RL: Reimagining Trucking: Forging an Equitable and Driver-Centered System in a Highly Automated World, RI: Small: Foundations and Applications of Observer-Aware Planning, Collaborative Research: SLES: No Bad Surprises: Aligning Agent and Human Norms via Specification Refinements, CISE: Large: Causal Foundations for Decision Making and Learning
OpenAlex ID: W4404187949