Work

Evaluating Innovation with AI: A Behavioral Analysis of Decision-Making in Startup Assessments

Year: 2025
Type: dissertation
Abstract: This dissertation investigates how artificial intelligence (AI) influences human decisionmaking behavior during startup idea evaluations, with a focus on developing a methodology to detect one of the risks of this collaboration: the potential for overreliance on AI.The study is based on a dataset provided through a research affiliation with the Laboratory for Innovation Science at Harvard (LISH), in which students from the University of Washington evaluated startup applications submitted to the MIT Solve Global Health Equity Challenge, where three conditions were randomly assigned: No AI, Black Box AI (binary recommendation), and Narrative AI (recommendation with explanation).Two types of data were extracted from screen recordings of the evaluation sessions: session level behavioral data (clicks, scrolls, tab switches, hovers) and decision data (final answer, confidence).These inputs were used not only to examine behavioral shifts across AI conditions, but also to support the creation of an innovative framework to detect overreliance.As startup evaluation is a task with subjective correctness, traditional operationalizations of overreliance based on disagreement with ground truth are inadequate.This work instead proposes a novel behavioral operationalization in which overreliance is identified when evaluators follow AI recommendations while displaying low cognitive effort.The framework computes a score for each AI-assisted evaluation based on its projection along a fixed direction manually defined to represent low effort behavior, using predefined weights for each interaction feature.The development process is thoroughly documented across five refinement iterations.Results of this investigation indicate that AI assistance was associated with evaluations that were shorter in duration, with reduced maximum scroll depth, scroll frequency, and number of tabs opened.This signals lower cognitive effort and engagement.Additionally, providing explanations did not prove to significantly alter evaluators' reliance on AI compared to simply presenting the recommendation.The final version of the framework demonstrates robustness and good alignment with human judgments of evaluator overreliance.Ultimately, this work contributes a generalizable approach to understanding responsible human AI collaboration in contexts of high subjectivity.
Author: Ana Sofia Guedes Vieira Santos Costa
Language: English
Open Access status: green
OpenAlex ID: W7112089980
Cites: 45
Cited by:
Related to: 10