Work

Human‐Agent Teaming on Intelligence Tasks ( HATIT ): A Testbed for Evaluating AI in Intelligence Analysis

Year: 2025

Type: article

Abstract: ABSTRACT Artificial intelligence (AI) has been proposed to overcome distributed team cognition and information challenges (e.g., volume, velocity) in intelligence analysis. However, before deploying AI in the workplace, designers should evaluate the effects of proposed AI in realistic simulated environments, also known as testbeds. We conducted interviews with intelligence professionals which, combined with our research needs, resulted in requirements, designs, and the creation of the Human‐Agent Teaming on Intelligence Tasks (HATIT) testbed. HATIT includes a web‐based software platform, a shift handover intelligence task in a fictional world with 427 pages and 60 documents, and an initial, static AI agent called “Illuminate” that summarizes documents and provides social media topic models. HATIT enables controlled experiments in a realistic simulation. This testbed lets us evaluate the effects of different AI on perceptions (e.g., trust, workload), problem solving and team cognition, and information search before deploying in actual work settings with real consequences.

Source: Proceedings of the Association for Information Science and Technology

DOI: 10.1002/pra2.1272

Authors: Susannah B. F. Paletz, Aimée A. Kane, Madeline Diep, Tammie M. Nelson, Adam Porter, Sarah H. Vahlkamp

Institutions: University of Maryland, College Park, Duquesne University, Fraunhofer USA Center Mid-Atlantic CMA

Language: English

Topic: Human-Automation Interaction and Safety

Subfield: Social Psychology

Field: Psychology

Domain: Social Sciences

Open Access status: bronze

Funders: Duquesne University, Army Research Office

Award: G7452299184