Year: 2025
Type: preprint
Abstract: The rapid development and widespread adoption of Audio Large Language Models (ALLMs) demand rigorous evaluation of their trustworthiness. However, existing evaluation frameworks are primarily designed for text and fail to capture vulnerabilities introduced by the acoustic properties of audio. We find that significant trustworthiness risks in ALLMs arise from non-semantic acoustic cues, such as timbre, accent, and background noise, which can be exploited to manipulate model behavior. To address this gap, we propose AudioTrust, the first large-scale and systematic framework for evaluating ALLM trustworthiness under audio-specific risks. AudioTrust covers six key dimensions: fairness, hallucination, safety, privacy, robustness, and authenticition. It includes 26 sub-tasks and a curated dataset of more than 4,420 audio samples collected from real-world scenarios, including daily conversations, emergency calls, and voice assistant interactions, and is specifically designed to probe trustworthiness across multiple dimensions. Our comprehensive evaluation spans 18 experimental settings and uses human-validated automated pipelines to enable objective and scalable assessment of model outputs. Experimental results on 14 state-of-the-art open-source and closed-source ALLMs reveal important limitations and failure boundaries under diverse high-risk audio scenarios, providing critical insights for the secure and trustworthy deployment of future audio models. Our platform and benchmark are publicly available at https://github.com/JusperLee/AudioTrust.
Source: arXiv (Cornell University)
Authors: Li, Kai, Shen, Can, Liu, Yile, Han, Jirui, Zheng, Kelong, Zou, Xuechao, Wang, Lionel Z., Zhang, Shun, Du, Xingjian, Luo, Hanjun, Jin, Yingbin, Xing, Xinxin, Ma, Ziyang, Liu, Yue, Zhang, Yifan, Fang, Junfeng, Wang, Kun, Yan, Yibo, Deng, Gelei, Li, Haoyang, Li, Yiming, Zhuang, Xiaobin, Chen, Tianlong, Wen, Qingsong, Zhang, Tianwei, Liu, Yang, Hu, Haibo, Wu, Zhizheng, Hu, Xiaolin, Chng, Eng-Siong, Xu, Wenyuan, Wang, XiaoFeng, Dong, Wei, Li, Xinfeng
Institutions: King Center, Tsinghua University, Waseda University, National University of Singapore, Nanyang Technological University, Beijing Jiaotong University, Shanghai Jiao Tong University, University of Rochester, Zhejiang University, Hong Kong University of Science and Technology, ByteDance (China), University of Hong Kong, Hong Kong Polytechnic University, Chinese University of Hong Kong, Shenzhen
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W6966855172
Cites:
Cited by:
Related to: