Year: 2026
Type: preprint
Abstract: Semantic search with large language models (LLMs) enables retrieval by meaning rather than keyword overlap, but scaling it requires major inference efficiency advances. We present LinkedIn's LLM-based semantic search framework for AI Job Search and AI People Search, combining an LLM relevance judge, embedding-based retrieval, and a compact Small Language Model trained via multi-teacher distillation to jointly optimize relevance and engagement. A prefill-oriented inference architecture co-designed with model pruning, context compression, and text-embedding hybrid interactions boosts ranking throughput by over 75x under a fixed latency constraint while preserving near-teacher-level NDCG, enabling one of the first production LLM-based ranking systems with efficiency comparable to traditional approaches and delivering significant gains in quality and user engagement.
Source: arXiv (Cornell University)
Authors: Borisyuk, Fedor, Vasudevan, Sriram, Wu, Muchen, Li, Guoyao, Le, Benjamin, Zhang, Shaobo, Shen, Qianqi Kay, Juan, Yuchin, Behdin, Kayhan, Dong, Liming, Yang, Kaixu, Jing, Shusen, Pothamsetty, Ravi, Arora, Rajat, Sheng, Sophie Yanying, Abdrashitov, Vitaly, Zhao, Yang, Su, Lin, Wang, Xiaoqing, Zheng, Chujie, Metkar, Sarang, Gupta, Rupesh, Lapchuk, Igor, Racca, David N., Mohan, Madhumitha, Li, Yanbo, Li, Haojun, Gandhi, Saloni, Lu, Xueying, Bhole, Chetan, Hooshmand, Ali, Yang, Xin, Muthuregunathan, Raghavan, Zhang, Jiajun, Teoh, Mathew, Coler, Adam, Gupta, Abhinav, Ma, Xiaojing, Ramachandran, Sundara Raman, Ramezani, Morteza, Wang, Yubo, Zhang, Lijuan, Li, Richard, Sheng, Jian, Nguyen, Chanh, Chen, Yen-Chi, Zhu, Chuanrui, Zhang, Claire, Xu, Jiahao, Kulkarni, Deepti, Lan, Qing, Subramaniam, Arvind, Fatahibaarzi, Ata, Shimizu, Steven, Chen, Yanning, Wang, Zhipeng, He, Ran, Zhou, Zhengze, Song, Qingquan, Dai, Yun, Johnson, Caleb, Liu, Ping, Gharghabi, Shaghayegh, Mohanasundaram, Gokulraj, Bottaro, Juan, Sachindran, Santhosh, Guo, Qi, Ren, Yunxiang, Jiang, Chengming, Mo, Di, Simon, Luke, Shen, Jianqiang, Wu, Jingwei, Zhang, Wenjing
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7128513156
Cites:
Cited by:
Related to: