Work

Powerformer: Efficient and High-Accuracy Privacy-Preserving Language Model with Homomorphic Encryption

Year: 2025

Type: conference-paper

Abstract: We propose Powerformer, an efficient homomorphic encryption (HE)-based privacypreserving language model (PPLM) designed to reduce computational overhead while maintaining model performance.Powerformer incorporates three key techniques to optimize encrypted computations: 1) A novel distillation technique that replaces softmax and layer normalization with computationally efficient power and linear functions, ensuring no performance degradation while enabling seamless encrypted computation.2) A pseudo-sign composite approximation method that accurately approximates GELU and tanh functions with minimal computational overhead.3) A homomorphic matrix multiplication algorithm specifically optimized for Transformer models, enhancing efficiency in encrypted environments.By integrating these techniques, Powerformer based on the BERT-base model achieves a 45% reduction in computation time compared to the state-of-the-art HE-based PPLM without any loss in accuracy.

Source: Annual Meeting of the Association for Computational Linguistics (ACL)

DOI: 10.18653/v1/2025.acl-long.543

Authors: Dongjin Park, Eunsang Lee, Joon-Woo Lee

Institutions: Chung-Ang University, Sejong University

Language: English

Topic: Cryptography and Data Security

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funders: National Research Foundation, National Research Foundation of Korea, Ministry of Science and ICT, South Korea, Institute for Information and Communications Technology Promotion

OpenAlex ID: W4412945423