Benchmarking Embedding Models for POJK Regulatory Retrieval in Indonesian Insurance Documents
Year: 2026
Type: article
Abstract: Indonesian insurance regulations are clause-dense and terminologically specific, making manual retrieval of applicable provisions difficult. This study presents a retrieval-only benchmark comparing embedding and lexical methods on POJK 26/2025 and POJK 36/2025. The corpus contains 181 context chunks and 181 synthetic Indonesian queries, each assigned one primary relevant chunk. Queries were generated in JSON mode using Gemma 4 E4B via Ollama (ollama:gemma4:e4b), with malformed outputs retried and heuristic fallbacks prohibited. Evaluation comprised a four-model local dense comparison and a separate native BGE-M3/BM25 ablation using exact exhaustive ranking. The local comparison reports MRR and Hit rates, while the native ablation additionally reports NDCG@10, MAP@10, bootstrap confidence intervals, paired tests, and rank bins. Qwen3 Embedding achieved the highest local MRR (0.8075). Under a prespecified descriptive composite rule combining MRR, Hit rates, and measured latency, BGE-M3 ranked first. In the native ablation, BGE-M3 combined retrieval achieved NDCG@10 of 0.8810. A practitioner-validated subset audit and artifact-backed qualitative analysis supplemented the quantitative results. Findings are limited to this two-document benchmark and do not evaluate answer generation, citation correctness, or end-to-end RAG faithfulness.
Institution: Mercu Buana University
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $32
OpenAlex ID: W7223350002