Get started
Work

The Geometry of Chinese Syntax in Pre-trained Language Models: Structural Probes, Pre-training Objectives, and Control Tasks

Year: 2026
Type: preprint
Abstract: Do the internal representations of Chinese pre-trained language models (PLMs) encode syntactic structure? This paper systematically applies the structural probe method of Hewitt & Manning (2019) to four models of the same size — three Chinese monolingual models with different pre-training objectives (bert-base-chinese, RoBERTa-wwm-ext, MacBERT) and the multilingual mBERT — examining, layer by layer on the UD Chinese-GSDSimp treebank, the geometric decodability of dependency syntax, and applying control tasks in two directions (selectivity) as well as an MDL information-theoretic test. We find: (1) Chinese syntactic trees can be effectively reconstructed from squared distances after a linear transformation, and the dev UUAS peaks of the three monolingual models all fall in layer 8 (0.587, 0.587, 0.589), with test peaks of 0.596–0.605, far above the random-tree baseline of 0.096; the curves are hump-shaped, rising first and then falling. (2) The three masking strategies (original MLM, whole-word masking, error-correcting masking) converge to almost the same syntactic geometry: the peak layer is identical, the maximum gap across all layers is 2.6 percentage points, and at the peak layer it is only 0.2 percentage points; replacing the word representation from "first subtoken" with "mean within the word" shifts the absolute values up by about 3 percentage points overall and shifts both the depth and the distance selectivity up in step (layer 8 distance ΔUUAS 0.49 → 0.52), while the peak layer and the curve shape are unchanged — the conclusion does not depend on the Chinese-specific character↔word alignment. (3) Multilingual mBERT is not weaker than the monolingual Chinese models under the same pipeline (dev peak 0.604, test 0.619, both higher than the three monolingual models), but its UUAS peak comes earlier, at layer 7, and the distance selectivity and the minimum MDL code length (158.3 kbit) also fall in layer 7 (the three lines agree in direction, each gap being on the order of 1 percentage point); its depth selectivity (+0.095 at layer 8) falls within the range of the monolingual models, showing that this result is not an illusion caused by word-form memorization. (4) Control tests show that the depth probe has negative selectivity in the lower layers (layers 0–4; as low as −0.16) — there the probe relies mainly on word-form/word-type memorization rather than syntax; the point where selectivity turns positive coincides exactly with the syntactic peak at layer 8 (Δ = 0.07, 0.07, 0.11). Under the fixed random-tree control, the distance probe shows strongly positive selectivity in all layers (ΔUUAS 0.28 → 0.49), and the control probe's dev correlation is ≈0 while its train correlation is positive, showing that the random structure is merely "memorized" and does not enter the representation. (5) The MDL test independently reproduces the same conclusion: the online code length for the 45 dependency relation labels is lowest at layer 8 (157–161 kbit, compression ratio 3.37–3.45×, test accuracy 0.798–0.801). Three mutually independent lines of evidence (probe metrics, control tasks, information theory) point to the same layer. To our knowledge, this is the first structural probe study of a Chinese PLM family with a complete set of control tests. 中文摘要:中文预训练语言模型(PLM)的内部表示是否编码了句法结构?本文将 Hewitt & Manning (2019) 的结构探针方法系统应用于四个同规模模型——三个预训练策略不同的中文单语模型(bert-base-chinese、RoBERTa-wwm-ext、MacBERT)与多语 mBERT——在 UD Chinese-GSDSimp 树库上逐层检验依存句法的几何可解码性,并施加两个方向的控制任务(selectivity)与 MDL 信息论检验。我们发现:(1) 汉语句法树可被线性变换后的平方距离有效重建,三个单语模型的 dev UUAS 峰值同在第 8 层(0.587、0.587、0.589),test 峰值 0.596–0.605,远高于随机树基线 0.096;曲线呈先升后降的驼峰形。(2) 三种遮蔽策略(原版 MLM、全词遮蔽、纠错式遮蔽)收敛到几乎相同的句法几何:峰值层一致,全层最大差距 2.6 个百分点,峰值层仅 0.2 个百分点;把词表示从"词首子词"换成"词内均值"后绝对值整体上移约 3 个百分点、深度与距离两类 selectivity 同步上移(层 8 距离 ΔUUAS 0.49 → 0.52),而峰值层与曲线形状不变——结论不依赖中文特有的字↔词对齐方式。(3) 多语 mBERT 在同一条流水线下并不弱于单语中文模型(dev 峰值 0.604、test 0.619,均高于三个单语模型),但 UUAS 峰值提前到第 7 层,距离 selectivity 与 MDL 最低码长(158.3 kbit)也同落在层 7(三条线方向一致,各自差距均在 1 个百分点量级);它的深度 selectivity(第 8 层 +0.095)落在单语模型的区间内,说明这一结果不是词形记忆造成的假象。(4) 控制检验显示,深度探针在低层(0–4 层)的 selectivity 为负(低至 −0.16)——探针在那里主要靠词形/词型记忆而非句法;selectivity 的转正与峰值恰好与句法峰重合于第 8 层(Δ = 0.07、0.07、0.11)。距离探针在固定随机树控制下全层 selectivity 强正(ΔUUAS 0.28 → 0.49),控制探针的 dev 相关≈0 而 train 相关为正,说明随机结构只是被"背"下来、并未进入表示。(5) MDL 检验独立复现同一结论:45 类依存关系标签的在线编码长度在第 8 层最低(157–161 kbit,压缩率 3.37–3.45×,test 准确率 0.798–0.801)。三条互相独立的证据线(探针指标、控制任务、信息论)指向同一层。据我们所知,这是首次对中文 PLM 家族进行带完整控制检验的结构探针研究。 Code, all result files and the paper sources: https://github.com/Shizuku-keop/chinese-plm-syntax-probe
DOI: 10.5281/zenodo.23130688
Language: English
Open Access status: green
OpenAlex ID: W7219762619
Cites:
Cited by:
Related to: