Log Probabilities Are a Reliable Estimate of Semantic Plausibility in Base and Instruction-Tuned Language Models
Year: 2024
Type: conference-paper
Abstract: Semantic plausibility (e.g.knowing that "the actor won the award" is more likely than "the actor won the battle") serves as an effective proxy for general world knowledge.Language models (LMs) capture vast amounts of world knowledge by learning distributional patterns in text, accessible via log probabilities (LOGPROBS) they assign to plausible vs. implausible outputs.The new generation of instruction-tuned LMs can now also provide explicit estimates of plausibility via PROMPT-ING.Here, we evaluate the effectiveness of LOGPROBS and basic PROMPTING to measure semantic plausibility, both in single-sentence minimal pairs (Experiment 1) and short contextdependent scenarios (Experiment 2).We find that (i) in both base and instruction-tuned LMs, LOGPROBS offers a more reliable measure of semantic plausibility than direct zero-shot PROMPTING, which yields inconsistent and often poor results; (ii) instruction-tuning generally does not alter the sensitivity of LOG-PROBS to semantic plausibility (although sometimes decreases it); (iii) across models, context mostly modulates LOGPROBS in expected ways, as measured by three novel metrics of context-sensitive plausibility and their match to explicit human plausibility judgments.We conclude that, even in the era of prompt-based evaluations, LOGPROBS constitute a useful metric of semantic plausibility, both in base and instruction-tuned LMs. 1 Dataset Plausible?Possible?Voice Example
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4404752302