Template-Type: ReDIF-Paper 1.0
Author-Name: Simon D Angus
Author-Email: simon.angus@monash.edu
Author-Workplace-Name: SoDa Laboratories & Dept. of Economics, Monash Business School
Author-Name: Lachlan O'Neill
Author-Email: lachlan.oneill@monash.edu
Author-Workplace-Name: SoDa Laboratories, Monash Business School
Title: Paired completion: quantifying issue-framing at scale with LLMs
Abstract: Detecting and quantifying issue framing in textual discourse - the slant or perspective one takes to a given topic (e.g. climate science vs. denialism, misogyny vs. gender equality) - is highly valuable to a range of end-users from social and political scientists to program evaluators and policy analysts. Being able to identify statistically significant shifts, reversals, or changes in issue framing in public discourse would enable the quantitative evaluation of interventions, actors and events that shape discourse. However, issue framing is notoriously challenging for automated natural language processing (NLP) methods since the words and phrases used by either 'side' of an issue are often held in common, with only subtle stylistic flourishes separating their use. Here we develop and rigorously evaluate new detection methods for issue framing and narrative analysis within large text datasets. By introducing a novel application of next-token log probabilities derived from generative large language models (LLMs) we show that issue framing can be reliably and efficiently detected in large corpora with only a few examples of either perspective on a given issue, a method we call 'paired completion'. Through 192 independent experiments over three novel, synthetic datasets, we evaluate paired completion against prompt-based LLM methods and labelled methods using traditional NLP and recent LLM contextual embeddings. We additionally conduct a cost-based analysis to mark out the feasible set of performant methods at production-level scales, and a model bias analysis. Together, our work demonstrates a feasible path to scalable, accurate and low-bias issue-framing in large corpora.
Creation-Date: 2024-06
File-URL: http://soda-wps.s3-website-ap-southeast-2.amazonaws.com/RePEc/ajr/sodwps/2024-02.pdf
File-Format: Application/pdf
Number: 2024-02
Classification-JEL: C19, C55
Keywords: slant detection,  text-as-data, synthetic data, computational linguistics
Handle: RePEc:ajr:sodwps:2024-02