Matthew Finlayson

About

Hello! I am a PhD student at USC, advised by Swabha Swa­yam­dip­ta and Xiang Ren. Previously, I was a Predoctoral Researcher at AI2, and before that I studied computer science and linguistics at Harvard.

My current research focuses on improving language modeling, sampling, and interpretability methods by building and exploiting our theoretical understanding of neural language models.

News

Posts

• The "right way" to ensemble language models.
• A differentiable function from binary to one-hot representations.
• Deep BA sampling (extending BAT).
• Research interest demos for working with me.
• Obtaining logprobs from an LLM API.
• The softmax function is linear.
• Visualizations

Software

• OpenLogProbs: a library for obtaining logprobs from API-protected language models.
• SS.py: my personal command line tool for searching and citing academic papers via Semantic Scholar.

Publications and preprints

1. Logits of API-Protected LLMs Leak Proprietary Information.
   Matthew Finlayson, Xiang Ren, and Swabha Swa­yam­dip­ta.
   ArXiv, 2024.
   PDF
   Abstract

   The commercialization of large language models (LLMs) has led to the common practice of restricting access to proprietary models via a limited API. In this work we show that, with only a conservative assumption about the model architecture, it is possible to learn a surprisingly large amount of non-public information about an API-protected LLM from a relatively small number of API queries (e.g., costing under $1000 USD for OpenAI’s gpt-3.5-turbo). Our findings are centered on one key observation: most modern LLMs suffer from a softmax bottleneck, which restricts the model outputs to a linear subspace of the full output space. We exploit this fact to unlock several capabilities, including (but not limited to) obtaining cheap full-vocabulary outputs, auditing for specific types of model updates, identifying the source LLM given a single full LLM output, and even efficiently discovering the LLM’s hidden size. Our empirical investigations show the effectiveness of our methods, which allow us to estimate the embedding size of OpenAI’s gpt-3.5-turbo to be about 4096. Lastly, we discuss ways that LLM providers can guard against these attacks, as well as how these capabilities can be viewed as a feature (rather than a bug) by allowing for greater transparency and accountability.

2. Closing the Curious Case of Neural Text Degeneration.
   Matthew Finlayson, John Hewitt, Alexander Koller, Swabha Swa­yam­dip­ta, and Ashish Sabharwal.
   ICLR, 2024.
   PDF Code
   Abstract

   Despite their ubiquity in language generation, it remains unknown why truncation sampling heuristics like nucleus sampling are so effective. We provide a theoretical explanation for the effectiveness of the truncation sampling by proving that truncation methods that discard tokens below some probability threshold (the most common type of truncation) can guarantee that all sampled tokens have nonzero true probability. However, thresholds are a coarse heuristic, and necessarily discard some tokens with nonzero true probability as well. In pursuit of a more precise sampling strategy, we show that we can leverage a known source of model errors, the softmax bottleneck, to prove that certain tokens have nonzero true probability, without relying on a threshold. Based on our findings, we develop an experimental truncation strategy and the present pilot studies demonstrating the promise of this type of algorithm. Our evaluations show that our method outperforms its threshold-based counterparts under automatic and human evaluation metrics for low-entropy (i.e., close to greedy) open-ended text generation. Our theoretical findings and pilot experiments provide both insight into why truncation sampling works, and make progress toward more expressive sampling algorithms that better surface the generative capabilities of large language models.

3. Attentiveness to Answer Choices Doesn't Always Entail High QA Accuracy.
   Sarah Wiegreffe, Matthew Finlayson, Oyvind Tafjord, Peter Clark, and Ashish Sabharwal.
   EMNLP, 2023.
   PDF Code
4. Decomposed Prompting: A Modular Approach for Solving Complex Tasks.
   Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark and Ashish Sabharwal.
   ICLR, 2023.
   PDF Code
5. Līla: A Unified Benchmark for Mathematical Reasoning.
   {Matthew Finlayson, Swaroop Mishra,} Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan.
   EMNLP, 2022.
   PDF Data Model Website Leaderboard
6. What Makes Instruction Learning Hard? An Investigation and a New Challenge in a Synthetic Environment.
   Matthew Finlayson, Kyle Richardon, Ashish Sabharwal, and Peter Clark.
   EMNLP, 2022.
   PDF Code
7. Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models.
   {Matthew Finlayson, Aaron Mueller,} Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov.
   ACL, 2021.
   PDF Code