Melanie Walsh

Assistant Professor | Information School | University of Washington | Seattle, USA 🌦️😎

Hi, I’m Melanie Walsh. I’m an Assistant Professor in the Information School at the University of Washington. Before UW, I was a Postdoctoral Associate in Information Science at Cornell University, where I worked with David Mimno. I received my PhD in English Literature from Washington University in St. Louis.

My research interests include data science, digital humanities, cultural analytics, literature, and libraries—preferably all of the above combined. You can download my full CV here.

I am currently at work on a book project, When Postwar American Fiction Went Viral: Protest, Profit, and Popular Readers in the 21st Century, which follows the social media afterlives of American authors.

I also designed and released a free, open-source textbook, Introduction to Cultural Analytics & Python, which introduces the programming language Python to people interested in the humanities and social science.

With wonderful collaborators, I also lead several projects that I’m proud of:

In my free time, I like to play basketball 🏀, hike ⛰️, and paddle board among seal friends 🦭 (like in Shilshole Bay pictured above!).

I am recruiting one student to join my group in 2025-2026. For more info, see “Mentorship”.

news

Sep 15, 2024	Grant news! The Post45 Data Collective received a Level II NEH Digital Humanities Advancement Grant. This award will help us make book data more useable and interoperable.
Oct 30, 2023	Grant news! Very excited to share that we received a Mozilla Responsible Computing Challenge Award. This award is supporting Responsible Datasets in Context—a repository of datasets that are paired with rich documentation, data essays, and teaching resources. Read the press release.
Sep 01, 2023	Job news! I started a new position as a tenure-track Assistant Professor in the Information School at the University of Washington.
Jan 15, 2023	Grant news! We were awarded a Level III NEH Digital Humanities Advancement Grant for our AI for Humanists Project. Read the press release.

selected publications

Sonnet or Not, Bot? Poetry Evaluation for Large Models and Datasets

Melanie Walsh, Maria Antoniak, and Anna Preus

In Findings of the Association for Computational Linguistics: EMNLP, Nov 2024

arXiv PDF Code & Data Copy Bibtex Citation Read the Abstract

Large language models (LLMs) can now generate and recognize poetry. But what do LLMs really know about poetry? We develop a task to evaluate how well LLMs recognize one aspect of English-language poetry—poetic form—which captures many different poetic features, including rhyme scheme, meter, and word or line repetition. By using a benchmark dataset of over 4.1k human expert-annotated poems, we show that state-of-the-art LLMs can successfully identify both common and uncommon fixed poetic forms—such as sonnets, sestinas, and pantoums—with surprisingly high accuracy. However, performance varies significantly by poetic form; the models struggle to identify unfixed poetic forms, especially those based on topic or visual features. We additionally measure how many poems from our benchmark dataset are present in popular pretraining datasets or memorized by GPT-4, finding that pretraining presence and memorization may improve performance on this task, but results are inconclusive. We release a benchmark evaluation dataset with 1.4k public domain poems and form annotations, results of memorization experiments and data audits, and code.

@inproceedings{Walsh_Antoniak_Preus_2024,
  title = {Sonnet or Not, Bot? Poetry Evaluation for Large Models and Datasets},
  abstractnote = {Large language models (LLMs) can now generate and recognize poetry.
     But what do LLMs really know about poetry? We develop a task to evaluate how well LLMs recognize one aspect of English-language poetry—poetic form—which captures many different poetic features, including rhyme scheme, meter, and word or line repetition. By using a benchmark dataset of over 4.1k human expert-annotated poems, we show that state-of-the-art LLMs can successfully identify both common and uncommon fixed poetic forms—such as sonnets, sestinas, and pantoums—with surprisingly high accuracy. However, performance varies significantly by poetic form; the models struggle to identify unfixed poetic forms, especially those based on topic or visual features. We additionally measure how many poems from our benchmark dataset are present in popular pretraining datasets or memorized by GPT-4, finding that pretraining presence and memorization may improve performance on this task, but results are inconclusive. We release a benchmark evaluation dataset with 1.4k public domain poems and form annotations, results of memorization experiments and data audits, and code.},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP},
  author = {Walsh, Melanie and Antoniak, Maria and Preus, Anna},
  year = {2024},
  month = nov,
  language = {en}
}