Melanie Walsh

Assistant Professor | Information School | University of Washington | Seattle, USA 🌦️😎

Hi, I’m Melanie Walsh. I’m an Assistant Professor in the Information School at the University of Washington. Before UW, I was a Postdoctoral Associate in Information Science at Cornell University, where I worked with David Mimno. I received my PhD in English Literature from Washington University in St. Louis.

My research interests include data science, digital humanities, cultural analytics, literature, and libraries—preferably all of the above combined. You can download my full CV here.

I am currently at work on a book project, When Postwar American Fiction Went Viral: Protest, Profit, and Popular Readers in the 21st Century, which follows the social media afterlives of American authors.

I also designed and released a free, open-source textbook, Introduction to Cultural Analytics & Python, which introduces the programming language Python to people interested in the humanities and social science.

With wonderful collaborators, I also lead several projects that I’m proud of:

In my free time, I like to play basketball 🏀, hike ⛰️, and paddle board among seal friends 🦭 (like in Shilshole Bay pictured above!).

I am recruiting one student to join my group in 2025-2026. For more info, see “Mentorship”.

news

Oct 22, 2024	This week I am attending the Mozilla Responsible Computing Challenge (RCC) Global Conclave in Kerala, India on behalf of the Responsible Datasets in Context project.
Sep 15, 2024	Grant news! The Post45 Data Collective received a Level II NEH Digital Humanities Advancement Grant. This award will help us make book data more useable and interoperable.
Oct 30, 2023	Grant news! Very excited to share that we received a Mozilla Responsible Computing Challenge Award. This award is supporting Responsible Datasets in Context—a repository of datasets that are paired with rich documentation, data essays, and teaching resources. Read the press release.
Sep 01, 2023	Job news! I started a new position as a tenure-track Assistant Professor in the Information School at the University of Washington.
Jan 15, 2023	Grant news! We were awarded a Level III NEH Digital Humanities Advancement Grant for our AI for Humanists Project. Read the press release.

selected publications

Does ChatGPT Have a Poetic Style?

Melanie Walsh, Anna Preus, and Elizabeth Gronski

In Computational Humanities Research (CHR), Dec 2024

arXiv:2410.15299

arXiv PDF Code Copy Bibtex Citation Read the Abstract

Generating poetry has become a popular application of LLMs, perhaps especially of OpenAI’s widely-used chatbot ChatGPT. What kind of poet is ChatGPT? Does ChatGPT have its own poetic style? Can it successfully produce poems in different styles? To answer these questions, we prompt the GPT-3.5 and GPT-4 models to generate English-language poems in 24 different poetic forms and styles, about 40 different subjects, and in response to 3 different writing prompt templates. We then analyze the resulting 5.7k poems, comparing them to a sample of 3.7k poems from the Poetry Foundation and the Academy of American Poets. We find that the GPT models, especially GPT-4, can successfully produce poems in a range of both common and uncommon English-language forms in superficial yet noteworthy ways, such as by producing poems of appropriate lengths for sonnets (14 lines), villanelles (19 lines), and sestinas (39 lines). But the GPT models also exhibit their own distinct stylistic tendencies, both within and outside of these specific forms. Our results show that GPT poetry is much more constrained and uniform than human poetry, showing a strong penchant for rhyme, quatrains (4-line stanzas), iambic meter, first-person plural perspectives (we, us, our), and specific vocabulary like “heart,” “embrace,” “echo,” and “whisper.”
@inproceedings{Walsh_Preus_Gronski_2024, title = {Does ChatGPT Have a Poetic Style?}, doi = {10.48550/arXiv.2410.15299}, note = {arXiv:2410.15299}, number = {arXiv:2410.15299}, publisher = {arXiv}, author = {Walsh, Melanie and Preus, Anna and Gronski, Elizabeth}, year = {2024}, month = dec, booktitle = {Computational Humanities Research (CHR)} }

Sonnet or Not, Bot? Poetry Evaluation for Large Models and Datasets

Melanie Walsh, Maria Antoniak, and Anna Preus

In Findings of the Association for Computational Linguistics: EMNLP, Nov 2024

arXiv PDF Code & Data Copy Bibtex Citation Read the Abstract

Large language models (LLMs) can now generate and recognize poetry. But what do LLMs really know about poetry? We develop a task to evaluate how well LLMs recognize one aspect of English-language poetry—poetic form—which captures many different poetic features, including rhyme scheme, meter, and word or line repetition. By using a benchmark dataset of over 4.1k human expert-annotated poems, we show that state-of-the-art LLMs can successfully identify both common and uncommon fixed poetic forms—such as sonnets, sestinas, and pantoums—with surprisingly high accuracy. However, performance varies significantly by poetic form; the models struggle to identify unfixed poetic forms, especially those based on topic or visual features. We additionally measure how many poems from our benchmark dataset are present in popular pretraining datasets or memorized by GPT-4, finding that pretraining presence and memorization may improve performance on this task, but results are inconclusive. We release a benchmark evaluation dataset with 1.4k public domain poems and form annotations, results of memorization experiments and data audits, and code.

@inproceedings{Walsh_Antoniak_Preus_2024,
  title = {Sonnet or Not, Bot? Poetry Evaluation for Large Models and Datasets},
  abstractnote = {Large language models (LLMs) can now generate and recognize poetry.
     But what do LLMs really know about poetry? We develop a task to evaluate how well LLMs recognize one aspect of English-language poetry—poetic form—which captures many different poetic features, including rhyme scheme, meter, and word or line repetition. By using a benchmark dataset of over 4.1k human expert-annotated poems, we show that state-of-the-art LLMs can successfully identify both common and uncommon fixed poetic forms—such as sonnets, sestinas, and pantoums—with surprisingly high accuracy. However, performance varies significantly by poetic form; the models struggle to identify unfixed poetic forms, especially those based on topic or visual features. We additionally measure how many poems from our benchmark dataset are present in popular pretraining datasets or memorized by GPT-4, finding that pretraining presence and memorization may improve performance on this task, but results are inconclusive. We release a benchmark evaluation dataset with 1.4k public domain poems and form annotations, results of memorization experiments and data audits, and code.},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP},
  author = {Walsh, Melanie and Antoniak, Maria and Preus, Anna},
  year = {2024},
  month = nov,
  language = {en}
}

The Challenges and Possibilities of Social Media Data: New Directions in Literary Studies and the Digital Humanities

Melanie Walsh

Nov 2023

publisher: U of Minnesota Press

Debates in DH PDF Copy Bibtex Citation

@inbook{Walsh_2023,
  title = {The Challenges and Possibilities of Social Media Data: New Directions in Literary Studies and the Digital Humanities},
  url = {https://dhdebates.gc.cuny.edu/read/f3f87448-138c-4d19-8ff8-b06acf40ddd1/section/a57b98ab-0f10-45d0-b205-3e563aab7ea8#ch18},
  note = {publisher: U of Minnesota Press},
  booktitle = {Debates in the Digital Humanities 2023},
  publisher = {University of Minnesota Press},
  author = {Walsh, Melanie},
  year = {2023}
}

Where is all the book data?

Melanie Walsh

Public Books, Nov 2022

🎙️I spoke about this essay with Australian radio ABC.🎙️

Public Books Sydney Review of Books Interview

The Goodreads “Classics”: A Computational Study of Readers, Amazon, and Crowdsourced Amateur Criticism

Melanie Walsh, and Maria Antoniak

Post45 and Journal of Cultural Analytics, Nov 2021

Post45 Journal of Cultural Analytics PDF Interactive Plots Code & Data Goodreads Scraper Copy Bibtex Citation

@article{Walsh_Antoniak_2021,
  title = {The Goodreads “Classics”: A Computational Study of Readers, Amazon, and Crowdsourced Amateur Criticism},
  volume = {6},
  url = {https://culturalanalytics.org/article/22221.pdf},
  number = {2},
  journal = {Post45 and Journal of Cultural Analytics},
  author = {Walsh, Melanie and Antoniak, Maria},
  year = {2021},
  bibtex_show = true
}