publications

2026

1. MulTTiPop: A Multitrack Transcription Dataset for Pop Music

   Nathan Pruyne, Benjamin Stoler, William Chen, and 3 more authors

   arXiv preprint arXiv:2607.08756, 2026

   Abs PDF Website Dataset

   We present MulTTiPop, a dataset of pop music segments and their associated multitrack MIDI recordings for the evaluation of automatic music transcription models. MulTTiPop contains 572 segments of popular music totaling 3.5 hours of audio, and contains songs from diverse genres and decades from the 1930s to 2000s. To collect this dataset, we perform metadata-based matching on song segments from the Lakh MIDI and TheoryTab datasets, manually identify an anchor beat between the audio and MIDI, then use beat tracking on the audio and warp the MIDI to match its tempo and timing. We evaluate state-of-the-art automatic music transcription models on MulTTiPop and find substantial room for improvement, with the best model achieving 38% Onset F1. More details and sound examples of MulTTiPop are available at https://gclef-cmu.org/multtipop

2025

1. HARP 3.0: Generalizing I/O and API Support for Machine Learning in Digital Audio Workstations

   Frank Cwitkowitz, Christodoulos Benetatos, Qixin Deng, and 6 more authors

   In NeurIPS 2025 AI for Music Workshop, 2025

   Abs Code

   HARP is a framework that integrates deep learning models into digital audio workstations (DAWs) through hosted, asynchronous, remote processing, allowing users to access state-of-the-art models without leaving their production environment. In HARP 3.0, we introduce generalized input–output handling for audio, MIDI, and text; expand API compatibility beyond Gradio to include providers such as Stability AI; and add workflow features including a media clipboard for storing and reusing processed files. These improvements make HARP 3.0 a more versatile and extensible platform for creative audio applications. HARP is open source and available for download at https://github.com/TEAMuP-dev/HARP.

2. The Rhythm In Anything: Audio-Prompted Drums Generation with Masked Language Modeling

   Patrick O’Reilly, Julia Barnett, Hugo Flores Garcia, and 4 more authors

   In ISMIR 2025, 2025

   Abs PDF Code Website

   Musicians and nonmusicians alike use rhythmic sound gestures, such as tapping and beatboxing, to express drum patterns. While these gestures effectively communicate musical ideas, realizing these ideas as fully-produced drum recordings can be time-consuming, potentially disrupting many creative workflows. To bridge this gap, we present TRIA (The Rhythm In Anything), a masked transformer model for mapping rhythmic sound gestures to high-fidelity drum recordings. Given an audio prompt of the desired rhythmic pattern and a second prompt to represent drumkit timbre, TRIA produces audio of a drumkit playing the desired rhythm (with appropriate elaborations) in the desired timbre. Subjective and objective evaluations show that a TRIA model trained on less than 10 hours of publicly-available drum data can generate high-quality, faithful realizations of sound gestures across a wide range of timbres in a zero-shot manner.

2024

1. HARP 2.0: Expanding Hosted, Asynchronous, Remote Processing for Deep Learning in the DAW

   Christodoulos Benetatos, Frank Cwitkowitz, Nathan Pruyne, and 4 more authors

   In ISMIR 2024 Late Breaking Demos, 2024

   Abs Code

   HARP 2.0 brings deep learning models to digital audio workstation (DAW) software through hosted, asynchronous, remote processing, allowing users to route audio from a plug-in interface through any compatible Gradio endpoint to perform arbitrary transformations. HARP renders endpoint-defined controls and processed audio in-plugin, meaning users can explore a variety of cutting-edge deep learning models without ever leaving the DAW. In the 2.0 release we introduce support for MIDI-based models and audio/MIDI labeling models, provide a streamlined \textttpyharp Python API for model developers, and implement numerous interface and stability improvements. Through this work, we hope to bridge the gap between model developers and creatives, improving access to deep learning models by seamlessly intrgrating them into DAW workflows.

2. Fine-Grained and Interpretable Neural Speech Editing

   Max Morrison, Cameron Churchwell, Nathan Pruyne, and 1 more author

   In INTERSPEECH 2024, 2024

   Abs PDF Code Website

   Fine-grained editing of speech attributes—such as prosody (i.e., the pitch, loudness, and phoneme durations), pronunciation, speaker identity, and formants—is useful for fine-tuning and fixing imperfections in human and AI-generated speech recordings for creation of podcasts, film dialogue, and video game dialogue. Existing speech synthesis systems use representations that entangle two or more of these attributes, prohibiting their use in fine-grained, disentangled editing. In this paper, we demonstrate the first disentangled and interpretable representation of speech with comparable subjective and objective vocoding reconstruction accuracy to Mel spectrograms. Our interpretable representation, combined with our proposed data augmentation method, enables training an existing neural vocoder to perform fast, accurate, and high-quality editing of pitch, duration, volume, timbral correlates of volume, pronunciation, speaker identity, and spectral balance.

3. Crowdsourced and Automatic Speech Prominence Estimation

   Max Morrison, Pranav Pawar, Nathan Pruyne, and 2 more authors

   In 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024

   Abs PDF Code Website

   The prominence of a spoken word is the degree to which an average native listener perceives the word as salient or emphasized relative to its context. Speech prominence estimation is the process of assigning a numeric value to the prominence of each word in an utterance. These prominence labels are useful for linguistic analysis, as well as training automated systems to perform emphasis-controlled text-to-speech or emotion recognition. Manually annotating prominence is time-consuming and expensive, which motivates the development of automated methods for speech prominence estimation. However, developing such an automated system using machine-learning methods requires human-annotated training data. Using our system for acquiring such human annotations, we collect and open-source crowdsourced annotations of a portion of the LibriTTS dataset. We use these annotations as ground truth to train a neural speech prominence estimator that generalizes to unseen speakers, datasets, and speaking styles. We investigate design decisions for neural prominence estimation as well as how neural prominence estimation improves as a function of two key factors of annotation cost: dataset size and the number of annotations per utterance.

4. Machine learning-guided discovery of gas evolving electrode bubble inactivation

   Jack R. Lake, Simon Rufer, Jim James, and 7 more authors

   Nanoscale, 2024

   Abs PDF Code

   The adverse effects of electrochemical bubbles on the performance of gas-evolving electrodes are well known, but studies on the degree of adhered bubble-caused inactivation, and how inactivation changes during bubble evolution are limited. We study electrode inactivation caused by oxygen evolution while using surface engineering to control bubble formation. We find that the inactivation of the entire projected area, as is currently believed, is a poor approximation which leads to non-physical results. Using a machine learning-based image-based bubble detection method to analyze large quantities of experimental data, we show that bubble impacts are small for surface engineered electrodes which promote high bubble projected areas while maintaining low direct bubble contact. We thus propose a simple methodology for more accurately estimating the true extent of bubble inactivation, which is closer to the area which is directly in contact with the bubbles.

2023

1. Cross-domain Neural Pitch and Periodicity Estimation

   Max Morrison, Caedon Hsieh, Nathan Pruyne, and 1 more author

   In arXiv preprint arXiv:2301.12258, 2023

   Abs PDF Code

   Pitch is a foundational aspect of our perception of audio signals. Pitch contours are commonly used to analyze speech and music signals and as input features for many audio tasks, including music transcription, singing voice synthesis, and prosody editing. In this paper, we describe a set of techniques for improving the accuracy of widely-used neural pitch and periodicity estimators to achieve state-of-the-art performance on both speech and music. We also introduce a novel entropy-based method for extracting periodicity and per-frame voiced-unvoiced classifications from statistical inference-based pitch estimators (e.g., neural networks), and show how to train a neural pitch estimator to simultaneously handle both speech and music data (i.e., cross-domain estimation) without performance degradation. Our estimator implementations run 11.2x faster than real-time on a Intel i9-9820X 10-core 3.30 GHz CPU—approaching the speed of state-of-the-art DSP-based pitch estimators—or 408x faster than real-time on a NVIDIA GeForce RTX 3090 GPU. We release all of our code and models as Pitch-Estimating Neural Networks (penn), an open-source, pip-installable Python module for training, evaluating, and performing inference with pitch- and periodicity-estimating neural networks. The code for penn is available at github.com/interactiveaudiolab/penn.

2. Segmentation of tomography datasets using 3D convolutional neural networks

   Jim James^*, Nathan Pruyne^*, Tiberiu Stan, and 6 more authors

   Computational Materials Science, 2023

   Abs PDF Code

   Dendritic microstructures are ubiquitous in nature and are the primary solidification morphologies in metallic materials. Techniques such as X-ray computed tomography (XCT) have provided new insights into dendritic phase transformation phenomena. However, manual identification of dendritic morphologies in microscopy data can be both labor intensive and potentially ambiguous. The analysis of 3D datasets is particularly challenging due to their large sizes (terabytes) and the presence of artifacts scattered within the imaged volumes. In this study, we trained 3D convolutional neural networks (CNNs) to segment 3D datasets. Three CNN architectures were investigated, including a new version of FCDenseNet which we extended to 3D. We show that using hyperparameter optimization (HPO) and fine-tuning techniques, both 2D and 3D CNN architectures outperform the previous state of the art. The 3D U-Net architecture trained in this study produced the best segmentations according to quantitative metrics (intersection-over-union of 95.56% and a boundary displacement error of 0.58 pixels), while 3D FCDense produced the smoothest boundaries and best segmentations according to visual inspection. The trained 3D CNNs are able to segment entire 852 × 852 × 250 voxel 3D volumes in only ∼60 s, thus hastening the progress towards a deeper understanding of phase transformation phenomena such as dendritic solidification.