John Thickstun | projects

musicnet

Musical scores comprise a dense set of labels on performances of classical western music. These labelings are analogous to semantic segmentations of visual imagery. However, in order to use a score as a segmentation map, we must first align it to particular audio recording by warping precise timings in the score onto expressive timings chosen by the performers. This is the music alignment problem. Using an alignment algorithm, we created the MusicNet dataset by aligning scores to a collection of freely-licensed recordings. We can use these labels to, for example, train an automatic music transcription system. In work appearing at ICASSP 2018, we describe a state-of-the-art transcription model trained using MusicNet that to-date (2019) is the best-performing algorithm in the MIREX transcription challenge.

conditional sampling

We can use a generative model as a prior for decomposing a linear mixture of sources into its constituent parts. We demonstrate this on the left for linearly superimposed images of churches and bedrooms. The idea is to train train two generative models: one that estimates the likelihood of the distribution over images of churches, and another that does the same for images of bedrooms. We can separate a mixture of images by decomposing it into two images that are likely under the priors for churches and bedrooms respectively, that satisfy the constraint that decomposition must sum to the given mixture. This can be interpreted as conditional sampling from the posterior distribution over sources given a mixture. Our work on visual source separation has appeared at ICML 2020. Extensions of these results to audio separation and more general conditional sampling problems appeared at ICML 2021.

controllable generation

Generative models are powerful tools for revealing structure in data. Features learned by fitting an unsupervised generative modeling objective can be transferred to other tasks. Or, as seen in the source separation project above, we can directly leverage these generative models as priors. The anticipatory music transformer is a controllable generative model of music. Just like ChatGPT generates text, this model is trained to generate music compositions. Unlike most music models, the anticipatory music transformer can infill missing parts of a composition. For example, given a melody, this model can generate an accompaniment to that melody. Previous music transformers generate continuations of music given the past. Our models are designed to also have foreknowledge of the future; we say that they anticipate the parts of the music that are already written. They use this knowledge to infill the rest of the music.