Music has always been central to Willie Payne’s life.
Growing up, he sang in children’s choirs, played drums in concert and marching bands, and played guitar in the jazz band and pit orchestra. As a teenager, he became obsessed with video game music and even attended a summer camp at the Berklee School of Music to learn how to score music for games.
As an undergraduate student at the University of Colorado, Payne learned software to create custom guitar effects and began composing music for computers and other musicians. He also joined several electronic music ensembles, including the Boulder Laptop Orchestra, which uses laptops, tablets, and other devices as instruments to create new sounds and forms of musical expression.
It wasn’t just the intersection of music and technology that inspired Payne. His mother’s work as a psychologist at Craig Hospital, which serves patients with spinal cord injuries — many of whom are in wheelchairs — opened his eyes to the importance of accessibility.
“I was thinking about accessibility from an early age, in terms of the accessibility of lived spaces and access to buildings,” Payne recalls. “I even played mixed pair doubles tennis with a wheelchair user for a few years.”
These experiences led him to look for projects that would bridge the gap between music, technology, and accessibility. An opportunity came during his junior year, when a group of professors received a seed grant to make the university’s PhET simulations more accessible. Used to teach scientific concepts, these interactive websites were inaccessible to screen readers, software used by many blind people to access their devices.
“I joined the project and used my coding and sound design skills to build prototypes and try to convey these really visual things into sound, touch, or other alternative modalities,” Payne says.
Payne’s journey brought him to New York University, where he earned a PhD in Music Technology. While there, he partnered with blind musicians at the Filomen M. D’Agostino Greenberg (FMDG) Music School to develop accessible music notation software and form FilOrk, a laptop orchestra comprised of teens using code to perform electronic music.
Now, as a professor in the UNC School of Information and Library Science, he’s continuing to use inclusive design processes to make music more accessible for people who are blind or have low vision.
Bridging musical barriers
In 2017, while searching for a dissertation topic, Payne met Amy Hurst, a professor who specializes in accessibility research and participatory design — when scholars work closely with a community to learn its needs and then collaboratively build and implement solutions.
With Hurst’s encouragement, Payne reached out to FMDG Music School about volunteering opportunities and was offered a position as an Accessible Music Technology Fellow. He wasn’t sure if this would lead to any research opportunities, but during that time he began to develop a camaraderie with the students and staff at the school.
“It started out with a long period of just learning from them and building trust and becoming a part of the community,” Payne explains. “To have lasting impact, the research needs to emerge organically.”
One of his roles was transcribing sheet music so students could access it in braille and large-print formats. Converting sheet music into an embossed braille document is a challenging process that involves multiple steps and requires entering the score into several software programs.
To reduce the need for such a complex workflow, Payne worked with two experienced blind musicians from the school and a team of undergraduate researchers to design an open-source software SoundCells. In SoundCells, music is notated with a unified language, like text. The software then outputs scores in print and braille formats all at once.
Soon after, Payne began working with FiLOrk, helping teen musicians write code that produces sounds in real-time. Typically, the code is projected onto a wall for the audience to see.
One of the biggest challenges in making this style of music accessible to vision-impaired musicians is figuring out how to get several different and unpredictable setups to work together.
“For example, a student might be using a Mac, the Safari browser, the VoiceOver screen reader, and a 40-cell braille display. Our software might work for three of those four, but not the fourth,” Payne shares.
While the technical challenges were annoying at times, the students embraced the improvisational spirit of the music and even ended up incorporating some of the technical mishaps into their performance.
“My favorite thing in the world is seeing people make stuff they’re proud of,” Payne says. “It was so cool hearing the kids talk about why those pieces were meaningful to them. Building tools that can empower people in those moments is what gets me going.”
Expanding accessible learning
Now at Carolina, Payne continues his dedication to making music education and STEM learning more accessible to low-vision and blind students through two upcoming projects.
The first will design tactile graphics — physical representations of visual content that aid in teaching music-related concepts. They will be printed on heat-sensitive paper, which allows the ink to swell, creating a raised texture that people with visual impairments can feel.
“There are distinct challenges in designing useful learning materials that are meant to be felt more than seen. Line thickness, information density, and texture are all important concerns,” Payne explains.
The other project involves interviewing low-vision musicians about how large-print notations impact access to music. Early findings indicate that vision ability goes far beyond notation access and affects fundamental aspects of one’s musical identity, including their choice of genre, instrument, and collaborators.
“Designers tend to get fixated on solutions that work for say 80% of people but leave out the remaining 20%,” Payne says. “Usually, people with disabilities fall into this minority. The argument behind Inclusive Design methodologies is that if we focus instead on the excluded groups, we’ll find a better solution for everyone.”