YouTube Adds Automatic Sound Effect Captions

YouTube now has a sound effect captioning system(Opens in a new window) that can automatically identify music, sound, and laughter in videos.

It’s a small but significant addition to the accessibility features of the internet’s go-to video streaming site, a Google subsidiary. YouTube has offered automatic captions for dialogue tracks based on Google’s voice recognition since 2009. But captioning sound effects is a much harder task, according to Google engineers, and one that’s only feasible with the advancements in machine learning that have taken place over the past couple of years.

The problem is not so much a computer’s ability to detect and classify things (products like Google Photos already have detection capabilities for images, for instance), but instead the lack of a significant database of sound effects to use when training the neural network that would identify them.

“While labeled ambient sound information is difficult to come by, we were able to generate a large enough dataset for training using weakly labeled data,” Google engineer Sourish Chaudhuri wrote in a blog post(Opens in a new window). The team decided to focus on music, sound, and laughter first since they add meaningful context to a video’s dialogue for people who are deaf or hard of hearing.

After processing thousands of hours of videos, YouTube now has a trained artificial intelligence algorithm for sound effects, which you can check out in videos like this clip(Opens in a new window) from America’s Got Talent (click the CC button to activate captions). The work is not yet done, however, according to Google engineer Noah Wang.

Recommended by Our Editors

How to Download YouTube Videos

“Future challenges might include adding other common sound classes like ringing, barking and knocking, which present particular problems—for example, with ringing we need to be able to decipher if this is an alarm clock, a door or a phone,” he wrote(Opens in a new window).