Allegro

So how does A.I. actually work?

Volume 123, No. 9October, 2024

William Meade

This month’s submission for Local 802’s A.I. series is by William Meade, the chair of Local 802’s A.I. Committee.

So how exactly does big tech gather the vast music and information necessary to create AI music?

The short answer is that to a large extent, we give it to them…free of charge.

I thought it would be valuable for everyone to understand the process of how AI music is created. So why not start with the biggest creator of them all, TikTok. Let’s take a walk through the process.

TikTok, the social media giant is owned by a Chinese company, Byte Dance. It is renowned for its user-generated content, with music being a core element of its viral success.

This article will explore how TikTok uses music from video submissions, alongside its partnerships with Universal Music and other major labels, to feed into its AI models, which ultimately help in creating new music, enhancing recommendation systems, and optimizing content creation tools.

User-Generated Content: A Goldmine for Music Data

One of TikTok’s most significant assets is the vast amount of user-generated content that it hosts. Every day, millions of videos are uploaded to TikTok, many of which incorporate music in some form. These videos range from lip-sync performances and dance challenges to original content where users overlay their creations with popular songs or sounds from the platform’s library.

When users submit videos to TikTok, they often attach music from TikTok’s vast licensed library or user-uploaded sound bites. TikTok collects detailed data from these videos, including:

User behavior data: This includes how users engage with specific songs, the number of videos created using a particular track, and how often a song is shared or liked.

Contextual data: TikTok gathers insights into how music is used in conjunction with certain types of content (e.g., dancing, comedy, or tutorials). This helps TikTok understand trends and the contextual relevance of particular songs.

Audio analysis: The platform analyzes the audio properties of the music used in videos, such as tempo, rhythm, pitch, and structure. This allows TikTok’s AI systems to learn more about the musical attributes that users respond to.

How TikTok Uses Video Submissions to Train AI Models

TikTok’s AI-driven ecosystem relies on sophisticated algorithms that recommend content, curate playlists, and even generate music. Here’s how TikTok leverages video submissions to train its AI models.

DATA COLLECTION FROM USER INTERACTIONS

Each time a user uploads a video, TikTok captures detailed metadata related to the music used, such as:

Song popularity: The number of times a song is used in user videos provides insights into its trending status. This data helps the AI understand which songs are likely to go viral.

User demographics and geography: TikTok collects information on the age, gender, and location of users interacting with particular songs. This data can influence music recommendations by AI and even shape the kind of music TikTok’s AI-generated tools might create.

Content type and music pairing: The AI also learns which types of content (dance, comedy, tutorials) typically use certain kinds of music. This contextual pairing informs future AI-generated music, allowing it to suggest or create tracks better suited for specific content types.

AUDIO PROCESSING AND ANALYSIS

The audio component of TikTok videos is analyzed using AI techniques like spectral analysis and neural networks. This allows TikTok to break down the characteristics of songs used in UGC. For instance, the AI can learn the:

Rhythmic patterns: By examining the beat and tempo, TikTok’s AI can understand how different rhythms align with content trends.

Melodic structures: It learns about the tonal qualities and musical motifs that resonate with users. This is useful for future AI-generated compositions.

Lyrical themes: TikTok also employs natural language processing (NLP) to analyze the lyrics of songs and understand their thematic appeal.

Through this analysis, TikTok’s AI models are trained to generate new music that can mimic these characteristics or suggest appropriate soundtracks to users when they create videos.

TRAINING A.I. TO RECOGNIZE MUSIC

The data gathered from user videos is used to train TikTok’s recommendation algorithms. These AI-driven recommendations are not limited to suggesting videos but extend to recommending music for content creators. TikTok’s algorithm understands which songs are trending, which fit certain content types, and even which tracks are likely to gain traction based on early interactions.

By training its AI models on user behavior, TikTok can suggest music that has a high likelihood of being well-received. For example, if a particular song is used frequently in dance challenges, TikTok’s AI will recommend it to users making similar content, fueling a feedback loop that enhances the song’s visibility and popularity.

TikTok’s Licensing Deals with Universal Music Group and Other Labels

In addition to user-generated content, TikTok has signed significant licensing deals with major record labels, including Universal Music Group (UMG), Sony Music Entertainment, and Warner Music Group. These partnerships give TikTok access to vast catalogs of popular music, which it can incorporate into the platform’s features, allowing users to include licensed tracks in their videos. Clearly, these partnerships are the major concern of all creative musicians which we have discussed in previous issues.

Here’s how TikTok leverages these licensing deals to benefit its AI programs:

ACCESS TO HIGH-QUALITY MUSIC DATASETS

The music catalog provided by UMG and other labels includes a wide array of songs, ranging from iconic hits to contemporary chart-toppers. This vast dataset of licensed music serves as additional training material for TikTok’s AI models. This is an extremely important point; AI analyzes the music without necessarily using the music itself to build its models. The hits teach it how to create hits.

AI-generated music programs require extensive datasets to learn how to compose or recommend music. By using licensed tracks from UMG, TikTok can train its AI to understand the characteristics of commercially successful music.

Hit structures: TikTok’s AI can analyze the composition of popular tracks (e.g., chord progressions, song structure) to learn what makes certain songs resonate with listeners.

Emotional tone: Songs often evoke specific emotions, and TikTok’s AI learns to associate musical elements with emotional responses. For example, major keys might be linked to happiness, while minor keys could evoke sadness or tension.

Production techniques: Licensed music often has polished production, with AI learning about mixing, layering, and instrumentation from these high-quality examples.

ENHANCING A.I.-GENERATED MUSIC

The integration of licensed music allows TikTok to feed its AI with data on songs that have already proven popular. As the AI learns from this content, it becomes better at creating new, original compositions that appeal to users’ tastes. For instance:

AI-generated music: TikTok’s AI programs can create new music tracks having learned from the structure, tempo, and style of UMG’s catalog to generate compositions that align with current musical trends.

Remixing and customization tools: TikTok also offers users’ tools to remix or modify music tracks. The AI learns from these interactions, analyzing how users alter licensed songs to create personalized versions for their videos. This data helps AI develop features that allow more creative freedom for users in music editing.

Ethical Considerations and the Future of AI Music on TikTok

TikTok’s use of music from video submissions and its partnerships with major labels like Universal Music brings a host of opportunities and challenges. While the platform is poised to revolutionize how music is created, distributed, and monetized, there are ethical concerns surrounding the use of AI-generated music.

TikTok’s innovative approach to combining user-generated content with licensed music from deals such as the one with Universal offers an unprecedented data source for training its AI music programs. By analyzing millions of user videos, TikTok’s AI learns from both the creative ways users incorporate music and the structural elements of commercially successful tracks. As AI continues to play a larger role in content creation, TikTok’s ability to synthesize human behavior with sophisticated algorithms will likely lead to even more personalized and innovative music experiences in the future.

Multi-instrumentalist William Meade, a member of Local 802 since 1979, is the chair of Local 802’s A.I. Committee. William is a seasoned performer and producer with over three decades of experience in television, Broadway, concerts and global events. During the last 12 months, he has spent considerable time and effort on understanding the accelerating impact AI is having across all professions, including that of musician’s. His knowledge of AI encompasses practical, academic and regulatory aspects, all of which continue to evolve at a rapid pace. E-mail William Meade at: wmeade@stellationentertainment.com. Send feedback on Local 802’s A.I. series to Allegro@Local802afm.org.

CLICK HERE TO JOIN OUR A.I. COMMITTEE E-MAIL LIST AND TO LEARN ABOUT UPCOMING MEETINGS