'AI Synthesized Commentary Team in video games' patent pending and demo videos

[Ryzza5]

 

Ian Bell, CEO of Straight 4 Studios (and former CEO of Slightly Mad Studios, known for the Project CARS and Need for Speed Shift racing games), revealed in a tweet some innovative work using game event triggers (such as a player scoring a goal or crashing a racing car) to dynamically form prompts to generate relevant AI commentary or pit-crew advice, influenced by real-life voices.  The work is likely to be included in an upcoming racing simulation GTR Revival.

 

This first video explains the concept, how it works, and provides a demonstration.  The game featured below is said to be Automobilista 2, which uses the same Madness game engine as Project CARS 2.

 

 

This second video demonstrates a race by internal QA tester Austin Ogonoski speaking to a virtual pit crew member, an AI borrowing its voice from Ben Collins (Straight 4 Studios Handling consultant, also famous for his role as The Stig (BBC Top Gear) and a stunt driver in recent James Bond and Fast & Furious movies).  

 

Partial transcript from first video:

A method for spontaneously writing and reading aloud unique, contextually-accurate, non-repetitious, simulated "sports commentary" and "sports color-commentary" script, including but not limited to "analysis", "coaching", "tactics" and "assistance", inside sport or motorsport events or video games.

In other words, 'A fully AI-generated AI broadcast crew including a commentator and color-commentator inside a video game'

How is this done?

1. For starters, every action within the game world activates and associated line of code linked to that action, for example, there's a line of code activated when a player scores a goal, when a player makes a body-check in a hockey game, when the period ends in a hockey game, when the race is finished, things like that.
2. This individual game action in code is then linked to a human-written writing prompt. For example, if a goal is scored, the writing prompt might say: "excitedly describe, as if you are a sports commentator, [player name] scoring a goal to win the hockey game against [team] in overtime."
3. For step 3, that human written writing prompt is immediately sent to an AI text synthesis generator when that game code is triggered. So if a goal is scored in the game, that writing prompt is immediately sent to an AI.
4. The AI text synthesis then follows the writing prompt instructions to generate unique lines of "sports commentary" text script in real-time, including but not limited to additional parameters such as player name and team name.   i.e. "What an incredible goal to win in overtime by number ninety seven, Connor McDavid!"
5. That AI generated line of "sports commentary" script is then sent to AI voice synthesis... to be read aloud.
6. A second AI text synthesis reads and/or interprets the unique line of "sports commentary" text script generated by the first AI text synthesis, and composes a unique "response" script, otherwise known as "color commentary"... from a different sounding voice.
7. These steps are repeated throughout the duration of the game/match/session giving users the impression of a "commentary team talking back and forth" with each other.

 

 

It's interesting to see multiple layers of AI (text generation, conversational response, and speech synthesis) being used in video games, particularly the possibilities of assisting or coaching players.  Numerous sim-racing games support API extensibility which has allowed for community-created Pit Crew chief mods, which while not AI-based, increase immersion and allow visual HUD elements to be hidden.  This new AI-based system needs to do even better, without feeling cheap or over-done.   

Whilst simulated commentary has existed in many sports-based video games such as FIFA for decades, I cannot think of any racing games that feature anything at a similar level (besides background repetitive commentary in games like GRID Legends).  Some larger organised racing leagues stream or record races with live human commentary but smaller groups may not have the time or resources for this so these might be interested in this endeavour. 

One concern however is that a successful patent application may dissuade other studios from implementing similar features.

 

Sources

 

What do you think about the strides made by AI this year such as Chat GPT (language model) and Stable Diffusion (art generator)?   Are you happy to see them added to video games?  Do you have other ideas how they could be used in games?

[jamiemb17]

Hey, was just visiting this site and thought I'd say thanks for this post, didn't realise this stuff was happening, very interesting indeed.