An 8-bit plumber with a mustache and a taste for mushrooms was the height of cutting-edge entertainment in 1985. Now, an AI model can recreate the entire game based on some basic prompts (and with sometimes hilarious glitches in the process).
The MarioVGG demonstrated in a new academic paper by video game character developer Virtuals Protocol shows how AI might help collaborate in future AI-driven video game production or possibly why it shouldn’t.
MarioVGG is an experiment in getting AI to produce plausible gameplay videos from a series of prompts on how the environment should look and behave and how characters should act. It’s a long-form version of the kind of text-to-video tools quickly growing in popularity. It is trained on more than 737,000 frames of gameplay footage from Super Mario Bros.
However, the AI Mario doesn’t do everything the character in the Nintendo game does, as the researchers limited the model’s interaction capabilities to just two moves; “run right” and “run right and jump.” The resulting images form a video, albeit one suggesting a player deliberately making it more challenging by avoiding power-ups, only jumping to one height, or moving left.
You can see a bit of MarioVGG’s performance below. It’s simultaneously astonishingly good at recreating the game’s look while also full of glaring errors and flaws. Though the AI-generated sequences are mostly coherent in matching a user’s input to the way Mario acts, it’s very much not the same speed, with each frame taking a little bit of time for the AI to create. Not to mention the occasional disappearance of Mario from the screen or his transformation into an enemy character for a bit.
Super MarAIo Modeling
Still, seeing the AI show a partial understanding of cause-and-effect between user inputs and the resulting gameplay moves potential code-free video game development a lot closer to reality. AI models like MarioVGG aren’t going to replace video game developers or the standard engine any time soon, but the idea of just explaining to an AI how you want a game’s physics and environments to function instead of coding it all manually is far too tempting for technical obstacles to discourage.
There’s a lot left to solve since video games involve a constantly evolving series of interactions between the player and the game environment. That’s much more complex than static images or simple actions in a video. Accurately recreating this interactivity in real-time is a challenge that MarioVGG has not yet solved – not that MarioVGG is the only effort in that regard.
Google’s GameNGen recently showed off a playable version of Doom produced by an AI model. Despite being an older game, Super Mario Bros. requires more nuanced control over character movement and environmental interaction than Doom, so the result is not quite as fast or accurate. AI is currently much better at text-based games with occasional illustrations than something like a Mario game. But, like the early days of video games, an evolution toward the fast-paced, interactive video games of Nintendo’s golden age could be coming faster than any goomba.