Breaking the Cycle - The Potential of Creative AI in Game Development

Game development is in crisis. Can creative AI tooling save it?

April 26, 2023

Tagged: Game Development   AI Art   GPT  

I’ve been thinking a lot recently about Creative AI tooling and its intersections with game development, and mostly how I think a lot of the applications (and discussions) of it right now are naïve. This really hit when listening to a recent episode of the Interdependence podcast with Runway ML founder Cristóbal Valenzuela.

In the podcast they talk deeply about mental models around the current creative AI moment, and what’s possible. But more specifically, they talk about how the thinking by everyone right now (themselves included) is fundamentally short-sighted. We’re in the stone age of this type of thing, feeling around in the dark with what little we know and can work with. This isn’t to say that we’re thinking about the wrong things, but that it’s truly difficult (if not impossible) to understand where things will go. Attempting to just graft AI tooling onto existing workflows may be a nice gap-filler for now (looking at you, “prompt engineer”), but that what exists now is likely nowhere close to what mature forms of this tooling will look like.

---

Game Development is Collapsing

The intersection with game development is less one of direct parallels, but more about how, in our current model of game development, we’re in a bit of a crisis.

AAA is unsustainable. The size and scope of a modern AAA game demands an incredibly large labor pool of talent that is only getting more expensive, and as such continues to be outsourced and cost-cut. From the linked article:

Sixty-five of Virtuos’s employees spent over two years building 11 of Horizon’s 32 enemy robot types; modeling many of the bandit settlements spread across the game world.

Virtuos was only one of 18 different outsourcing companies that worked on the game. Among the others: 3Lateral Studio in Novi Sad, Serbia, which helped character modeling and facial animation for human characters; Territory Studios, a graphic design company in New York that specializes in interface design; Audiomotion in London, which helped with motion capture; Kokku in Recife, Brazil helped build models for some of the robotic animals; and XPEC Art Center in Taipei, which helped build environmental assets to fill the game’s world.

18 separate outsourcing studios for ONE game! The ability to even participate in AAA at this scale obviously narrows more and more over time as player demands increase.

(NEW: In light of the issues around Jedi Survivor happening after I initially posted this, I added some context on what “collapse” means at the bottom of this piece.)

This anteing will reach a breaking point, either through total collapse or the efforts of a planet’s worth of creators working on a single game [4]. Or, said differently through a graph:

I think it’s easy to be like “hahaha fuck AAA!” but this position is a bit short-sighted. Instead of just AAA, there’s more of an “ecosystem” of development encompassing a series of streams. The prevailing modes right now are “AAA Production”, kicking up around the 1980s (albeit “AAA” meaning has shifted), and then “Indie” kicking up around the 2000s:

The stream of AAA is basically just “bleeding edge possible for game development”. Aka it is just “what is currently recently possible”, be it hardware, technology, etc.

2000s era indie is stems from a cultural reaction to what AAA was at the same time (think the classic “brown shooter”), in addition to affordances enabled by new digital distribution technology (Steam + Xbox Live), as well as democratized tooling to access those mechanics like XNA + SDL. I’m paraphrasing the history here — for a deeper reading on “indie” I highly recommend Liz Ryerson’s recent post The California Problem.

These new indie games were gobbled up both because they were legitimately good but also because “modern” games had been around enough to serve a generation that was getting its first pangs of nostalgia [1]. And so this happens:

Indie games, fueled by hitherto untapped nostalgia (that platforms now are much more willing to engage with), cemented themselves as remediating both the design tendencies and aesthetic qualities of what was the zeitgeist of AAA approximately 15-20 years ago (hence the profusion of puzzle-platformer games from this time, captured in Indie Game: The Movie).

The cultural reaction/technological democratization cycle has continued since then, providing a parallel timeline to AAA production in Indie, just offset by about 20 years. This builds to our current moment, where Indie is currently in its PS1 era (hence all the PS1/PSX horror games) [2].

To be clear this isn’t pure nostalgia: the tools and technology to engage with what was once bleeding-edge (we’ll come back to this) take time to filter down to non-professionals and become “accessible”. It’s a real “medium is the message” thing here where its also no coincidence that modern “indie” happened when it did (SDL came out in…. drumroll… 1998).

The net result of this is that modern “indie” production effectively lives in the wake of whatever AAA is doing. Though interestingly it also seems like there is actually a waterfall of “indie” happening, with the cycle starting over again (maybe Minit and Undertale are ground zero?). A whole new generation of people inspired by “Indie” 1 now wanting something that, again, gets “back to basics”. [3]

Which also means based on our previously established trends:

The incoming death of AAA doesn’t just mean the death of AAA, but the impending collapse of all of game development. As long as “indie” in any form lives in the wake of precedent-setting AAA, this seems bound to happen (also probably on shorter subsequent timelines).

The issue, I think, is that indie developers desperately want to look and act like AAA, just on a smaller scale. I think about this article a lot (and this picture from it):

“Indie” in some ways is just “the internalization of the AAA games industry into a single (or few) individuals”. Legitimacy stems from that mimetic desire/execution, so the idea of breaking free from it is hard to comprehend because being part of it is what it means to “be” indie — “indie” as “independent from (but next to) AAA”. Take away AAA and there is no “Indie” praxis.

This also speaks a bit to the waterfall — indie games perpetually being beholden to design and production constraints from elsewhere (AAA). We’re making the same games, but with less money and less support than the thing we’re trying to copy.

So in short, we’re all headed to collapse. Not just AAA:

What can you do?

1. Make games until the collapse I guess?
2. Break the cycle.

---

Break The Cycle

Let’s revisit something I just said:

The tools and technology to engage with what was once bleeding-edge take time to filter down to non-professionals.

I think a more apt way to describe this is

The tools and technology to engage with what was once bleeding-edge take time to filter down to non-professionals to allow them to create the same types of experiences those tools were created for in the first place.

Additionally this:

These tools then become the tools and technologies through which game development becomes accessible.

Should really be:

These tools then become the tools and technologies through which a very specific permutation of game development becomes accessible.

“Everything” is collapsing because the ecosystem modern game development operates in is largely dependent on the design and technical thinking of the AAA industry, and by using those tools we are align ourselves to its same value system and/or become beholden to create the things the tools were designed to create. If one thing falls it all falls down.

Said differently, using a DCC editor like Maya or Blender presupposes you’re working in a framework that relies on 3D models and “rigging”. This then presupposes a way a “character” inhabits a “virtual world”, which then necessitates tooling for lighting an imported model, etc.

It can feel impossible to think outside of that framework - isn’t this just what games are? Well in this way, yes:

This is “videogames”. Interactive experiences created with tooling like:

• Bitmap image editors (Photoshop/Aseprite)
• Music sequences + engines (Ableton/FMOD/Wwise)
• Code (C++/C#)
• Asset Generation
• Realtime Rendering (GPU + APIs/SDL)
• Etc.

Additionally, AAA, though it does move somewhat slow with tech adoption, is usually the quickest to adapt to new technology, in part to provide a moat around its possibilities and maintain defensibility around what AAA means. Or said differently, AAA defines AAA largely through technical implementation of new tooling. So:

Using tooling that defined/defines “videogames” (especially “with the grain of the tool”) will likely result in “videogames as we know it” in part because the tooling is a self-referential ecosystem that all points to itself. As soon as you use some part of this toolset the others emerge as answers to problems created by those tools. In the piece I wrote on “asking good questions” I mentioned something similar, albeit through the lens of the types of problem a tool creates:

In software, you are very likely using tools and libraries created by other people - by the nature of using those tools and libraries you are creating and running into problems produced by the circumstances of the tools that you use. This already then bounds your problem/solution space to those that can be produced — or said differently, by using a tool like MS Paint, you would produce“MS Paint-shaped problems”, not Adobe premiere shaped problems.

But this is creative production right? These tools are the tools, and so as we make the things that the tools can allow us to make, we inevitably funnel back into The Cycle. It’s inescapable.

Or at least, it seems like it. There’s actually another layer to this graph:

Large changes in tooling live downstream of the research output of scientific and academic communities. Though some research is targeted directly to games as we know them, The Research is often agnostic to implementation conditions and instead is about suggesting methods independent of any sort of directly-prescribed circumstances of application.

Because bleeding edge tooling must be bleeding edge by definition, research is often focused on “raising the ceiling” of knowledge (eg. advances in raytracing, meshlets (aka “Nanite”)) vs. “lifting the floor” (eg. better texture compression). New advances must additionally be conformed such that they are copacetic with the steady progress of the bleeding edge (ie. you don’t rewrite your whole game engine just because a new paper is published, you adapt it to your workflow).

This process of “conforming the bleeding” is where the cycle begins — by the time the bleeding edge has been interpreted for use in games, that interpretation filters down the funnel to then become the way to use something. Living downstream (“Indie”, etc.) means you inherit that current (like water) of interpretation.

This is obviously a undesirable way of things for those trying to break the cycle/avoid collapse:

• New research comes out that pushes the bleeding edge forward (mesh shaders, raytracing, neural radiance fields, etc.)
• The groups most able to adapt those technologies do so (AAA), setting precedent for use and implementation
• Those precedents cascade into something akin to law, and gather momentum around understanding and resources
• By the research is democratized to be accessible beyond AAA at the indie level, the way of doing the thing is calcified. A technology can be democratized in an infinite number of ways, but by the time we reach this level that initial decision implicitly cuts out other outcomes.

What do?

Creative AI

Though this seems to have been the state of things for a while, I think we’re on the verge of something new and potentially cycle-breaking. If you don’t care about breaking the cycle and just want to make puzzle platformers the old way, then stop reading now. If you’re interested in what actually gets us to something new, keep reading.

Let’s first look at this graph again:

Instead of considering all scientific breakthroughs, let’s just focus on the latest stuff around creative AI tooling like StableDiffusion, DALL-E, Midjourney, GPT, etc. (largely, transformers/diffusion models), roughly described as “Creative AI”.

Assuming similar logic that has been the past precedent, this is probably roughly the timeline that you could expect “normalized creative AI tooling” to be utilized in things other than AAA (think democratized versions of Ghostwriter).

However even as I type this I know this is not the case. This is because of one thing that very unique to most (all?) of these creative AI tools:

You can use the exact same tools that AAA is using, in the exact same form, right now

So the “cascade” doesn’t happen. The tooling is already democratized. It’s more like this:

Everyone is receiving the state of the art at the same time.

Any person with access to an internet connection can use the cutting edge A.I tooling for free, immediately.

Every large game company that is figuring out how to utilize AI tooling is using the exact same tools that anybody sitting in a bedroom in their underwear has access to. It’s all the same interfaces, and at the same cost. This is huge.

Naturally though, The Cycle does still exert itself. As said in a recent Axios newsletter on creative ai in game production:

The big picture: Most of the generative AI concepts for gaming amount to easing the pipeline congestion that makes game creation so time-consuming and labor-intensive.

AAA is caught up trying to do the same thing they’ve been doing — use new breakthroughs to augment existing workflows. Unity has already announced some in-development AI stuff. Assuming the same trajectories hold, it’s all still headed for collapse, even with AI.

Which is why I think it’s important to point out another characteristic of this immediate access: there are not yet any norms around use. Or more tangibly, you don’t need to rely on access via the cascade from “Bleeding edge tooling”. You don’t even need to rely on current tooling to access it. Instead, it represents a near-entirely new starting point:

While lots of other technology that has come into games has been transformative, the “local scope” of those transformation has limited the range of possibility to just hawking some software at a GDC booth. Or said differently, other technology usually just affects the means, but the ends remain the same.

What’s particularly special about current wave of A.I. technology is that, while the end can resemble what currently exists (hence Creative AI still pointing to “Videogames” as well as whatever the new thing is), the means are not just different, but can fully circumvent established game production pipelines. Let me say this again for the people in the back:

Creative AI Tooling allows you to fully circumvent established norms around game production

You can literally “render” a frame without needing to do any programming at all. You can get near final text of any sort (including code) without needing to write code. The reality is of course that you will need to have skills to coerce output to the desired resolution/fidelity, but the bones are there.

This is breaking the cycle.

Working Backwards

We now have access to outcomes without requiring traditional means. You can now basically start with the desired end state and work backwards, whereas for so long games have required starting from 0 (or close enough to it). I talked about this a bit in my previous post on Hallucinated Functions:

What this new mode posits is that you can treat an entire program *as if it already exists*. You aren’t so much chiseling as you are focusing in on the parts of the program that matter enough to be enumerated. Incidentally, it feels similar to watching something like DALL-E or Midjourney work. The final elements are present in the lowest resolution output, but get incrementally more clear with further revisions.

What does this even mean for games? Given the nature of these generators, the end state is effectively infinite — how do you even start?

Going back to the Interdependence podcast, Cristóbal talks about how to even “navigate the potential” of such a change:

A lot of folks I think are focusing on technology for the sake of technology. But I think of technology as the means to an end. It’s not about technology on its own, and it’s not about models and infrastructure. It’s about what’s the value of using that and for what. And if you start from that, if you start from first principles thinking, then focus on what really matters, which is the output.

And for us, the output is a story. You want to tell a story and you’re using this technology to tell a story. Let me help you craft that story. Sure, that involves knowing a lot about models and deep learning and building software. But at the end of the day, everyone who uses Runway is using Runway to tell a story. The big opportunity for us is to continue building on that idea, building that expressive layer to interact with the algorithms and with the new set of possibilities.

The idea is — basically go back to the very first idea of a “game”. What is it, fundamentally? What are the first principles of videogames?

There’s obviously a few answers (and some that surely conflict with each other!) but you can see how the framework can be illuminating.

For example, what if a game, really, is just a series of images on a screen with some degree of interactivity?

Applied to image generators, maybe you can think of them as a really slow, high-fidelity renderers — what does a game look like/play like that runs at 1 frame per 15 seconds? Obviously not an FPS, but what about a murder mystery game with mostly static scenes? An adventure game where the backgrounds are generated, but populated with a few “normal” character?

Or maybe games are just “digital interactions”, and “graphics” aren’t even essential.

Things like MUDs and text-based games then come into view, but augmented and expanded with things like GPT. Can you have an infinite MUD? Would that even be fun? Would it matter?

Regardless, this sort of first principles thinking is, I think, what helps us reason about here:

It’s easy to see this stuff as “different ways to make the same games”, and as I said above, this is definitely already happening.

But I think it’s also more potent to reconsider, through these new tools, what “making a game” even means. Are they even “games”?

Forward

I think for a while we’ll see stuff like what I mentioned above — older types of games reconsidered with new technology. But given that the tools trajectory is different, that there is a fundamental different starting point, I suspect new things will definitely emerge. Things that aren’t just playing the hits of past games, and are instead something new entirely. “Games” that only make sense with creative AI at their core.

And the thing I want to drill in on through this post is that, you can do this all right now.

You can choose to keep living in the wake of whatever features Unity and Unreal implement and be at the whims of a specific hegemonic idea of making a game, or you can strike out into a totally different domain where there are no rules, no norms, and everyone is just trying to figure out what works.

Cristóbal again:

Before the invention of NLEs (Non-Linear Video Editors — Premiere, Resolve, etc.), before the invention of software to edit video on a computer, you literally were cutting strips of film. You literally had destructive editing. You had to take a piece of film, cut it with like a razor and then glue it together.

The people who had access to those things and techniques were very limited. And then with NLEs, it wasn’t like a better margin improvement. It wasn’t like a better glue or a better razor or a better machine that like piece the film on pieces like faster. It was just a completely new way. It’s just a completely new, different way of making stories.

Fast forward to 2022 and you have TikTok, which builds on this idea of remixing video in a way you just can’t do that with physical film.

Imagine being in 1940 and predicting TikTok. You can’t. There’s so many things that you need to understand to make that possible. And so I think for what’s really exciting about this technology is that we’re entering that similar momentum. Don’t think about AI as a better razor, as a better gluing system for your film. Think about it as a fundamentally different piece of technology that will allow you to do things that just weren’t possible before. And the way that we’re going to tap into that is for me fascinating because it allows for a lot of different creative avenues and also allows for more folks that didn’t have access to all those very complicated tools before to now have access to it.

For games — again, don’t think of this as “just better pipelines”. We’re really touching on something new here, that over time will totally alter what it even means to make “games” (in the same way TikTok is “film”). AAA and general game creation public is still thinking in the “better razors” (or “faster horses”) mentality, but the potential here is so beyond that. And that trajectory is being set right now as people experiment with these new tools.

If you haven’t given any of these tools a go, I strongly encourage you to do so with an open mind. Push them to their limits and learn them (like any tool) — figure out their strengths and weaknesses, and start to build an intuition and understanding for what this type of thing can allow. You want to break the cycle? This is how you do it.

---

Some Predictions

We’re in Cristóbal’s proverbial 1940s right now, so extrapolating into the future is nearly impossible when we’re really just at Gen 1 of this stuff. However I think you can look at what’s currently possible and start extrapolating from there.

Here’s some things that I think are true/nearly-true that would be a good starting point:

• High quality assets are “solved”
  • Scaling bespoke creative pipelines will be trivial — think AAA games like Red Dead being able to 10x their size with 1.2x the effort.
  • This is definitely the “faster horses” angle, but will nonetheless be really compelling for a period of time
    • On the long term, these types of “fixed” games will just become a commodity - everyone can make a Red Dead or a GTA.
• Content can be personalized on demand
  • “Adaptive content customization” — think of this somewhere between creating a mod and having a game use telemetry to target you a specific way.
• Lower fidelity content can be created on demand
  • This is the big one I think, and the bar for fidelity on this will be slowly raised — basically, anything can be generated from scratch, on demand. Shipping games will be less about shipping a complete game, but instead be shipping some sort of generative-kernel that you engage with to produce new content.
    • Things like “seeds” in the Minecraft sense become the norm. Your permutation of a game will be something like a “seed tree” that you can share with others so they can play the game you “found”.
  • I again think of “incremental determinism” here — imagine a “farming” game with loose hooks for play that get more specialized over time. If you decide the farming game is about tractor repair, the game turns into that, whereas someone else’s experience of the game could be all about soil cultivation.
    • Maybe implicit here, but games become a mix of shipping an ai model + “saturation logic”
• Games themselves can be customized or “skinned” aka Style-transfer a whole game
  • This is some real next level stuff, but I think it’s not crazy to think you can eventually take a rhythm game and “style transfer” it to a sokoban game, or an FPS, etc.
    • One thing I think is going to become really important is having a structured language for actually talking about “design”. Something separate from logic, but basically a way to describe how a game works, separate from the implementation of that ruleset. I’ve seen a few things like this float out of GDC talks, but this will need to be something that is more production ready that can have hooks to logic.

Top level “ideas”

• Infinitude
  • In the new model, “games” will not be “fixed”. I don’t mean in some bullshit NFT sense, but more that every game will become malleable, per player, with scope of all things assuming infinitude at a core.
  • Despite varying ways that players can interact with the game, it’s still possible that players can interact with each other in meaningful ways.
• Do not assume you know what the player wants
  • I think game creation will become less of a realm for a designer to directly make an experience (at least at the mass-produced level), and instead become about providing enough “hooks” in a game that a player can latch onto to explore.
    • Game differentiation will be less about exact content, but how well a game is able to adapt to user wants while still remaining cohesive

And with that I’ll finally end this post. If you got all the way to the bottom, thanks for reading! I’m hoping this can be informative and generative for people who are wary of some of the AI stuff going on or are interested in what its implications are for game development. I definitely encourage you to try and experiment! Have comments or other thoughts? Hit me up on Twitter/Mastodon.

Also, if you’re someone out there who wants to work on whatever this stuff is/will be, please reach out to me by email at [email protected]. I would love to talk with other people thinking similarly about creative AI and what it can do for game development!

---

(UPDATE) Jedi Survivor woes

As if by magic, I posted this piece and then only a few days later a major AAA game by a major AAA publisher and a major AAA developers release on PC and… was almost unplayable.

This is not an indie team. This is top shelf AAA development, and they were unable to release a major title without not just bugs, but catastrophically bad bugs. They will likely fix these bugs in due time, but will now be climbing out of an algorithmic hole they dug themselves on Steam. The whole narrative around the game is just that it has bad performance.

These are the signs of collapse I’m pointing to when I talk about collapse. That the people supposedly best suited to put out the best product can’t even do that. It’s funny, because I also saw a few takes like this:

I think it’s a false dichotomy to assume that there is large overlap between people who play whatever big AAA game there is and “hot indie stuff”. I think most people just buy whatever is big, people who proactively seek “indie games” are still niche by an order of magnitude (or more?).

But more to the point — indie devs want to make stuff like Jedi Survivor. “Making a game alone or with a few people” is something that is hard because they are trying to make traditional games and match the production of AAA as much as possible. So much of what I’m talking about in this article is trying to think about ways of making games where doing it with one or two people is “easy” or like “obvious”. Thinking back even to something like Myst, which was made by ~5 people, which was “big” at the time, but still went to be an all time best seller. I don’t want to appluad the efforts of a small team busting their ass just to compete — I want to applaud the efforts of two people that punched above their weight class by getting way smarter about what they were making then just trying to “do AAA but with less people”.

Otherwise, “indies stepping in” doesn’t fix the problem. Jedi Survivor isn’t the problem, it’s the “the type of game that Jedi Survivors is is both what players want while also becoming increasingly untennable to produce.” Being “indie” doesn’t solve that, it just means you’ll try and do the same thing with less.

More Collapse:

Someone responded to me on Twitter saying I didn’t cite sources enough for “collapse”. I think that’s fair. So I’ll do some of that now.

One thing I really want to get across is that “gamedev being really big” is both true and false. Games do pull in a lot of revenue, but by the numbers sort of obscures the broader ecosystem. Namely, that, on average, games make less money than they did a few years ago, but that hits hit harder. This article goes really deep into this topic.

I think it’s a bit up to what you think is a “healthy dev ecosystem” to determine if this is an issue or not. If you think it’s okay for one game to suck up all the revenue in the world, then this is probably fine for you. But if you want a plurality of titles to be available, this is pretty grim (the average title makes less than $2K). Here’s some more stuff one numbers and (bad) median sales.

Paolo Perdicini has a similar perspective to me (I think) and frames the issue as looking for new markets (as ai tooling wasn’t around at the time). We both agree that things as is are really saturated and new horizons for games are needed.

All that said, there’s no smoking gun of “GAME DEV IS COLLAPSING!” anywhere, for various reasons. I’m more looking at the smoke signals and going from there.

---

Footnotes

1 It is really interesting to consider where indie would be now if its breakout era wasn’t so set on remediating the past. Could it have instead been a truly alternate stream?

2 There are obvious exceptions to this, I’m mostly just talking about zeitgeist-y aesthetics

3 Want some predictions? I think we’ll start seeing a lot more “indie MMORPGs” soon, in part due to cloud services having a more accessible SaaS model.

4 One of my predictions for where AAA goes is to make high-profile games for streamers only. Companies bid streamers to create bespoke games that only they have access to, ensuring a revenue stream for studios and maintain novelty for streamers.

---

Published on April 26, 2023.

Tagged: Game Development   AI Art   GPT