Iterative game dev

Update: newest addition to article series is State of gaming.

Its now June 2021. I have written a few articles on iterative game dev in the past years. And I created a few well visited video’s on Youtube on how to automate your game build and deployment. Most of this is from a point of view of a software developer, versed in Agile development, taking a look at game dev.

However with me doing more and more game dev myself and actively professionalizing my craft, I got a few new insights. These are different from my original articles. Here a summary of my changes in vision:

• All game studio’s do use iterative development already, however depending on their target markets they have different approaches on how and when to iterate.
• Iterative development, CI/CD, Agile: none of these are silver bullets. There are consequences and costs associated with these approaches, and it is these that withhold the industry as a whole to make a switch.
• A lot that goes wrong in game dev has to do with changing the requirements/scope too much during the development phase. This is a known issue, its part of the deal when you want to make creative products for a quickly changing market. No amount of iterative development is going to fix this.
• There is much to win with proper testing strategies. However, this is not game dev related as software in general has trouble adapting ‘making all things testable’.

Lets start with looking at what the current practices are in game dev.

Reality

On reddit I have seen a few post of game devs from AAA studio’s (like Ubisoft) telling they utilize sprints, build stuff incremental, and emphasize modular patterns with automated testing. Can’t recall the posts, but they are from 2020-2021.

On my visit of the Amsterdam White Nights conference in 2020 I realized that iterative development is and always has been part of game development. The early phase of a game is about prototyping, and designers make an art of how to come up with good designs. Fawzi Mesmar from EA Dice did the talk ‘Effective prototyping’ on this.

And in mobile there is this frantic focus on iterative development and data driven decisions. I think that even in business software development nobody takes this stuff as serious as the big mobile development studio’s do. ‘Defining the MVP’ was a talk from Tatiana from Etermax on how to properly launch games on the market through iterative development. How to manage these iterations, call it quit, etc.

This all is backed up by video’s you can find on Youtube. I have gathered a few in my articles, for instance in this blog. They are a bit dated (2019) but still very relevant.

This brings me to a recent video that triggered me to write down my current vision on iterative development in gamedev. A major part of next gen business software development is the use of CI/CD to enable quality and speed with iterative development.

Focus on continues integration and/or deployment

Take for instance this video on CI/CD from a guy I respect. He is totally right, but something stings me.

A lot of the stuff this video discusses is known for years already. See for instance the book ‘Agile game development with Scrum’ from Clinton Keith (2010). He accepts the reality of the industry and gives practical advice. To my observation, the trick he presents is a different approach to production in general. And yes CI/CD can then be adopted, but that’s then a natural consequence of the different attitude of higher management / directors.

Even CI/CD can’t change the reality of game development: not knowing what to aim for (floating target to aim for), having to work in secrecy for a long time because thats how the industry works (so again no proper feedback), heavy investment needs (big risks), changing demands from the audience (again floating target), massive teams (high cost of communications) and massive (and I mean MASSIVE) structures of code+scripts+assets (high cost of change).
See my article on failing early access experiments in 2019 for more on this reality.

And to be honest, the ‘integration’ part is something all game studio’s naturally understand the benefits from. You don’t have to tell them that. Most studio’s even attribute a lot of their market value to this whole ‘tooling’ and ‘asset pipeline’ they have. These pipelines are ranging from ‘helping streamlining the process’ to a real continues integration where hundreds of employees can live change code+assets and see their contribution come to life immediately. You need this if you want to efficiency create games.

With this ‘silver bullet’ thing out of the way, the video addresses a good point: continues quality assessment is problematic in games. That is because testing is hard in games.

Testing

To manage complexity and changing scope, you need to be able to rely on your game to keep working according to spec. And gauging from what I see continues automated testing of all parts of the game is still not that big of a deal yet.

I myself work with Unity 3D and yes the code is unit test-able, but there is so much more than code that makes the game. Games are layers of mechanics, scripts and code. Unity 3D doesn’t lend itself for writing good tests on all the ‘live’ MonoBehaviour stuff. Yes they introduced some helpers to easier set up test scenes, but that is only a start. For proper test coverage, you need much quicker and better maintainable ways to assert the code is still within specs.

Again, this CI/CD guy delivered an interesting video on the certain types of tests you want to have. I do feel it needs a bit of adjusting to the requirements of game devs: not all target markets favor A/B like testing stuff (for instance you can’t introduce a new unit in an RTS on live servers, you want this to go to specific test servers only) and not all markets like having stuff live all the time (they explicitly want big bang releases for the hype train to work).

But then again, delivering quality games is something very much valued by the industry. There is a lot of emphasis on QA already in games. But naturally the biggest bunch of it takes only place late in production because only at that stage enough of the game is finished to judge it as a whole.

I myself do understand this issue of not being able to write test to ‘value’ the perceived quality that my audience will attribute to my game. However these are just 1 types of tests. A lot of the tests types addressed in the previous video are very technical of nature. And to be honest, a lot of the bugs we all saw in Cyberpunk where also of technical nature only…

So yeah, more automated testing on specs in game dev.

One note here: a lot of business I visit as a software developer don’t employ ‘proper’ testing methods too. Its hard to do right as in: get better quality and higher release speed at the same time. A lot of them struggle with setting up test and instead create things that slow development down over time.

With the apparent issues out of the way and the delicious low hanging fruit in our baskets, I now want to look at iterative development again to see if there are still a few things left to improve. Lets start with where it always starts, the prototypes.

Prototyping phase

When I look at prototyping, I see 2 distinct approaches:

• Do all prototyping only in-house or on specific test groups. Selection is by group vote and major production members. Even when moving forward with a prototype, still keep things in-house or in these test groups.
  Effects: 1) nobody knows what you are up to, 2) and the judgement quality depends on the in-house selection mechanisms and representation of the test groups, 3) only high quality gets released eventually.
• Do in-house prototyping but encourage live tests as soon as possible. When moving forward with a prototype, be data driven while constantly measure audience reception. Selection is mostly data driven with targets set by major production members.
  Effects: 1) players see an evolving game over time, 2) and the judgement quality depends on in-house selection combined with live data but limited by the reception of the early adopters, 3) quality varies over time in a product and between products.

To make a quick link back to the CI/CD video: this CI/CD stuff works very nicely with the second approach. And the second approach naturally aligns much better with how the mobile market works. But due to the side effects, especially effect 3, AAA games like Cyberpunk don’t want a continues live release. So these games naturally use the first approach.

It is a nasty thing however that the first approach has a big downside: somewhere during production you have to commit to the end goal you are going for. This is where things start to go wrong. You can’t alter course after you did this without massive costs. But what to do if you later find out that the game isn’t as fun as you wanted it to be but already committed? And are a few years in already?

Its an industry secret that every game does experience this. Always somewhere half way. This just seems to be how it works when you are making a creative work and optimize for ‘fun’.

It gets in a sh*tshow when you do this too often, promise too much, keep your release dates fixed and create toxic work environments. Something has to give.

I do think there is still place for this ‘committing to a prototype, not going live until it is finished’. The benefit of a hype train is commercially too big of a thing to ignore. But it should be done carefully with constant high technical quality during each phase of development, no cutting of the corners early or late production.

That is because this second prototyping style ‘live testing’ is also not a holy grail. Games produced this way vary greatly in quality and consistency. And are often way too optimized for ‘just enough fun’ instead of ‘maximal fun, maybe less revenue’. And another downside is the much shorter shelf-life of these games with half finished games and a lot of promises still left in the open or even killed and taken offline, when the team decides its not worth the effort anymore.

Wrapping things up

Again, the summary:

• All game studio’s do use iterative development already, however depending on their target markets they have different approaches on how and when to iterate.
• Iterative development, CI/CD, Agile: none of these are silver bullets. There are consequences and costs associated with these approaches, and it is these that withhold the industry as a whole to make a switch.
• A lot that goes wrong in game dev has to do with changing the requirements/scope too much during the development phase. This is a known issue, its part of the deal when you want to make creative products for a quickly changing market. No amount of iterative development is going to fix this.
• There is much to win with proper testing strategies. However, this is not game dev related as software in general has trouble adapting ‘making all things testable’.

If you like what you are seeing here and want to start using some CI/CD of your own, I got you covered.

I created a full walk-through on making a basic CI/CD pipeline for Unity 3D builds with Jenkins and Azure DevOps. (Although without the ‘proper’ testing part hehe, even for me hard to get right lol.)