Tuesday, 20 June 2017

Gaps of the Imagination

We don't perceive every single piece of information around us. Instead we must constantly fill in blanks in our knowledge to properly create a mental image of the world. This processes of filling in gaps is really important to understand and to exploit when crafting games.

At the very core, games are driven by systems. We have extremely powerful machines that can quite accurately simulate various forms of physics, economics, group dynamics, intelligence and so on. This means that at our fingertips are the tools to create entire worlds. It is therefore very tempting to use these in order to solve every single problem. But this is not always the best approach. Sometimes it's best to just leave gaps, and let the player's imagination handle the rest.

There are five core reasons why this is so:

  • We are all unique individuals with different needs and likes. By leaving things unsaid, players can personalize their experience without any asset requirements from the creator. A typical example is that players can envision the sort of monster they personally are the most afraid of.
  • It is impossible to simulate things on an atomic level; there always needs to be a cut-off at some level. Since highly complicated things tend be unstable and encourage scrutiny, it is often better to make the cut off at a higher level.
  • There are certain things which are, for various reason, not possible to implement properly. In this case a lot can be gained by simply leaving it to the imagination. This includes everything from creating immense battle scenes to conveying a particular emotional state.
  • For the sake of pacing; sometimes it's simply not possible to give the player all of the required information and it's better for them to just work it out for themselves. This could be things such as background story, details on a map, or the areas that it's possible to visit in a location.
  • There is an inherent enjoyment in filling out blanks in various forms of art. The reasons are a bit fuzzy, but you can clearly see it in all art. It's fun to wonder what a character is really feeling, what events lead to a scene, and to mentally picture what a shadow is hinting at.
All of this works because of how our brains is constructed. We are all equipped with highly capable pattern-matching machines. Our brains, and therefore we ourselves, want the world to make sense and will always try and fit things into a coherent and convincing narrative. The simplest example of this at work is motion.  Just consider this animation:

This is really just a series of circles that we see in rapid succession. Despite the fact that it's quite easy to notice the individual frames, we still see it as a ball jumping up and down. We fill in the gaps between the frames and see it all as an animation. Notice that this doesn't really take any conscious effort - it comes naturally. In fact, it is quite hard (possibly impossible) to not see this motion. When done right, gaps of the imagination never really feel like such - they simply get incorporated into the overall experience.

Here is another classic example:

There is no white triangle pointing downwards, but the other shapes suggest that there should be, and hence we see one. Again, this is something that we carry out automatically and while we can rationally understand that there really is no triangle, we cannot mentally unsee it.

Closing gaps like this is not just a fun side effect. It is a crucial way in which our brains function. Taking decisions on incomplete information is an essential skill in the function of our day to day lives. In fact, most of the information we receive is incomplete - it's quite rare that we are directly exposed to situations where we have all possible information to hand. So it's not so strange that closing gaps should come so naturally to us, we literally wouldn't be able to survive otherwise.

As I have said before, any decisions we make are based around a mental model. In our heads is a mental simulation of how we think the world works, and before we do anything we first run through the action in our model to see if it would have the effect we intend. However, it is not possible to get complete information about our surroundings. So in order to get a working map of the terrain we need to fill some of the information. That is what the gaps in imagination are all about - our process of crafting a working mental model.

Given that this is a foundational part of how people work, it is essential that we always have it in mind when designing games. After all, the end result of a game is really to create a specific mental model in the player. This can be done through direct system access, sensory input or, what this post is all about, through suggestions of something unseen.  I will now go over the most common, and most important, ways in which gaps can be used.


Sensory Gap
This is the most common one and also easiest to understand. When we are presented with sensory information that is somehow incomplete, we try and fit it into a working context. There are many different ways in which to do this. It can be quick, disjointed shots of something, events seen through the eyes of a monster, a character reacting to a smell. or a sound that is so loud it shatters glass. Normally you build this sort of gap up by presenting a negative space where one or more object/events/reactions/etc. all related to something unseen. 

We tend to use this sort of gap more than we realise, simply because it is such an ingrained way of working with various artforms. But that just means you have to be extra aware of how it works. For instance, it's good to know that we are extremely prone to seeing familiar shapes, such as faces, in just about anything. 

It's actually quite hard to not see anything in an image. We have evolved in an environment where there was nothing like pure noise, everything we saw was part of the environment somehow. Therefore, when we see random patterns the brain goes into overdrive to find a connection. Of course, it is not just the direct information received that we use to do this, but anything related is accessed as well. Because of this, you can often get away with very little if the scene is just set up in the right manner. Learning to do this is incredibly important.

Informational Gap
This is very similar to the sensory gap, but instead of sensory data it deals with information in the abstract. Jeff Vogel recently wrote a blog post on how games often have too many words [1], and properly using gaps is the best remedy for that. It is often easy to underestimate just how much information a player is prepared to fill out for themselves. You might think that a player really needs to know details of everyday routines in order to fully understand how a certain society functions. But often it is just as good to simply have some brief glimpses (visually or through text) of what they are doing along with something that sets the tone (music, animation, etc). 

This sort of advice might seem a bit obvious, but it is worth keeping in mind how good we humans are at filling out these sort of gaps in information, especially if it comes in form of gossip. Gossip is something that has probably been part of our lives since before we evolved into modern humans. We are social animals and gossip serves as a tool to keep track of the group we are living in. This means our brains are extra-sensitive to this sort of information, similar to seeing shapes and motion, and can easily assess even the smallest hints.

Spatial Gap
Games are often very spatial and requires the player to navigate various places. Because of this it is often very tempting to accurately represent every single area to give a proper sense of place. However, for various reasons this might not be feasible or required. It is then important to note that spaces can also be created from gaps, just as much as sensory information and lore. Spatial gaps are a bit trickier though as we are now messing with the actual play area which means you you have to be extra careful.

The sloppy way to implement is to simply have a lot of locked doors that the player cannot access. But this can be bad for pacing and, worse, it can lead to a weak mental model. The brain is usually quite lazy and will try to optimize. If the pattern becomes 'locked door equals the room is irrelevant', it will be noticed and incorporated into the model. After all, dismissing these spaces is much easier than assuming they are there.

Instead it is better to vary the ways in which the rooms are blocked off, and to build up a negative space hinting that it is there, such as glimpses into it from a small window. It is especially effective to make the player believe that room might contain something of value. That way it is much more likely it will be part of the mental model. Remember, what matters is not what is actually there, but what the player perceives to be there.

Causal Gap
This next time is very similar to the animation example from above, but happens over longer time scales. As I said before, closing gaps is all about creating a working mental model. Where the previous gaps where are all about what we see, causal gaps is more about why we see it. Just like we humans are quick to see familiar objects in random patterns, so are we quick to correlate two events with one another. A black cat runs over the street and then you get a headache. We humans often correlate the most strange occurrences and it is quite hard not to. Many of our psychological fallacies are really based around this fact, and it takes effort to learn not to do this. Why is this so? Because for our ancestors it was okay to have a few weird beliefs if it meant we could pick up on dangerous situations and survive better. If you are a skeptic about the cat causing the headache, you might also be skeptical towards the signals of an impending avalanche. 

This is obviously the sort of brain glitch that we can make use of in art. Film is a medium that uses it all the time, in editing. See a gun fire, and then a person being hit, and it is obvious what has happened. Correlating these two events into a single event comes naturally to us. But because games are so driven by the player understanding the flow of input and output, or how the player's actions correlate to events in the game world, it gets a bit harder to deal with gaps.

The simplest way to do it is with events taking place. For instance, say the player suddenly notices that a door is blocked, and hears the sound of feet running away. They will now assume that whoever is running away is also the one who blocked the door. 

A little harder is for us to provoke imagined causal connections from actual interaction. What I mean by this is that you actually leave out certain information when actions take place, and ask the player to fill in the middle. For instance, if I click on an object in a game and it simply pops into my inventory we would like the player to mentally simulate this as "I picked up the item" and not "the item got magically teleported into my inventory". The three key elements for achieving this are consistency, negative space and optimization-avoidance.

Consistency means that we need it to happen in a way that makes it possible to distil it into a couple of universal rules. For instance, is the distance you can pick up items at consistent with the player using their hands? Negative space means that we use other events to reinforce the fantasy. These can be sounds effects when you pick up, feedback messages when you are too far away, and cutscenes where we see this happening for real. Finally, optimization-avoidance means that our intended mental model, that of the player actually picking these objects up, must be the "simplest" [2] available explanation. If there are too many edge cases, weird behaviors or simply not enough supportive negative space the "magical teleportation"-theory will win.

Anticipatory Gap
This one is quite similar to causal, but worth having in its own category. These are basically events that might happen to the player in the future and that they will take into account when planning. As explained in an earlier blog, planning is a core reason why gameplay is engaging, and thus it is important to shape what sort of consequences the player can conceive of taking place. When the player plays a game, they will generate their mental models not just based on what they are playing, but also what they know from before. This is gives us an opportunity to make the player think certain things are possible, without them never having witnessed them.

Suppose that the player hears footsteps from afar. These might just be coming from a couple of scripted sound effects, but the player doesn't know that. Given the right context the player will conjure up a monster that is making these sounds and assume that it might attack them. Now the player will start making plans based on a few sound effects and project a lot more onto the game than what is actually there. This is something that we saw a lot in Amnesia. The player could create long, and engaging, gameplay situations for themselves only because of a few sound effects. 

Of course, this sort of trick can't last forever. As I have mentioned several times, the brain likes to optimize and once the pattern gets to clear the illusion will go away. So it is important to constantly update the negative space, to not make the events so predictable and to setup situations in ways that feel exciting to partake in. Even more important is to make sure that this trickery is not the only gameplay there is. If the player has a baseline of actual planning  and execution going on, this sort of illusory anticipation can be sustained for quite a long time and add a lot to the experience.

Agency Gap
If you are in the jungle and the high grass suddenly moves. Do you:
  1. Assume it is a tiger with the intention of eating you?
  2. Be skeptical and consider that it might just be the wind playing tricks?
In our ancient past, the people who thought like number 2 were much less likely to survive. Sure, they escaped the embarrassment of being afraid of grass from time to time. But they also got eaten a lot more by tigers. Thinking of events as being caused by something with agency (e.g. an animal) is powerful concept for survival. It is also something that leads to all sort of weird beliefs like tree spirits and demonic possession. For many events it comes naturally to think of them as caused by intentional beings. And once again, this is great stuff to use in games.

This video that is a great example of the whole thing in action. Just note how you directly project agency on the shapes.

The most common use of this is in enemy AI. Most of the time, AI that feels smart is because the player thinks it is so, not because the underlying systems are complex [3]. F.E.A.R. is a great example of this. While the AI does have some clever systems at play, it derives most of its impressiveness from being good at giving the player feedback. For instance, by saying "cover me" while a grenade is thrown, it gives the impression that much more is happening than what actually occurs. The player projects a set of thoughts running through the soldier's mind and incorporates that into their mental model. But in reality it is just very simple code that is executed.

This is an area where the brain works a lot to our advantage. We are ill-equipped to mentally simulate things like state machines, but it comes naturally to think in terms of people. This means that we can get a lot of content to the player only through suggestion. The player's brain is really apt at simulating people and can do so much better than any existing computer system. So the more of that power we can use, the better.

All of this get harder when we get away from combat though. While players want to think of other characters in games as entities with rich mental lives, there is a limit to how far they are willing to go. If animations start to look weird, if responses come out as canned or if interaction possibilities are too limited the brain gets lazy and optimization kicks in. Characters go from intentional beings to simply being objects.

A big problem of interactive storytelling is how to keep this from happening. Just as I discussed with causal gaps, the key elements in achieving this are consistency, negative space and optimization-avoidance. How exactly to achieve this with characters is still too unclear and complex to cover this time around - I will go over some possible directions in a future blog post.

Volition gaps
Finally we have arrived at the last gap type, which is similar to causal gaps. Previously we have mostly talked about things happening when the the player is the one causing it, but volition gaps contradict that. These are events that the player thinks they cause when in fact they don't.

This is something that I think currently is quite unexplored and has a lot of potential. It is quite hard to construct control interfaces that allows the player to take all the possible actions. Therefore it would be nice if we could have actions that the game does automatically but that the player believes they used their volition to cause. This is what this gap is all about.

Just like we construct a mental model of what is happening on the outside world we are also constructing a narrative of what we are doing ourselves. While we are not conscious about it in everyday life, a lot of the time we have reasons and beliefs because we do certain things, instead of the other way around.

For instance, take choice blindness. In an experiment the subject had to pick which face they thought were the most beautiful, and then when handed back a different picture, most subjects continued to explain why they had chosen it. So a lot of the time, we make up reasons based on the actions we find ourselves doing. And as always, whenever there is a brain deficiency, we should try and exploit it for art.

The best example of this effect in use is in Assassin's Creed. When you run across the roofs in a city, the game will automatically make the character jump. Despite that, it always feels like you are the one who is willing those jumps to happen. This works as a great way of streamlining the controls and making the experience more fluent without taking away a sense of agency in the player. Another example of is in games like Uncharted where the character will interact with the world in ways that make sense and enhance the feeling of being there.

I believe that these sort of gaps can be taken further, though. An interesting example of this is The Path, where the protagonist will carry out actions of their own choice when you leave them alone. It is by no means perfect, and doesn't really provide an illusion of volition. But I think it shows the potential of this and I think causes the player to feel intimately involved with various events if used correctly. Currently, though, it is too unexplored to say for sure what the possibilities really are. I will explore some more thoughts on this in a later blog post.


That should summarize the basics for the various gap types. I am sure there are even more than these, but this selection is what I found most useful for games. In upcoming blogs I will dig deeper into some aspects of this and give more examples on how it affects gameplay.

Until then I recommend reading Ian Thomas's excellent article on the subject where he discusses gaps for LARPs, SOMA, and much more.

1) The follow-up post that looks at the game Pillars of Eternity specifically is also well worth reading:

2) This is simplest in terms of effort used by the brain, not in a strict theoretical sense. For instance "a witch did it" is a simple explanation for a human brain, but in reality is quite complex because it assumes a lot of attributes for the witch. But we as humans are great at just ignoring this, making the silly explanation seem simple.

3) This video shows it off nicely: https://www.youtube.com/watch?v=9bbhJi0NBkk

Thursday, 15 June 2017

Choices, Consequences and the Ability to Plan

This article goes over why it is so important for choices to matter in a game and how it all has to do with planning. If a user perceives that their actions have no consequences, you remove a core component of engagement - the ability to plan.

Say you are playing a game like The Walking Dead, or any other interactive movie, and you are faced with the choice whether or not to help someone who is hurt. You decide that you want to help the person, after which you never see them again for the rest of the game. Reloading a save and playing through the scenario you find out that if you chose not to help, the same thing plays out. Simply put: in this case, your choice really has no consequences.

While the scenario is made up, it presents a very typical situation that opinions are heavily divided on. Some people are totally okay with it for various reasons. But others will argue that this lack of consequences ruins the entire experience, as your choices doesn't really matter. It's really easy to say that people who feel this way are simply playing the game the wrong way or are not properly immersed. However, I think it's really important to investigate this reaction further as it gets us closer to some fundamental problems of narrative games.

The argument from people who get annoyed by these non-choices goes something like this: if every branch leads back to the same path, then you really don't have any say in how the game plays out. You are not playing a game, you are only pretending that you are. It's like when you are playing a split-screen game and notice you've been watching the wrong side. The feeling of play is just an illusion. Nobody would tolerate a Super Mario where a pre-written script - not the player's skill - determines whether or not they survive a jump, so why tolerate games where all choices lead to the same conclusion?

One could counter that by saying the intention is to put you into a hard position and the game is about your varied emotional reactions as you ponder the different choices. It isn't about affecting how the game plays out - it is about making an emotional journey. If you require the game to show you the consequences of your actions, you are not immersed in the game's story - you are simply trying to optimize a system. This might sometimes be the case, but I also think this line of thinking is missing what the actual problem is: the failure of the player's mental model.


Let's start by breaking down the problem. A mental model, as explained in this previous post, is how the player perceives the game's world and their role in it. As you are playing a game, you slowly build a mental model of the various objects and systems that make up the game and attach various attributes to them. At first a box might just be a piece of the background, but as you learn you can destroy it in order to gain items, attributes are added. The object gains complexity. The reverse can also happen. For instance, when you first see a character you might think that you are able to speak to it and therefore label it with various attributes you know that humans usually have. But when you find out that the character is really just a piece of the background without any sort of agency, most of those attributes are lost.

Your mental model of a game is something that is continually revised as you are playing, and it is something that always happens, no matter what the game is. In fact, this is a process that is a core part of any medium, including books and films. So, obviously, when you are playing an interactive movie game, you are not simply reacting to a direct stream of information. You are answering questions based on your mental model.

Take my "will you help your hurt companion?" scenario from above. The knowledge you take into account about that choice is not just what is currently projected at you from the TV screen. It is a combination of everything you have gone through up to this point, along with a bunch of personal knowledge and biases. Even basic concepts like "hurt" and "companion" aren't just created in this moment. They are ideas that the game has spent a lot of time building up, be that for good or bad, from the very moment you started playing.

When you are faced with the hypothetical scene of  a hurt companion, you are not just dealing with an animated image on a screen. You are dealing with a whole world constructed in your mind. This is what your choice will be based around. While it might objectively seem that everyone is reacting to the same scenario, they may in fact be dealing with quite different setups.

So when someone gets annoyed by the lack of consequences, it is not necessarily the direct consequences that are missing. The issue is that they have constructed a mental around a real person in need, along with that person's future actions. So when it becomes apparent that the game doesn't simulate that as part of its own model, the player's mental model is broken and it feels like a big let down. Remember that we don't play the game that is on the screen, we the play game as we perceive it in our heads. So when it turns out that your imagined world is fake, it has a huge impact.

It gets even worse once we take into the fact that planning is fundamental to a sense of gameplay. As explained in a previous post, engaging gameplay is largely fueled by the ability to make plans. The way this works is that the player first simulates a course of action using their mental model, and then tries to execute that in the game. This is a continuous process and "planning and executing the plan" is basically the same as playing. Interactive movies normally don't have a lot of gameplay and it is really only in the choice moments that the player gets to take part in any actual play. Hence, when the choices turn out to have no consequences, it becomes clear that planning is impossible. In turn, this means that any meaningful play is impossible and the experience feels fundamentally broken.

As an example, take this experience I had with Heavy Rain:
[...] one scene I had made a plan of actions: to first bandage an unconscious person and then to poke around in his stuff. There really was nothing hindering me from doing so but instead the game removed my ability to interact directly after caring for the person. The game interpreted me wanting to help the guy as I also did not want to poke around, thinking that they two were mutually exclusive actions. Of course I thought otherwise and considered it no problem at all to do some poking afterward.
I think that people to complain the loudest about the lack of consequences are extra sensitive to situations like this. But, as I said, this is not due to lack of consequences per se, but due to the impact it has on the consistency of their mental model and sense of play. It is really important to note that this is not due to some sort of lack in immersion or ability to roleplay. On the contrary, as I have described above, many of the issues arise because they mentally simulate the game's world and characters very vividly.


So the problem that we are faced with is really not a lack of consequences. It is because the underlying systems of the game are not able to simulate the mental model for a subset of players. One way of mending this is of course to add more consequences, but that is not a sustainable solution. Additional branches increase exponentially, and it quickly becomes impossible to cover every single possible outcome. Instead it is much better to focus on crafting more robust mental models. Sure, this might entail adding consequences to choices, but that is just a possible solution - it is not the end goal.

As I outlined in the previous blog on the SSM framework it is incredibly important to keep track of how systems and story help form a mental model in the player's mind. For instance, if you start your game saying "your actions will have consequences", that will immediately start filling up your player's imagination with all sort of ideas and concepts. Even how pre-release PR is presented can affect this. All of these then become things that lay groundwork for how the game is modeled in the player's head and it is vitally important to make sure this mental model remains stable over the course of the game.

One of the main things to have in mind is consistency. Remember that as someone is playing a game, they are building up a mental simulation for how things are supposed to work. If you provide information that certain events are possible when they are in fact not, you are running the risk of breaking the player's mental model. You either need to remove this sort of information or to make sure that they never take part in situations where these sort of events feel like a valid option.

However, the most important thing to keep in mind is the ability to plan. A major reason why the lack of consequences can feel so bad is because these consequences were part of the player's gameplay plans. So when it becomes apparent that they don't exist, the whole concept of play breaks down. In all fairness, this might be OK for certain genres. If the goal is to simply to make an interactive movie, then losing a subset of player might be fair. But if the goal is to make proper interactive storytelling, then this is of paramount importance - planning must be part of the core experience.

That doesn't mean that every choice is something the player needs to base their plans on. But in that case then there need to be other things that lie on a similar time scale and which are possible to predict and incorporate into plans. I think that one way around this problem is to have a more system-focused feature that runs alongside the more fuzzy narrative choices. When the players make choices, their mental model will have the best predictive skills around this more abstract system, and play revolves mostly around this. Then when more narrative choices are presented they will feel more game-like and part of the a solid simulation, despite not really having any consequences.

A simple and good example is the choices you have to make in Papers, Please. This game is driven by a type of survival simulation where you need to gain credits (though doing proper passport check) in order to keep your family live. Entwined into this are choices about who you will allow into the country. Many of these don't have any far reaching consequences, but that that doesn't really matter because your ability to plan is still satisfied. But despite that, these choices still feel interesting and can have an emotional effect.

 This sort of approach relies on combining several elements in order to produce the feeling of something that might not actually be there. This is something that is used in a wide range of applications, from how we view images on a TV, to how films can create drama through cuts. We don't always have to have solve problems straight on, but often the best way is to split the problem into many and to solve each problem on its own. The combined effect will then seem like a solution to the original problem. This is a technique that is super important for not just this, but many other narrative problems. I will write a blog post later on that goes into more details.

Once you have a game that is consistent and that has some sort of planning apart from the more narrative choices, the probability of satisfying the people will be greatly improved. And not only that, your narrative experience will improve over all, for all players, not just a subset. In this case I think it is fair to view these extra sensitive people as canaries in a cave, something that is first to react on a much bigger issue.


This blog post by no means presents the solution to end all problems with choices and consequences. But hopefully it will give a new way of thinking about the problem and some basic directions for finding a solution. I don't think we will ever find a perfect way of dealing with choices, but the better informed we are at underlying causes, the better experiences we can provide.