From Pixels to Symbols

- January 25, 2020

Simulate a Game - Like a Brain Models the World

I talked about my plans to simulate games in a previous post, Game Simulation. I’d like to clarify that I’m using the term “simulate” to be analogous to what the brain is doing when modeling the world around us. The brain has constant input from the world, and that input is used to keep our internal model consistent with the world around us. If we were to shut off all of our sensory input, the model in our heads would start to diverge from reality.

In the same way, my simulations of video games are intended to have constant pixel input from the game. Consider a sidescrolling game like Super Mario World. There might be many things happening off-screen. It would be great to have simulated knowledge of the entire level structure, as well as off-screen objects, but that doesn’t come easily to our brains. It’s also completely impossible if we are playing a level we’ve never seen before. Even when we do have off-screen knowledge, it’s usually a fuzzy representation and prone to diverging from reality. The real computation is spent keeping our mental model up to date with reality so that we can reason about the current state of things, solve problems, take actions, and learn new concepts.

Model Structure

Right now, I’m focussed on constructing the simulation model manually, starting with just pixel data and gradually building up the components of the game. I took a step back to think about how the model should be composed and operate in terms of a final solution, and here’s where I’ve landed. (Building this data up from scratch is a different problem that I’m not going to discuss in this post.)

1. Objects: Detect discrete objects, positions, orientations, and sizes from the pixel data with an assumption of spacial consistency from frame to frame. Predict the next state of the object using the Behaviors, Interactions, and Hidden States listed below.
2. Behaviors: Detect discrete object behaviors across frames (like movement arcs, velocities, image sequences, etc) with an assumption that a single behavior conforms to a single linear function. Predict the next object behaviors using the Interactions and Hidden States listed below.
3. Hidden States: Detect object hidden states from the set of current object behaviors. We can also retroactively detect object hidden states from mis-predicted Interactions listed below. Predict the next object hidden states using the Interactions listed below.
4. Interactions: Detect that an interaction has taken place when object Behaviors change. Interactions typically involve preconditional rules between objects and states and result in postconditional changes to objects and states. Predict the next object interactions by finding interactions which fully satisfy preconditional rules.

Note how each component of the model needs a way to detect the state of that component using the states of the other components and ultimately the latest pixel input data from the game. Each component also needs to be able to predict the next state of that component. Otherwise, it will be difficult to evaluate the model and the model will be unusable for search and action planning.

I’m sure there will be complications in every single area when constructing the model. I can predict some of them, but many more might only be discovered when I try to build it. It’s also possible that I’m missing one or more key ideas and that my entire strategy might change as a result. Whatever solution I finally land on, my hope is that creating a model of a game will become a straightforward process which can be applied to many types of games.