Sensor Prediction is Enough for Emergent Awareness or Understanding of the World

A model trained for next-token prediction is forced to build compressed representations of latent structure in text. Ilya Sutskever correctly refers to this phenomenon as understanding. Here, a model trained for next-step sensor prediction, with a robot that has proprioception and touch sensors but no vision, is forced to build compressed representations of latent structure in the physical world. The robot becomes aware of the shape of external objects. That is, it understands the physical properties of the external world that enable it to make better next-step sensor predictions.

This research was previously done by a diverse team of expert engineers at DeepMind over a month. Remarkably, this reproduction with a completely different robot took only a few hours to implement.