‘[Definitions] of intelligence used throughout the cognitive sciences converge towards the idea that “Intelligence measures an agent’s ability to achieve goals in a wide range of environments”,’ write Anna Salamon and Luke Muehlhauser in Intelligence Explosion: Evidence and Import, a research paper published by the Machine Intelligence Research Institute. ‘We might call this the “optimisation power” concept of intelligence, for it measures an agent’s power to optimise the world according to its preferences across many domains.’
But now, neuroscientists are directly measuring the neurons whose firing rates encode value and produce our choices. We know a lot more about the neuroscience of human motivation than you might think. Now we can peer directly into the black box of human motivation, and begin (dimly) to read our own source code