Instead of training individually, DeepMind trained a number of agents to play with, and against each other. Further, the matches were played on a procedurally generated maps, that changed from match to match. The agents will have to learn from the ground up on how to see, act, cooperate and compete in unseen environments.
Each of the agents learn their own internal reward signal. This allows each of hem to develop their own goals. The agents operate at two timescales, fast and slow and DeepMind says that this improves their ability to use memory and generate consistent action sequences.
The resulting agent was dubbed For The Win (FTW) agent, and DeepMind notes that it learned to play CTF to a very high standard. DeepMind ran a tournament including 40 human players, in which humans and agents were randomly matched up in games. It notes that FTW agents exceed the win-rate of human players. Further, a survey among the participants rated the agents as being more collaborative than humans.