A visual explanation of reinforcement learning

Learn more

Thanks for reading chapter two of Myriad, a series exploring how minds emerge from matter. Myriad is a collaboration between Arkarup Banerjee, a neuroscientist who studies behavior, and Aatish Bhatia, a physicist turned science journalist.

How the spark moves

The sparks in this article are inspired by the work of the neuroscientist Valentino Braitenberg. We implemented their neural circuitry as follows:

vL=σ(ws bL+wc bR+d)vR=σ(wc bL+ws bR+d)\begin{aligned}v_L &= \sigma(\textcolor{#7ee081}{w_s}\, b_L + \textcolor{#c49bff}{w_c}\, b_R + \textcolor{#ff8fb8}{d})\\ v_R &= \sigma(\textcolor{#c49bff}{w_c}\, b_L + \textcolor{#7ee081}{w_s}\, b_R + \textcolor{#ff8fb8}{d})\end{aligned}vL​vR​​=σ(ws​bL​+wc​bR​+d)=σ(wc​bL​+ws​bR​+d)​

where vLv_LvL​ and vRv_RvR​ are the left and right flipper speeds, bLb_LbL​ and bRb_RbR​ are the brightness levels at the left and right eyes, ws\textcolor{#7ee081}{w_s}ws​ and wc\textcolor{#c49bff}{w_c}wc​ are the straight and crossed connection strengths between the eyes and the flippers, and d\textcolor{#ff8fb8}{d}d is a drive that sets the spark’s speed when there is no light. σ(x)\sigma(x)σ(x) is the ReLU function. The spark’s forward speed is the average of the two flipper speeds, and its turning speed is proportional to their difference.

How the spark learns

There are three parameters in our simplified setup: ws,\textcolor{#7ee081}{w_s}\textrm{,}ws​, wc\textcolor{#c49bff}{w_c}wc​ and d.\textcolor{#ff8fb8}{d}\textrm{.}d. The spark learns values for these parameters using a version of the REINFORCE algorithm published by Ronald Williams in 1992.

Each trial, the parameters were wiggled by a small random amount and the spark moved for 500 time steps, starting from a random point on the edge of the arena. Its reward was the average brightness along its path.

This reward was then compared to a running average of recent rewards. If the spark did better than usual, its parameters were nudged in the direction of this random change. If it did worse, its parameters were nudged in the opposite direction. The size of the nudge was proportional to how much better (or worse) it did:

ΔW(R‾)change=α(R‾)step sizeε(R‾)wiggle(R−R‾)(R‾)improvement\small \underset{\mathclap{\htmlClass{sans}{\text{change}}}}{\Delta W\vphantom{(\overline{R})}}\hspace{0.3em}=\hspace{0.5em}\underset{\mathclap{\htmlClass{sans}{\color{#66b3ff}\text{step size}}}}{\textcolor{#66b3ff}{\alpha}\vphantom{(\overline{R})}}\hspace{2.1em}\underset{\mathclap{\htmlClass{sans}{\color{#ff9b32}\text{wiggle}}}}{\textcolor{#ff9b32}{\varepsilon}\vphantom{(\overline{R})}}\hspace{1.2em}\underset{\mathclap{\htmlClass{sans}{\color{#ffc832}\text{improvement}}}}{\textcolor{#ffc832}{(R - \overline{R})}\vphantom{(\overline{R})}}changeΔW(R)​=step sizeα(R)​wiggleε(R)​improvement(R−R)(R)​

Here ΔW\Delta WΔW is the change in the three parameters, α\textcolor{#66b3ff}{\alpha}α is the learning rate, ε\textcolor{#ff9b32}{\varepsilon}ε is the random wiggle, and R−R‾\textcolor{#ffc832}{R - \overline{R}}R−R is how much better or worse than usual the spark did. We repeated this thousands of times.

Williams’ original approach adds randomness to each neuron’s output. We add it to the weights and biases instead, an approach known as weight perturbation. A similar idea, called evolution strategies, was used by OpenAI in 2017 to teach AI models to play Atari video games.

Sign up below for more rigorously researched visual explanations of neuroscience, and share this article with others who might enjoy it.