Daniel Scott (@DanielNScott) profile picture
Joined November 2021
Gradient steps aren't always ideal network weight updates; sometimes they interfere across tasks, for example. Directional derivatives are more general, can avoid interference, and can provide other benefits, as we show in work linked below.
1
15
3
41
Additional basic results are that, for random episodic task sets, Hebbian rules and correlated noise can improve performance substantially (over gradient descent), and in compositional settings they can route learning w.r.t. representational structure.
1
2
We elaborate on all this and discuss relations to other neuroscience, deep nets, etc, in our new manuscript: biorxiv.org/content/10.1101/…
1
18