Gradient steps aren't always ideal network weight updates; sometimes they interfere across tasks, for example. Directional derivatives are more general, can avoid interference, and can provide other benefits, as we show in work linked below.
Additional basic results are that, for random episodic task sets, Hebbian rules and correlated noise can improve performance substantially (over gradient descent), and in compositional settings they can route learning w.r.t. representational structure.