PReLU Activation Function update rule

Question

I just finished reading Delving Deep into Rectifiers paper. This paper proposes a new activation function called PReLU. Maybe it is obvious, because the paper did not mention it, but I want to know when is the parameter 'a' of PReLU updated? Is it updated before weight update or after weight update? or is it simultaneously updated with weight?

Answer 1

The weights are all updated sequentially as the error signal propagates back through each layer of the network. So the bias, and 'a' parameter both update before passing the signal to the next layer and so before the the weight update below them in the network.