What does zeta represent in the critic method? I believe it keeps track of the state-action pairs and represents eligibility traces, which are a temporary record of the state-actions, but what exactly does zeta represent and how does it look in c++ (e.g. vector of doubles)?
Critic
Like you already stated, zeta represents the eligibility traces. This can intuitively be understood as containing a "decayed mix of all the state-action feature vectors encountered in all the previous timesteps". It's a trace of things we saw previously, and therefore things we should also give a little bit of credit to for rewards we observe now.
More formally, it's just something that's required if you want to write incremental implementations (with computation time spread out evenly over all of your timesteps) of RL algorithms that, when written in a more straightforward/obvious/naive way, can only be implemented in a non-incremental manner because they have update rules that require information from all timesteps in your episode (e.g. lambda-returns / Monte Carlo returns). That probably sounds rather complicated, it's probably better to stick to the intuitive explanation though.
As for how it would look in C++, yes, pretty much a vector of doubles. The "
z \in R^d" right before the first line of code in the image means exactly that, it's ad-dimensional vector of real numbers (doubles or floats in C++), wheredis the dimensionality of your state-action feature vectors (phi).You can also tell that it has to be a
d-dimensional vector by the fact that it needs to be added to otherd-dimensional vectors (phiandtheta) in a couple of other places in the pseudocode. That can only work out correctly mathematically if zeta itself is also ad-dimensional vector.