exploration and exploitation in Q-learning

Question

In Q-learning algorithm, the selection of an action depends on the current state and the values of the Q-matrix. I want to know if these Q-values are updated only during the exploration step or they change also in the exploitation step.

Answer 1

If you read the Q-learning algorithm code, for example from Sutton & Barto book:

It seems pretty clear that Q-values are always updated, independently if the chosen action is exploratory or not.

Notice that line "Choose a from s using policy derived from Q (e.g., epsilon-greedy)" means that the action some times will be exploratory.