I was trying to upgrade 4 nodes in a Cluster. DC1 had nodes N1 and N2 DC2 had nodes N3 and N4
App server is connected to N1 and N2 nodes. RF = {DC1: 2, DC: 2} DC2 nodes where upgraded to 3.0.15 first. There were inconsistency in data in DC2 as was expected as data streaming (Hints) is not possible between different Cassandra versions. While upgrading N2 first in DC1, I noticed that the data was consistent among all nodes immediately while N1 still being older version. How is that possible
On
You have stated that RF=2 and there are 2 nodes in DC1. So N2 has a copy of 100% of the data. When you update N2 to 3.0.x, then all of the data on that node will be visible even though N1 has not been updated.
As an aside, you can upgrade from 2.1 to 3.11 directly if that is your ultimate goal.
It does not matter if one of the node is older Cassandra as a whole remains in sync. It means that even if one of the node is on older version it is syncing with the new version all the time. Logically same data will be present in all the nodes but physcially (sstables) will be different. Data is expected to be consistent among all the nodes during upgrade (old + new ). Once upgrade is completed you need to run upgradesstables to write old sstables to new sstables.