Kafka Raft Quorum add/remove voter limitations

Question

We are working with 5 kraft 3.5.1 clusters on our company. 3 of them test and 2 of them production. On test clusters our operating team found a problem how the VMs are deployed so they want to redeploy one of our Raft quorum controller member on 2 of our test clusters.

Setup: Lets say, our controllers node.ids are 1 2 and 3 Raft Quorum Voters [1, 2, 3] currently working and we have 6 brokers but they are not relevant.

My understanding to be able to do this was:

1. Deploy a 4th controller VM
2. configure kafka and everything on this VM, use kraft configs.
3. have voters list for this specific controller to include all voters -> [1, 2, 3, 4]
4. format the log dir with these configs using the cluster-id
5. start the controller
6. re-format (?) all the old quorum members voters list configs to include 4th controller
7. rolling restart all
8. then remove the old controller, reformat again, rolling restart again and be done with it.

so after I came to step 5, I was expecting to see the result of

kafka-metadata-quorum.sh --bootstrapserver describe --status

To show [1, 2, 3, 4] as QuorumVoters but instead it shows the 4th controller as an Observer. Now I cant move forward cause the voters list doesn't update. And I tried to update old controllers configs and add the 4th controller there and reformat, but that also doesn't work. You cant reformat a disk space that already formatted.

I know there is no way to add Kraft quorum members dynamically currently. But, is there really no way to do it without losing data as well? I am ok with down time also.

Did anyone succesfully replaced a quorum member when using Kraft mode of kafka?

Thanks in advance.