Cannot update fabric channel config using new admin identity

Question

Background

We have a production fabric cluster setup and has been been running for a year. Now most of the certs expire and the cluster crash, including both tls and identity certs.

I tried to fix by completely removing old certs and private keys, generate and enroll new identities for peer, peer admin, orderer, orderer admin.

Everything works again, but I cannot instantiate/upgrade chaincode in existing channel because the channel was configured with old admin certs.

Problem

So now look like I'm stuck in a deadlock. In order to update channel config with new cert, I need to sign the update with matching old cert, which is already expired and blocked by orderer.

I find out that we can disable expired cert check in orderer using ORDERER_GENERAL_AUTHENTICATION_NOEXPIRATIONCHECKS=true. But now I don't have the old admin private key so I still cannot update the channel config.

Questions

I already replaced old private keys with new one so there is no way to use the old cert again. Can I do something to resolve this channel issue?

Suggestions are greatly appreciated.

Answer 1

• [!] What I'm suggesting is an idea. I haven't tested it.
• [!] It seems to be feasible enough, but side-effect is not considered.
• [!] It's just a trick, it's correct that it should never be done.

The conclusion is that the orderer and peer's binary can be artificially manipulated and updated.

0. For fabric, refer to $GOROOT/src/crypto when building binary.
1. Build in the fabric repository after artificially modifying all ecdsa verify functions in crypto to return true immediately.

cd $GOROOT/src/crypto
vi ecdsa/ecdsa.go # modify `Verify` function
cd $GOPATH/src/github.com/hyperledger/fabric
make peer
make orderer

2. Back up the binaries of the currently running docker container, and rerun after planting the newly built binaries in the container.

docker cp <peer_container_name>:/usr/local/bin/peer ./
docker cp $GOPATH/src/github.com/hyperledger/fabric/build/bin/peer <peer_container_name>:/usr/local/bin/peer

docker cp <orderer_container_name>:/usr/local/bin/orderer ./
docker cp $GOPATH/src/github.com/hyperledger/fabric/build/bin/orderer <orderer_container_name>:/usr/local/bin/orderer

docker-compose -f <your_docker_compose_file_path> restart

3. Now all verify is valid unconditionally. so, update all recent status.

4. Afterwards, the backed up binary is replanted into the container to solve this problem.

docker cp ./peer <peer_container_name>:/usr/local/bin/peer

docker cp ./orderer <orderer_container_name>:/usr/local/bin/orderer

docker-compose -f <your_docker_compose_file_path> restart