Hyperledger 2.0: reconciliation error when trying to fetch missing items from different peers: Empty membership

Question

I have 3 Orgs with currently 1 peer per org running and one Orderer. I have a private data collection defined for 2 orgs.

   "name": "privateOrg1-2",
   "policy": "OR('Org1MSP.member','Org2MSP.member')",
   "requiredPeerCount": 0,
   "maxPeerCount": 3,
   "blockToLive": 30000,
   "memberOnlyRead": true

However, when I add data as member of Org1, these data are not synced with Org2. When I add data for Org2, these data are not synced with Org1. The following errors are seen in logs:

2020-05-11 15:30:28.137 UTC [gossip.privdata] fetchPrivateData -> WARN 7a0a Do not know any peer in the channel( data-channel ) that matches the policies , aborting
2020-05-11 15:30:28.137 UTC [gossip.privdata] reconcile -> ERRO 7a0b reconciliation error when trying to fetch missing items from different peers: Empty membership
2020-05-11 15:30:28.137 UTC [gossip.privdata] run -> ERRO 7a0c Failed to reconcile missing private info, error:  Empty membership

Non-private data is synced without problems.

What could be the problem?

Answer 1

I fixed the issue. But I believe the way everything works is not optimal.

I simulate a distributed network. Each peer runs on a separate machine. All the docker containers run standalone and not as a part of kubernetes or docker network.

I did the following steps:

1. update the config of each peer with the Org1MSPanchors.tx, Org2MSPanchors.tx, Org3MSPanchors.tx generated by configtxgen. Earlier I didn't do that.

2. I analyzed the logs and found out that each peer tries to connect to an anchor peer of another org directly. This connection failed. To make it work, I added the anchor peers of all the orgs to extra_hosts of my peer docker_compose files.

Initially I thought that the ordering service knows each anchor peer and peers of Org1 should get IP adresses of Org2 peers from the ordering service. This was a little bit naive.