Error: No valid responses from any peers. Errors: peer=undefined, status=grpc, message=Endorsement has failed

Question

I m working on hyperledger blockchain and Kubernetes(minikube) on Ubuntu 18-04. In my network there two Organisations with one peer each and the orderer type is Solo. This whole network I deployed on minikube. Chaincode install and instantiation did successfully on pods. After that am trying to invoke by using SDK.

I am using the below code as invoke.js

'use strict';

const { Gateway, Wallets } = require('fabric-network');
const fs = require('fs');
const path = require('path');

async function main() {
    try {
        // load the network configuration
        const ccpPath = path.resolve(__dirname, '..', '..', 'first-network', 'connection1-org1.json');
        let ccp = JSON.parse(fs.readFileSync(ccpPath, 'utf8'));

        // Create a new file system based wallet for managing identities.
        const walletPath = path.join(process.cwd(), 'wallet');
        const wallet = await Wallets.newFileSystemWallet(walletPath);
        console.log(`Wallet path: ${walletPath}`);

        // Check to see if we've already enrolled the user.
        const identity = await wallet.get('user1');
        if (!identity) {
            console.log('An identity for the user "user1" does not exist in the wallet');
            console.log('Run the registerUser.js application before retrying');
            return;
        }
        
        // Create a new gateway for connecting to our peer node.
        const gateway = new Gateway();
         await gateway.connect(ccp, { wallet, identity: 'user1',discovery: { enabled: true, asLocalhost: true }  });
       
        // Get the network (channel) our contract is deployed to.
        const network = await gateway.getNetwork(channelname);
       
        // Get the contract from the network.
        const contract = network.getContract(contractname);
       
        // Submit the specified transaction.
        // createCar transaction - requires 5 argument, ex: ('createCar', 'CAR12', 'Honda', 'Accord', 'Black', 'Tom')
        // changeCarOwner transaction - requires 2 args , ex: ('changeCarOwner', 'CAR10', 'Dave')
        await contract.submitTransaction('arumnet','argument');
       
        console.log('Transaction has been submitted');

        // Disconnect from the gateway.
        await gateway.disconnect();

    } catch (error) {
        console.error(`Failed to submit transaction: ${error}`);
        process.exit(1);
    }
}

main();

configTx.yaml

Organizations:
  - &Orderer
    Name: Orderer
    ID: OrdererMSP
    MSPDir: ./crypto-config/ordererOrganizations/acme.com/msp
    # Policies are mandatory starting 2.x
    Policies: &OrdererPolicies
      Readers:
          Type: Signature
          Rule: "OR('OrdererMSP.member')"
      Writers:
          Type: Signature
          Rule: "OR('OrdererMSP.member')"
      Admins:
          Type: Signature
          # ONLY Admin Role can carry out administration activities
          Rule: "OR('OrdererMSP.admin')"
      Endorsement:
          Type: Signature
          Rule: "OR('OrdererMSP.member')"

  - &Acme
    Name: Acme
    ID: AcmeMSP
    MSPDir: ./crypto-config/peerOrganizations/acme.com/msp
    Policies: &AcmePolicies
      Readers:
          Type: Signature
          # Any member can READ e.g., query
          Rule: "OR('AcmeMSP.member')"
      Writers:
          Type: Signature
          # Any member can WRITE e.g., submit transaction
          Rule: "OR('AcmeMSP.member')"
      Admins:
          Type: Signature
          # Either Acme admin OR Orderer Admin can carry out admin activities
          Rule: "OR('AcmeMSP.admin')"
      Endorsement:
          Type: Signature
          # Any member can act as an endorser
          Rule: "OR('AcmeMSP.member')"
    AnchorPeers:
      - Host: acme-peer-clusterip
        Port: 30751

  - &Budget
    Name: Budget
    ID: BudgetMSP
    MSPDir: ./crypto-config/peerOrganizations/budget.com/msp
    Policies: &BudgetPolicies
      Readers:
          Type: Signature
          # Any member
          Rule: "OR('BudgetMSP.member')"
      Writers:
          Type: Signature
          # Any member
          Rule: "OR('BudgetMSP.member')"
      Admins:
          Type: Signature
          # BOTH Budget Admin AND Orderer Admin needed for admin activities
          Rule: "OR('BudgetMSP.member')"
      Endorsement:
          Type: Signature
          Rule: "OR('BudgetMSP.member')"
    AnchorPeers:
      - Host: budget-peer-clusterip
        Port: 30851

Connection.json

{
    "name": "first-network-acme",
    "version": "1.0.0",
    "client": {
        "organization": "AcmeMSP",
        "connection": {
            "timeout": {
                "peer": {
                    "endorser": "300"
                }
            }
        }
    },
    "organizations": {
        "AcmeMSP": {
            "mspid": "AcmeMSP",
            "peers": [
                
                "peer1.acme.com"
            ],
            "certificateAuthorities": [
               
            ]
        }
    },
  
    "channel":{
        "airlinechannel":{
            "orderers": [
                "orderer.acme.com"     
            ],
            "peers": {
                "peer1.acme.com": {}
            }
        }
    },
    "peers": {
        "peer1.acme.com": {
            "url": "grpc://10.109.214.71:3005",
            "tlsCACerts": {
                "pem": "/crypto-config/peerOrganizations/acme.com/tlsca/tlsca.acme.com-cert.pem"
            },
            "grpcOptions": {
                "ssl-target-name-override": "peer1.acme.com",
                "hostnameOverride": "peer1.acme.com"
            }
        }
    },
    "certificateAuthorities": {
       
    }
}

Logs after running invoke.js

2020-11-26T05:31:09.252Z | connectivity_state | dns:localhost:30751 CONNECTING -> CONNECTING
2020-11-26T05:31:09.252Z | dns_resolver | Resolved addresses for target dns:localhost:30751: [127.0.0.1:30751]
2020-11-26T05:31:09.252Z | pick_first | IDLE -> IDLE
2020-11-26T05:31:09.252Z | resolving_load_balancer | dns:localhost:30751 CONNECTING -> IDLE
2020-11-26T05:31:09.253Z | connectivity_state | dns:localhost:30751 CONNECTING -> IDLE
2020-11-26T05:31:09.253Z | pick_first | Connect to address list 127.0.0.1:30751
2020-11-26T05:31:09.253Z | subchannel | 127.0.0.1:30751 refcount 3 -> 4
2020-11-26T05:31:09.253Z | pick_first | IDLE -> TRANSIENT_FAILURE
2020-11-26T05:31:09.253Z | resolving_load_balancer | dns:localhost:30751 IDLE -> TRANSIENT_FAILURE
2020-11-26T05:31:09.253Z | connectivity_state | dns:localhost:30751 IDLE -> TRANSIENT_FAILURE
2020-11-26T05:31:12.254Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Endorser- name: acme-peer-clusterip:30751, url:grpc://localhost:30751, connected:false, connectAttempted:true
2020-11-26T05:31:12.254Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server acme-peer-clusterip:30751 url:grpc://localhost:30751 timeout:3000
2020-11-26T05:31:12.254Z - error: [DiscoveryService]: _buildPeer[dsg-test] - Unable to connect to the discovered peer acme-peer-clusterip:30751 due to Error: Failed to connect before the deadline on Endorser- name: acme-peer-clusterip:30751, url:grpc://localhost:30751, connected:false, connectAttempted:true
2020-11-26T05:31:12.261Z - error: [DiscoveryHandler]: _build_endorse_group_member >> G1:0 - returning an error endorsement, no endorsement made
2020-11-26T05:31:12.261Z - error: [Transaction]: Error: No valid responses from any peers. Errors:
    peer=undefined, status=grpc, message=Endorsement has failed
Failed to submit transaction: Error: No valid responses from any peers. Errors:
    peer=undefined, status=grpc, message=Endorsement has failed

Below kubernetes pods setup kubectl get all

NAME                 READY   STATUS    RESTARTS   AGE
pod/acme-orderer-0   1/1     Running   0          107m
pod/acme-peer-0      2/2     Running   0          107m
pod/budget-peer-0    2/2     Running   0          107m

NAME                             TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)                           AGE
service/acme-orderer-clusterip   ClusterIP      10.108.218.191   <none>           30750/TCP                         107m
service/acme-orderer-nodeport    NodePort       10.111.186.82    <none>           30750:30750/TCP                   107m
service/acme-peer-clusterip      ClusterIP      10.98.236.210    <none>           30751/TCP,30752/TCP               107m
service/acme-peer-nodeport       NodePort       10.101.38.254    <none>           30751:30751/TCP,30752:30752/TCP   107m
service/budget-peer-clusterip    ClusterIP      10.108.194.45    <none>           30851/TCP                         107m
service/budget-peer-nodeport     NodePort       10.100.136.250   <none>           30851:30851/TCP,30852:30852/TCP   107m
service/kubernetes               ClusterIP      10.96.0.1        <none>           443/TCP                           112m
service/svc-acme-orderer         LoadBalancer   10.105.155.207   10.105.155.207   6005:30696/TCP                    27m
service/svc-acme-peer            LoadBalancer   10.98.44.14      10.109.214.71      3005:30594/TCP                    56m

NAME                            READY   AGE
statefulset.apps/acme-orderer   1/1     107m
statefulset.apps/acme-peer      1/1     107m
statefulset.apps/budget-peer    1/1     10

Answer 1

Some things to check:

1. connection1-org1.json - you've supplied connection1-org1.json in the code for invoke.js but connection.json config as a candidate you presumably expect to be used to load/access the network. Pls confirm if this is correct.
2. connection.json - specifies a tlsCACerts but the url scheme used is grpc:// rather than grpcs://
3. connection.json - does not specify the orderer url. This may not be necessary if it is discoverable via other means
4. Is TLS enabled on the peers/orderer?

If you can confirm/amend as necessary it will provide a better chance for someone to help.