NTE_BAD_DATA in CryptSetKeyParam while setting KP_P in wincrypt

Question

I am having the below code. I am setting a prime for diffie-hellman algorithm using char *. I am getting bad data after i set the prime. Where am i doing wrong? I followed the same example in this link. https://msdn.microsoft.com/en-us/library/aa381969(VS.85).aspx#exchanging_diffie-hellman_keys

What is the correct way to set prime in diffie-hellman using wincrypt?

#define DHKEYSIZE 1024
int fld_sz = 256;
BYTE* g_rgbPrime = new BYTE[DHKEYSIZE/8];   
char * prime = "A1BD60EBD2D43C53FA78D938C1EF8C9AD231F9862FC402739302DEF1B6BEB01E5BE59848A04C48B0069A8FB56143688678F7CC1097B921EA3E13E1EF9B9EB5381BEFDE7BBF614C13827493A1CA31DA76B4083B62C5073451D6B1F06A2F1049C291464AC68CBB2F69474470BBAD374073392696B6447C82BF55F20B2D015EB97B";
string s_prime(prime, fld_sz);
vector<std::string> res;
// split the string two charactes for converting into hex format
for (size_t i = 0; i < fld_sz; i += 2)
    res.push_back(s_prime.substr(i, 2));
for(int i = 0; i < res.size(); i++) {
    BYTE b = static_cast<BYTE>(std::stoi(res[i], 0, 16));
    g_rgbPrime[i] = b;
}
BYTE g_rgbGenerator[128] = 
{
    0x02
};
BOOL fReturn;
HCRYPTPROV hProvParty1 = NULL; 
HCRYPTPROV hProvParty2 = NULL; 
CRYPT_DATA_BLOB P;
CRYPT_DATA_BLOB G;
HCRYPTKEY hPrivateKey1 = NULL;
HCRYPTKEY hPrivateKey2 = NULL;
PBYTE pbKeyBlob1 = NULL;
PBYTE pbKeyBlob2 = NULL;
HCRYPTKEY hSessionKey1 = NULL;
HCRYPTKEY hSessionKey2 = NULL;
PBYTE pbData = NULL;

/************************
Construct data BLOBs for the prime and generator. The P and G 
values, represented by the g_rgbPrime and g_rgbGenerator arrays 
respectively, are shared values that have been agreed to by both 
parties.
************************/
P.cbData = DHKEYSIZE / 8;
P.pbData = (BYTE*)(g_rgbPrime);

G.cbData = DHKEYSIZE / 8;
G.pbData = (BYTE*)(g_rgbGenerator);

/************************
Create the private Diffie-Hellman key for party 1. 
************************/
// Acquire a provider handle for party 1.
fReturn = CryptAcquireContext(
    &hProvParty1, 
    NULL,
    MS_ENH_DSS_DH_PROV,
    PROV_DSS_DH, 
    CRYPT_VERIFYCONTEXT);
if(!fReturn)
{
    goto ErrorExit;
}

// Create an ephemeral private key for party 1.
fReturn = CryptGenKey(
    hProvParty1, 
    CALG_DH_EPHEM, 
    DHKEYSIZE << 16 | CRYPT_EXPORTABLE | CRYPT_PREGEN,
    &hPrivateKey1);
if(!fReturn)
{
    goto ErrorExit;
}

// Set the prime for party 1's private key.
fReturn = CryptSetKeyParam(
    hPrivateKey1,
    KP_P,
    (PBYTE)&P,
    0);
if(!fReturn)
{
    std::cout << GetLastError() << endl;
    goto ErrorExit;
}

// Set the generator for party 1's private key.
fReturn = CryptSetKeyParam(
    hPrivateKey1,
    KP_G,
    (PBYTE)&G,
    0);
if(!fReturn)
{
    std::cout << GetLastError() << endl;
    goto ErrorExit;
}

Thanks in advance.

Update 1: Thanks to @RbMm I was able to set the prime. The problem was with DHKEYSize. However i am getting an error in while setting KP_X. updated the code above to reflect the new code.

Here i converted the string to hex bytes array.

Answer 1

size of prime KP_P (and KP_G) and DH key size hard connected. must be cbKey == 8*cbP. look for example Diffie-Hellman Client Code for Creating the Master Key:

as key size if used cbP * 8 where cbP size of prime P. in your link also P.cbData = DHKEYSIZE/8;

also in code instead hard-code size of P (and G) you can get it in runtime:

ULONG dwDataLen;
CryptGetKeyParam(hPrivateKey1, KP_P, 0, &(dwDataLen = 0), 0);
CryptGetKeyParam(hPrivateKey1, KP_G, 0, &(dwDataLen = 0), 0);

and you can sure that dwDataLen == DHKEYSIZE / 8 where DHKEYSIZE is key size.

because you use 512 as key size, the length of data for P and G must be 512/8=64. but you use 256 (for P) and 1 (for G). as result and error.