Wix CustomActionData property value is empty when read from inside a Rust custom action 2 out of 10 times

Question

In WIX code we declare the deferred custom action that will secure the credentials and the auxiliary custom action that sets the CustomActionData value for it as follows:

    <CustomAction Id="SetUserNamePasswordForDefferedAction" Property="HandleAccountInfoCustomAction" Value="[ACCOUNT_USERNAME],[ACCOUNT_PASSWORD],[INSTALLLOCATION],[SERVICEACCOUNT]" HideTarget="no" Return="check" />
    <CustomAction Id="HandleAccountInfoCustomAction" DllEntry="handle_account_info" BinaryKey="service_account_dll" Execute="deferred" HideTarget="no" Impersonate="no" Return="check"/>
    <Binary Id="service_account_dll" SourceFile="service_account.dll" />

    <InstallExecuteSequence>
        <Custom Action="HandleAccountInfoCustomAction" Before="InstallFinalize">NOT Installed</Custom>
        <Custom Action="SetUserNamePasswordForDefferedAction" Before="HandleAccountInfoCustomAction">NOT Installed</Custom>
    </InstallExecuteSequence>

service_account_dll is implemented in Rust but exports the entry point handle_account_info in C format. The rust code that reads the custom action data is the following:

unsafe fn retrieve_property_from_msi_db(
    h_install: MSIHANDLE,
    property_name_str: &str,
) -> StdResult<String, u32> {
    let property_name = get_ptr_to_cstring(property_name_str);
    custom_action_log(&format!("Retrieve {property_name_str} from MSI db"));
    let mut msi_get_property_call_count = 0usize;
    let mut res;
    let mut req_status;
    let mut buff = [0u8; BUFFER_SIZE];

    res = 0u32;
    req_status = 234u32; // ERROR_MORE_DATA

    while (WIN32_ERROR(req_status) == ERROR_MORE_DATA || res == 0) && msi_get_property_call_count < 5 {
        res = buff.len() as u32;
        req_status = MsiGetPropertyA(
            h_install,
            PCSTR::from_raw(property_name),
            PSTR::from_raw(buff.as_mut_ptr()),
            Some(&mut res as *mut u32),
        );
        if msi_get_property_call_count > 0 && WIN32_ERROR(req_status) == ERROR_MORE_DATA {
            // Register the occurrence of a known race condition in MSI
            custom_action_log(format!("WARN: Multiple calls to MsiGetPropertyA ({msi_get_property_call_count}) returned ERROR_MORE_DATA.").as_str());
        }
        if res == 0 {
            custom_action_log(format!("WARN: MsiGetPropertyA returned 0 bytes for property {property_name_str}.").as_str());
            std::thread::sleep(Duration::from_millis(10));
        }

        msi_get_property_call_count += 1;

        if res as usize > BUFFER_SIZE {
            custom_action_log(format!("Custom action buffer size is too small: Needed {res}/available {BUFFER_SIZE}.").as_str());
            return Err(ERROR_INSTALL_FAILED.0);
        }
    }
    if ERROR_SUCCESS != WIN32_ERROR(req_status) {
        custom_action_log(format!("Failed to retrieve user name from MSI:WIN_32ERROR {:?}, req_status: {req_status}", WIN32_ERROR(req_status)).as_str());
        return Err(ERROR_INSTALL_FAILED.0);
    }

    let property = std::str::from_utf8_unchecked(&buff[..res as usize]);

    Ok(property.to_string())
}

Two or three out of ten I get the following in the custom action logs:

2023-11-14 13:29:47 - Retrieve CustomActionData from MSI db
2023-11-14 13:29:47 - WARN: MsiGetPropertyA returned 0 bytes for property CustomActionData.
2023-11-14 13:29:47 - WARN: MsiGetPropertyA returned 0 bytes for property CustomActionData.
2023-11-14 13:29:47 - WARN: MsiGetPropertyA returned 0 bytes for property CustomActionData.
2023-11-14 13:29:47 - WARN: MsiGetPropertyA returned 0 bytes for property CustomActionData.
2023-11-14 13:29:47 - WARN: MsiGetPropertyA returned 0 bytes for property CustomActionData.
2023-11-14 13:29:47 - service_account CustomActionData data retrieved from MSI db.
2023-11-14 13:29:47 - TROUBLESHOOTING: CustomActionData:
2023-11-14 13:29:47 - CustomActionData property is not in the expected format

A successful run produces the following logs:

2023-11-14 13:28:48 - Retrieve CustomActionData from MSI db
2023-11-14 13:28:48 - service_account CustomActionData data retrieved from MSI db.
2023-11-14 13:28:48 - TROUBLESHOOTING: CustomActionData: ,,C:\MyApp,NT Authority\Local Service

Help, please :-)

Answer 1

I solved my problem by changing my strategy. Long story short, instead of a custom action implemented in a DLL to be loaded by the wix installer app, the custom action is a standalone executable now where arguments are passed through the command line. Wix provides support for doing that without logging the command line arguments so the password is not logged. I followed this documentation article, in particular the WixSilentExec flavor.

My understanding as to why directly accessing a dll from Wix turned out to be unstable is that the mechanism to pass argument values going through the CustomActionData property in the MSI database is prone to race conditions.

Note: v3 of Wix target is a 32-bit executable, using a custom action dll might also be a source of instability.