Just to be explicit and clear. Does the OpenSSH agent keep the unencrypted private key in memory or does it keep the passphrase in memory, and decrypt the key as it needs it? I think this question, 9650410, answers me indirectly. (The unencrypted private key is in memory.)
I'm not a coder and couldn't make sense of of the source if I tried.
I believe ssh-agent uses the passphrase once to decrypt the private key, and the decrypted private key is held in memory. The passphrase is used only initially.
See: https://security.stackexchange.com/questions/39837/does-ssh-agent-store-the-ssh-key-passwords-in-cleartext-in-the-memory