How can I remove a file on uninstall and not on an upgrade?

Question

I want use the following declaration to remove a file on uninstall:

<Component Id="RemoveFilesComponent" Directory="BIN" KeyPath="yes" Guid="MY-GUID-HERE">
    <RemoveFile Id="RemoveTxtFile" Directory="BIN" Name="file.txt" On="uninstall" />
</Component>

Problem: The attribute On="uninstall" also removes the file during an upgrade.

I use the following declaration to enable upgrades of the software:

<MajorUpgrade DowngradeErrorMessage="A newer version of $(var.AppName) is already installed." AllowSameVersionUpgrades="yes" />

How can I rewrite On="uninstall" in the element RemoveFile, that the file will not be removed (replaced) on an upgrade?

Additional Info: The file file.txt must be explicit removed by the declaration RemoveFile, because it is an generated file by the app.