I am trying to configure BizTalk Server 2010 along with SQL Server 2008 R2 with SP3. I am able to configure Enterprise SSO and Business Rules Engine, but I cannot configure Groups. When configuring Enterprise SSO, it is able to successfully create the SSO Database along with their respective accounts (SSO Administrators and SSO Affiliate Administrators). But for Groups, it is only able to create the Administrative Roles (BizTalk Administrators Group, Operators Group, and B2B Operators Group). It fails when trying to create the Databases (BizTalkMgmtDb, BizTalkMsgBoxDb, BizTalkDTADb).
Here are a few errors I am getting from the log file:
2015-06-09 07:50:23:4123 [Info] CfgExtHelper Checking the connection to the BizTalk Management Database: BizTalkMgmtDb on server *****
2015-06-09 07:50:23:4748 [Info] CfgExtHelper The BAM Primary Import Database found from the BizTalk Management Database BizTalkMgmtDb on server ***** is not compatible.
2015-06-09 07:50:23:4748 [Info] CfgExtHelper Connecting to the BAM Primary Import Table Database BAMPrimaryImport on server *****
2015-06-09 07:50:23:4905 [Info] CfgExtHelper The BAM Primary Import Database found from the BizTalk Management Database BAMPrimaryImport on server ***** is not compatible.
BizTalk 2010 won't work with SQL 2013, which you have installed on your server. What seems to be happening is that BizTalk is connecting to the server, and the default instance is likely running on SQL 2013 (hence the message "Database x on server y is not compatible"). You have to configure BizTalk to connect to the correct instance, or set the default instance to be the SQL 2008R2 instance.
An ideal fix would be to remove SQL 2013 from the server. BizTalk makes very heavy usage of MessageBox and Management databases, and best practice is to have it on its own dedicated server (or servers).