Reference Path in Visual Studio not working when I unload the project in the same solution

Question

I have an ASP.NET MVC project with one solution and 52 projects in the same solution. As you know, It takes long time to build whole solution. So I decided to unload some of the projects until there is some change or sprint task in the code in the unloaded project. However we are a team and use TFS version control and we need to use update unloaded assembly. So I have a build server that makes me sure whenever somebody starts to check-in (for example -> C:\bin in our network) the last version of all assembly can be found there.

I added C:\bin path as Reference Path to the UI Project (main project of the solution) of the solution and I expect when some projects is unloaded the reference comes to visual studio from the build server path (C:\bin) otherwise the reference of the project comes to Visual Studio from the nested project (not from build server).

But the reference can not be found when the project is unloaded:

And my Build server contains the assembly:

Answer 1

Finally I found a way (only in Visual Studio 2019) to unload projects and however use updated class library of unloaded projects. And thanks to Hooman Bahreini for his idea of using two separated solutions.

The way is that, there is a feature of Visual Studio 2019 that we can Save As Solution Filter which is in .slnf extensions. So I can unload some of my projects and check-in .slnf file via my source control (like tfs). So I have two solutions Now, one is .sln which is contains all of projects and the other one is .slnf which is unloaded some of projects in it.

The things can we do is that whenever I want to get my project I have to use .sln file and builds that and then close the file and then open the .slnf file (I'm sure that I have updated class library of unloaded projects) and do my task and check-in my changes.