SxS .NET COM in vb6: ActiveX component can't create object?

Question

I am trying to get my .NET COM dll to work Side by Side (SxS) with my VB6 application. I want to use Side by Side (with manifest files) so that I can run the application without registering the libraries. After allot of trial and error I seem to have got it working. The TLB file is being referenced in the VB6 project and all methods are showing. The only problem now is that when I debug the VB6 application it results into an ActiveX component can't create object error when the class library is being used.

Private Sub Command1_Click()
    Dim myCustomer As New Customer
    
    myCustomer.Address = "test" <--- Error occurs here
    MsgBox myCustomer.Address
    
    Set Client = Nothing
End Sub

The Visual Studio 6 editor says everything is just fine. And when I compile the application as an EXE file it even runs without problems. Is Visual Studio 6 doing something odd that prevents Side by Side dll's from being used when debugging?

Some extra information: The Class Library is created for .NET Framework 4 and it is COM-visible and has COM interop enabled. The TLB file is generated and a manifest file has been generated with mt.exe.

---

Update 1: Since both regsvr32 and regasm fail to register my DLL file I might add some information about how I have build the COM assembly.

I have followed a walkthrough from Microsoft: Walkthrough: Creating COM Objects with Visual Basic. This walkthrough uses the COM Class templates. This is my final code the Customer class.

<ComClass(Customer.ClassId, Customer.InterfaceId, Customer.EventsId)>
Public Class Customer

#Region "COM GUIDs"
    ' These  GUIDs provide the COM identity for this class 
    ' and its COM interfaces. If you change them, existing 
    ' clients will no longer be able to access the class.
    Public Const ClassId As String = "08dbe20f-48b9-4ea3-8b19-7dbc3df311fa"
    Public Const InterfaceId As String = "f10cc4d2-b073-4c70-82c4-9fd393dea1e4"
    Public Const EventsId As String = "b13afd62-c892-40d7-aa60-d0a00993f4b7"
#End Region

    ' A creatable COM class must have a Public Sub New() 
    ' with no parameters, otherwise, the class will not be 
    ' registered in the COM registry and cannot be created 
    ' via CreateObject.
    Public Sub New()
        MyBase.New()
    End Sub

    Private _Name As String = "Initial value"
    Private _Address As String = "Initial value"

    Public Sub SetName(Value As String)
        _Name = Value
    End Sub

    Public ReadOnly Property Name As String
        Get
            Return _Name
        End Get
    End Property

    Public Property Address As String
        Get
            Return _Address
        End Get
        Set(value As String)
            _Address = value
        End Set
    End Property

End Class

In the project settings a have checked the checkboxes for Make assebmly COM-Visible and Register for COM interop. When I build the project I get a DLL file and a TLB file. After that I create a manifest file by running mt.exe with the following parameters: mt.exe" -managedassemblyname:MyComAssembly.dll -nodependency -out:MyComAssembly.dll.manifest. The resulting 3 files are being copied to my VB6 project folder. Within Visual Studio 6 I now select the TLB file as a reference and I can start coding. I added the DLL's manifest file on position 1 with the Resource Editor. That's about all. Now the compiled version of my VB6 application uses the .NET COM assembly without any errors. But debugging the VB6 project from within the Visual Studio 6 editor is the problem as mentioned in the original question.