Sitecore Experience Editor Links Not Editable

Question

When trying to edit items in Sitecore experience editor, editing links does not work. When I click on them i select the parent item instead of the link itself. If I click a second time it follows the link. This only happened for links, Images and Text are editable like they should be. The field type in Sitecore is a General Link.

This is an ASP.NET Core project, utilizing sitecore tag helpers.

My model binding looks like this for the variable

    [SitecoreComponentField(Name = "Site Url")]
    public HyperLinkField SiteUrl { get; set; }

And the html looks like this

    @if (Model.IsEditing)
    {
        <sc-link asp-for="SiteUrl">Edit Link</sc-link>
        <sc-img asp-for="SiteLogo"></sc-img>
    }
    else
    {
        <sc-link asp-for="SiteUrl">
            <sc-img asp-for="SiteLogo"></sc-img>
        </sc-link>
    }

Have tried disabling CSS and Javascript, tried not using Sitecore tag helpers, have made sure that the parent div isn't blocking the link from being hit in some way (text and images are editable just fine). Have tried changing some functions in Startup.cs .WithExperienceEditor function called on AddSitecoreRenderingEngine().