mouseenter makes button visible but button click event then doesn't fire

Question

I have made a button (btnImageRemove) get shown and hidden based on if the mouse is inside the component, but when i then click that button the event doesnt fire.

Private Sub btnImageRemove_Click(sender As Object, e As EventArgs) Handles btnImageRemove.Click
    lblAvatar.Image = Nothing
    lblAvatar.Tag = Nothing
End Sub

Private Sub lblAvatar_MouseLeave(sender As Object, e As EventArgs) Handles lblAvatar.MouseLeave
    btnImageRemove.Visible = False
End Sub

Private Sub lblAvatar_MouseEnter(sender As Object, e As EventArgs) Handles lblAvatar.MouseEnter
    btnImageRemove.Visible = True
    btnImageRemove.BringToFront()
End Sub

Private Sub btnImageRemove_MouseClick(sender As Object, e As MouseEventArgs) Handles btnImageRemove.MouseClick
    lblAvatar.Image = Nothing
    lblAvatar.Tag = Nothing
End Sub

I have tried all the click and mouseclick events on the button, and all components that take its area, but i cannot get an event to fire for that button

Narrowed down cause a little more: I have narrowed down to if i add anything to show / hide the button at runtime, during another event, the button doens't fire a click event, nor turn orange (default dotnetbar button action)