Are there any programming pitfalls of using a map with an empty interface as the KEY

Question

Are there any programming pitfalls of using maps in this manner:

type Set struct {
    theMap map[interface{}]struct{}
}

StringSet := NewSet("abc", "pqr")
IntSet := NewSet(1, 2)
DateSet := NewSet(time.Date(2021, 2, 15, 0, 0, 0, 0, time.UTC))

Just to be clear, I know what I'm doing is probably against the spirit of several 'best practices', but that isn't my question here. I'm specifically thinking of programming issues like memory issues due to different element sizes, increased chance of hash collisions, performance degradation due to increased type assertion, etc.

Some more info: I need to create some 'sets' of various datatypes in my application. I see in Essential Go that the best way to use sets is to use a map with an empty struct as the value.

However, in the absence of generics, I would either need to make a new type of Set for each type of data/element which I wish to store in my sets:

type StringSet struct {
    stringMap map[string]struct{}
}
type DateSet struct {
    dateMap map[time.Time]struct{}
}
type IntSet struct {
    intMap map[int]struct{}
}

...or, use the empty interface as the key of a hashmap:

type Set struct {
    theMap map[interface{}]struct{}
}

The 2nd option works very well (you can find my complete code here), but I'm worried that I'm overlooking something obvious and will run into problems later.

Thanks for your help.