C# EF - How to setup a one-to-many styled many-to-many junction table as one-to-many?

Question

Sorry if the title is confusing or incorrect. First time I've seen a junction table setup like this before and that's the best way I can think to describe it.

I have the following table classes setup for my DbContext:

    public class Deck
    {
        public long Id { get; protected set; }
        public string Name { get; protected set; }
        [Column("mtime_secs")]
        public int MtimeSecs { get; protected set; }
        public byte[] Common { get; protected set; }
        public byte[] Kind { get; protected set; }
        public ICollection<Note> Notes { get; protected set; }
    }

    public class Card
    {
        public long Id { get; protected set; }
        [Column("nid")]
        public long NoteId { get; protected set; }
        [Column("did")]
        public long DeckId { get; protected set; }
    }

    public class Note
    {
        public long Id { get; protected set; }
        public string Tags { get; protected set; }
        [NotMapped] //Convert space delimited Tags field to list
        public List<string> TagsList { get { return Tags.Split(' ').ToList(); } }
        [Column("flds")]
        public string Fields { get; protected set; }
        [Column("sfld")]
        public string SortField { get; protected set; }
    }

Deck to Note is a one-to-many relationship, but the junction table, Cards, can contain duplicate NoteId and DeckId pairs (Card contains other columns I don't care about, which are the reason it has duplicate pairs. It's essentially setup to duplicate the note entry but with different ways of being displayed.).

I want to setup an ICollection<Note> Notes field in the Deck class, but I'm not sure how to go about it in EF. I assume I could just brute force it and define the getter as a distinct Note LINQ query with .ToList(), but I'm not sure how efficient that is.

I tried messing around with overriding OnModelCreating(), but the .UsingEntity<Card>() function isn't available when using .WithOne() instead of .WithMany().

What would be the best way to go about this?