So I am given a function like 65536 n2 + 128 n log2n
and the only way that this would be O(n2 log2n) is if
C = 65664, n0 = 2 for all n ≥ 2 since
C1 = 65536 n1 = 2 when 65536 ≤ C1*n2 and
C2 = 128 n2 = 1 when 128 ≤ C2*n
but the number I've chosen for the constant seems a bit to high, is there a way to check this?
O(65536 n2 + 128 n log2n) is the same as O(n2 + n log2n) since you can ignore multiplicative constants. O(n2 + n log2n) is equal to O(n2) since n2 grows faster than n log2n.
Also, by the way, the base of logarithms doesn't matter in Big-O analysis. All logarithms grow at the same rate. After all, log2n = log n / log 2, and multiplicative constants can be factored out. You can simply say log n instead of log2n.
Caveat: Technically, it is actually a true statement to say that 65536 n2 + 128 n log2n ∈ O(n2 log2n) because Big-O gives an upper bound, but not a strict one. O(n2) is a lower upper bound, if that makes sense.
That said, you should not have come up with O(n2 log2n). That was merely the result of accidentally turning an addition into a multiplication. As a rule of thumb, if you have multiple things added together inside a Big-O formula, you just have to figure out which one of them grows the fastest and then discard the others.