Why typescript does not narrow in this case (rest parameter)?

Question

EDIT: Link to a MRE (thanks @Thomas)

I have these types :

export declare type Matrix2D = [
  number, number,
  number, number,
  number, number,
];

export declare type Matrix3D = [
  number, number, number, number,
  number, number, number, number,
  number, number, number, number,
  number, number, number, number,
];

export declare type Matrix = Matrix2D | Matrix3D;

export type Coords2D = [number, number]
export type Coords3D = [number, number, number]
export type Coords4D = [number, number, number, number]
export type Coords = Coords2D | Coords3D | Coords4D

...And this code :

export function translate(...x:[Coords]|number[]):Matrix{
  if(typeof(x[0]) == "number") {
    return [ // Transposed
      1    , 0    , 0    , 0    ,
      0    , 1    , 0    , 0    ,
      0    , 0    , 1    , 0    ,
      x[0] , x[1]??0 , x[2]??0 , x[3]??1 ,
//           ^^^^^^^   ^^^^^^^   ^^^^^^^    ERROR
    ]
  }
  else {
    const X = normalize(x[0])
    return [ // Transposed
      1    , 0    , 0    , 0    ,
      0    , 1    , 0    , 0    ,
      0    , 0    , 1    , 0    ,
      X[0] , X[1] , X[2] , X[3] ,
    ]
  }
}

The error is this one :

error TS2322: Type 'number | Coords' is not assignable to type 'number | undefined'.
  Type 'Coords2D' is not assignable to type 'number'.

       x[0] , x[1]??0 , x[2]??0 , x[3]??1 ,
              ~~~~~~~

If I am not mistaking, when I use [Coords]|number[] as the type of the rest parameter, I telling TS that the rest parameter is either a tuple containing a single value of type Coords, which is itself a tuple (thus an array) and not a number. Either an array of number ?

Why is the compiler complaining about the second value (x[1]) telling me "Type 'number | Coords' is not assignable to type 'number | undefined'" while it can't even be a Coord ?