Is it okay to pass non-ref objects as refs if they are contained within a ref?

Question

I have a ref that contains an object that contains two refs:

function useFocusTargets() {
  const from = useRef();
  const to = useRef();
  const focusTargets = useRef({from : from, to: to});  
  return focusTargets.current;
}

I pass the two inner refs around to components to get access to their DOM elements. It worked, but I changed it so that the two inner refs were just regular objects instead useRef({from : {}, to: {}}) and it still seemed to work...but I got this warning:

Unexpected ref object provided for input. Use either a ref-setter function or React.createRef().

I'm assuming that what I did isn't proper. But I can't figure out why. It seems to be working just fine and it makes sense to me (focusTargets is a ref so it's preserved so the {from : {}, to :{}} inside focusTargets should also be preserved).

Is it okay to do what I did, and if not, why?