In nearly every functional programming tutorial, a large section is dedicated to teaching you how to convert algorithms to a tail-recursive format, since this can be optimized to a loop.
This is fine, but it made me wonder why compilers cannot automatically convert an algorithm using "regular" recursion to use a separate stack object (allocated on the heap), and then convert the algorithm to something iterative.
I don't fully understand how the CHICKEN Scheme or Haskell compilers work (which I hear might be immune to stack overflows), but perhaps they are doing something like this? If so, why can't this be done in most languages?
I'm not saying this to badmouth compiler engineers, I just genuinely have no idea how this stuff works but would love to learn.
First, there is a difference between cannot and currently do not
As your suggestion is not what would be technically called hard, that is would not complete, or would not complete in your lifetime, it seems they in fact could
the implied question of why they do not is speculation, but likely based on how much resources it would take to guarantee the same results with or without the optimization
i do not miss the days of large uncertainty of the optimization level causing a bug, so I am happy to wait for this until its perfect
as far as i know, there is nothing in the standard precluding this optimization