Assume I have an application, where user can provide script written in JavaScript to perform some task in it. This is done using Jint. However, badly written script can cause Stack Overflow Exception in JintEngine.Run() and therefore crashing whole application. I would rather like to inform user about the error in script.
I tried to run Jint on another application domain, that did not help, cause AFAIK default action on SOE is exiting process. Could be it changed any other way than by using CLR hosting?
I know I can run separate process and that's my fallback, nevertheless I would like not to do that.
As you suspected, it's just not possible to catch a StackOverflowException generated by the runtime unless you're hosting the CLR yourself [1]. As the Microsoft documentation suggests, your best bet is to try and detect the stack overflow before it happens.
In the case of using Jint, this means finding some way to hook into the execution engine and attach your own code to arbitrary method calls. Fortunately, a brief overview of their documentation reveals that they have a debugging mode that does just that! [2]
No doubt running in debugging mode means slowing down your script execution; it's up to you to decide if the benefit of detecting overflows is worth the penalty--and I definitely suggest measuring that penalty to ensure it's really going to be a problem. One possible mitigation would be to allow the user to choose when debugging mode is active.
-Mark
[1] http://msdn.microsoft.com/en-us/library/system.stackoverflowexception.aspx (see remarks)
[2] http://jint.codeplex.com/wikipage?title=Debugging&referringTitle=Documentation