What happens if a process calls out to code belonging to another process?

Question

What happens if a process calls out, for code execution, to a far pointer that belongs to another process?

In 32-bit operating mode, the GDT is non-trivially used and segmentation can be fully used.

So, what happens if the requested CS pointer points to another process's segment?

Sometimes this is allowed, I think.

But, generally it's not.

In 64-bit mode, segmentation is halfway turned off.

All segments run from 0 to the top of the processor's linear space.

Also, the GDT seems to be trivially used; Linux, for example, throws all user code from all processes into one segment.

Same for system code.

So, in 64-bit mode, you wouldn't even need a far pointer to call out to another process's code.

Maybe my question can be boiled down to this:

What happens when a process tries to call another process's code?

Answer 1

What OS are you running?

In general, your OS runs your process in protected memory, meaning there is no access to memory not owned by your process. Attempting to access memory outside your virtual memory results in a segmentation fault. To access such memory you would need to be part of the kernel.