In a class the prof said "an [assembly] instruction knows if it's running in privileged or user mode because if a trap is produced then it's user mode".
First off I assume it can't be any trap but a specific trap saying that the command cannot be run in user mode.
Second, this seems awfully inefficient: run the whole command and check if there was a trap. Why isn't there a flag set or something? Isn't this like saying if it didn't work then we know there's a problem? So there really is no way to check?
This depends on your platform (i.e. ARM, x86 etc...). I work primarily with ARM so my answer might be ARM specific but I've kept it as generic as possible.
Usually, the CPU keeps track of what processor mode it is in and each instruction is checked as it is being executed. This is done in hardware and not in software so you don't worry about whether it is efficient (leave that to the hardware developers).
If the instruction is trapped by the hardware because there isn't sufficient privileges, the CPU usually starts executing at a set location in a privileged mode. This set location is a trap handler which is usually in the kernel. The trap handler then gathers some information about what caused the trap and handles it appropriately.