What are the new rules for variable scoping in Emacs 24?

Question

Emacs 24 now has lexically-scoped variables. It also still has dynamically-scoped variables, of course. Now that it has both, I'm quite confused about when a variable will have which kind of scope. There's a lexical-binding variable that controls when lexical binding is enabled, and I think I read something about defvar now declaring a dynamically-scoped variable, but in general I'm pretty lost. Is there a good explanation somewhere of Emacs 24's new scoping rules? Or put another way, when I look at a variable in Emacs Lisp code written for Emacs 24, how do I tell what scope that variable is using?

Answer 1

The manual is the definitive source. Start here:

C-hig (elisp) Variable Scoping RET

I had originally quoted the manual in this answer, but that information (dating back to Emacs 24.0.90.1) was slightly out of date. Better to read the manual from inside Emacs so that the information will be correct for the version you're using.

If you particularly want to read it on a web page, the current version is:
http://www.gnu.org/software/emacs/manual/html_node/elisp/Variable-Scoping.html

Answer 2

Let's say that some code is being evaluated step by step in Emacs (either because you just did C-x C-e or because an Emacs Lisp file is being loaded or because a function from a hook is being run, etc), and that Emacs is about to evaluate my-abc within that code. Maybe my-abc is a local variable in that code or maybe it is not declared or maybe it has some global value, etc. Anyway, the current step is evaluation of my-abc. At that point, Emacs checks just two things to decide whether to evaluate my-abc using lexical scope or not.

First thing Emacs checks is "is my-abc a special variable?". The answer to that question is yes if (defvar my-abc ...) or (defcustom my-abc ..) or etc was run at any point in the past. Maybe (defcustom my-abc ..) was run while loading some other Emacs Lisp file, or maybe you evaluated some code containing (defvar my-abc ...) on the scratch buffer, or maybe not. If the answer is yes for any reason, Emacs at this point will evaluate my-abc using dynamic scope.

If the answer is no, then Emacs checks the second thing, which is (A) "where is this code (containing use of my-abc) that I (Emacs) am stepping through?". This is not (B) "what is the current buffer now?". If you just pressed C-x C-e on a buffer, say foo.el, and if the expression you pressed C-x c-e on contained a call to a function named mah-hello which is defined in mah-stuff.el, and if mah-hello function body contained a call to a function named my-hello which is defined in my-stuff.el, and if the function body of my-hello contained use of a variable named my-abc, then when Emacs eventually get to executing my-hello and then is about to evaluate my-abc in there, at that point when Emacs asks Question A, it answers my-stuff.el to itself. Not the buffer foo.el that contains the starting expression.

Then Emacs asks "is my-stuff.el a lexically scoped buffer, in other words, is lexical-binding true on that buffer?". If yes, Emacs evaluates my-abc using lexical scope, otherwise using dynamic scope.

Some update: Also, when the code is quoted as data and then passed to the eval function, the answer to (A) will not be a buffer. Still, it's as if eval makes up an imaginary buffer to place the code, and set the buffer-local value of lexical-binding for that buffer to the second argument passed to eval. The answer to (A) is not the buffer containing the `eval' call. It is the imaginary buffer.

For Lisp macros, when some macroexpanded code is being run, it's as if the expanded code is written to the buffer that contains the code that invoked the macro. Therefore the answer to (A) in this case is not the buffer that defined the macro, but the buffer where code that called the macro resides.