unexpected interaction between macroexpand-1 and macrolet

Question

Consider the following sequence of forms in Common Lisp (evaluations performed in SBCL 2.4.2 for Windows):

(defmacro double-g (x)
  (list (quote +) x x))

(macroexpand-1 (quote (double-g 3))) => (+ 3 3), T

(macrolet ((double-l (x) (list (quote +) x x)))
  (macroexpand-1 (quote (double-l 3))))
=> (DOUBLE-L 3), NIL

Can someone help me understand why the second and third forms evaluate to different results?

I expected the second and third forms to evaluate to the same result.

Answer 1

Problem: you are trying to expand the macro in the wrong environment. The macrolet defines a local macro, but macroexpand-1 does not see it, because it does not get the lexical environment passed.

We'll modify the macro expander:

CL-USER 23 > (defmacro expand-1-form (form &environment env)
               (list 'quote (macroexpand-1 form env)))
EXPAND-1-FORM

Above defines a macro expand-1-form, which gets passed the current environment, which then is available via the variable env. The macro expands this form by using macroexpand-1 and also passes the environment to it. The macro expand-1-form gets the environment automagically passed into the environment variable env (which we defined in the parameter list). The expansion is returned as a quoted form.

Now we can expand the local macro double-l using the macro expand-1-form:

CL-USER 24 > (macrolet ((double-l (x) (list (quote +) x x)))
               (expand-1-form (double-l 3)))
(+ 3 3)

Above expands the macro expand-1-form, passing the current environment. This then calls macroexpand-1 with that environment.

We can also write it as nested macrolets:

CL-USER 25 > (macrolet ((expand-1-form (form &environment env)
                          (list 'quote (macroexpand-1 form env))))
               (macrolet ((double-l (x)
                            (list (quote +) x x)))
                 (expand-1-form (double-l 3))))
(+ 3 3)

Answer 2

macroexpand-1 and macroexpand need to know about the environment of a macro call in order to expand it, and they take an optional environment argument that can provide this context. By default these procedures use the null lexical environment.

Macro lambda lists allow the inclusion of an &environment parameter which is bound to an environment object representing the lexical environment of a macro call. You can use this to write macros that expand forms within their local environments:

(defmacro macroexpand-local-1 (form &environment env)
  `,(macroexpand-1 form env))

(defmacro macroexpand-local (form &environment env)
  `,(macroexpand form env))

CL-USER> (macrolet ((double-l (x) `(+ ,x ,x)))
           (macroexpand-1 '(double-l 3)))
(DOUBLE-L 3)
NIL
CL-USER> (macrolet ((double-l (x) `(+ ,x ,x)))
           (macroexpand-local-1 '(double-l 3)))
(+ 3 3)
T

Avoiding Trouble with Dynamic Extent

These notes attempt to shed some light on a subtlety relating to the macro environment objects associated with &environment parameters. I was caught out on this; hopefully those who read this won't be.

I had previously written the get-lexenv convenience macro to return a macro environment object:

CL-USER> (defmacro get-lexenv (&environment env) `,env)
GET-LEXENV

I then used the returned environment object in a macro expansion call like this:

CL-USER> (macrolet ((double-l (x) (list (quote +) x x)))
           (macroexpand-1 (quote (double-l 3)) (get-lexenv)))
(+ 3 3)
T

This appeared to work, but it is not guaranteed to work. Thanks to @RainerJoswig for pointing this out in comments. Rainer also linked to an interesting discussion about the matter.

A macro environment object bound to an &environment parameter has dynamic extent, which means roughly that it can only be referenced from within the form which binds it. In get-lexenv the dynamic extent of the environment object ends once the evaluation of the get-lexenv form has ended.

It appears that some Common Lisp implementations have treated macro environment objects as having indefinite extent, at least in some circumstances. According to the linked discussion, CMU Common Lisp was one such implementation, and SBCL has descended from that code base. Perhaps that is why this seemed to work in this instance.

In any case, this can not be relied upon.