What does the non-terminating-p argument of set-macro-character do?

Question

Set-macro-character has an optional argument called non-terminating-p. It seems to be used to indicate whether another character should be read after reading the macro character, but the reader algorithm seems to ignore this argument. Is there a difference whether I set it to true or false?

Answer 1

If you set non-terminating-p to t the macro character can appear unescaped within a symbol name (like # in the symbol foo#baz). If you leave it nil and the reader encounters the macro character while accumulating a symbol it terminates the collection of the symbol (like ' in foo'bar, which reads as the symbol foo and the list (quote bar)).

See step 8 in the reader algorithm -- the argument in question toggles the macro character between being treated as a non-terminating macro character and a terminating macro character.

Answer 2

The other answers explain the behavior and give examples, but it's worth pointing out and linking to the spec, I think. The HyperSpec for set-macro-character describes the non-terminating-p argument:

If non-terminating-p is true, char becomes a non-terminating macro character; otherwise it becomes a terminating macro character.

The relevant glossary entries:

non-terminating adj. (of a macro character) being such that it is treated as a constituent character when it appears in the middle of an extended token. See Section 2.2 (Reader Algorithm).

terminating n. (of a macro character) being such that, if it appears while parsing a token, it terminates that token. See Section 2.2 (Reader Algorithm).

Then, as m-n's answer points out, here's step eight of the reader algorithm:

8. At this point a token is being accumulated, and an even number of multiple escape characters have been encountered. If at end of file, step 10 is entered. Otherwise, a character, y, is read, and one of the following actions is performed according to its syntax type:

   • If y is a constituent or non-terminating macro character:

     • If y is a character with case, it might be replaced with the corresponding character of the opposite case, depending on the readtable case of the current readtable, as outlined in Section 23.1.2 (Effect of Readtable Case on the Lisp Reader).
     • Y is appended to the token being built.
     • Step 8 is repeated.

   • …

   • If y is a terminating macro character, then it terminates the token. First the character y is unread (see unread-char), and then step 10 is entered.

Answer 3

An example, where you see the difference:

We'll use two different Unicode-Characters (this is using LispWorks). It gets interesting, when these characters are used as part of a symbol.

(defun single-quote-reader (stream char)             ; just an example, it won't be called
  (declare (ignore char))
  (list 'quote (read stream t nil t)))

(set-macro-character #\℅ #'single-quote-reader t)    ; non-terminating
(set-macro-character #\℆ #'single-quote-reader nil)  ; terminating

(defun test ()
  (list (read-from-string "foo℅bar")                 ; non-terminating
        (read-from-string "foo℆bar")))               ; terminating

CL-USER 21 > (test)
(FOO\℅BAR                                            ; non-terminating
 FOO)                                                ; terminating