Lisp quiz of the day: macroexpansion gotchas

Suppose you want to use *MACROEXPAND-HOOK* to track something while debugging. For example, you're interested in SETF forms with the pattern (SETF (FOO ...) ...). You might decide to try something like the following. (The values returned by TRACKING-HOOK are just for demonstration.)

(defun setf-list-form-p (form)
  (and (consp form) (eq 'setf (car form)) (consp (cadr form))))

(defun setf-form-place-name (form)
  (when (setf-list-form-p form) (caadr form)))

(defun tracking-hook (expander form env)
  (if (eql 'foo (setf-form-place-name form))
      (values '(cons "Yes" nil) t)
      (values '(cons "No" nil) t)))

If you're not careful (see below) and use this directly in Allegro CL, you will have a problem. (Try to figure out why before reading on.)

To make things a little easier, try it out and see what happens. Assume we have entered the definitions for the above functions are the REPL already.

CL-USER> (let ((*macroexpand-hook* 'tracking-hook))
           (macroexpand-1 '(setf (foo 10) 10)))
Error: Stack overflow (signal 1000)
  [condition type: SYNCHRONOUS-OPERATING-SYSTEM-SIGNAL]

So, why does this happen?

This had me stumped for a while this morning until I looked at some stack traces by using BREAK at the beginning of TRACKING-HOOK. The problem will go away if you compile SETF-LIST-FORM-P and SETF-FORM-PLACE-NAME.

CL-USER> (compile 'setf-list-form-p)
SETF-LIST-FORM-P
NIL
NIL
CL-USER> (compile 'setf-form-place-name)
SETF-FORM-PLACE-NAME
NIL
NIL
CL-USER> (let ((*macroexpand-hook* 'tracking-hook))
           (macroexpand-1 '(setf (foo 10) 10)))
(CONS "Yes" NIL)
T

The problem lies in how the function is evaluated. Each call to TRACKING-HOOK results in the body of SETF-FORM-PLACE-NAME being evaluated, which results in a macroexpansion of the WHEN form. This means that TRACKING-HOOK gets called again!

Welcome to infinite recursion land.

Compiling will steer us away from this because no macroexpansion happens for the evaluation of a compiled function.

The same behaviour can be seen in LispWorks Personal 5.0.1, but not CLisp 2.4 or SBCL 1.0.8.27. SBCL wouldn't let me bind *MACROEXPAND-HOOK* to a non-compiled function. CLisp worked, but when I tried to insert a BREAK form in TRACKING-HOOK or SETF-FORM-PLACE-NAME, it would print out information on the break some number of times and start using all the CPU. I'm not sure why yet.

The moral of the story: compile functions to be called from macroexpansion hooks.