How do I font lock dollar signs (math mode delimiters) in AUCTeX buffer only outside comments?

Question

I wrote the following code to highlight dollar signs in AUCTeX buffers in different colors, but then I found that it's even highlighting dollar signs in comments, which was unintended, but I am starting to like it. But now just for curiosity, I wonder if that can be avoided.

(defun my-LaTeX-mode-dollars ()
  (font-lock-add-keywords
   nil
   `((,(rx "$") (0 'success t)))
   t))
(add-hook 'LaTeX-mode-hook 'my-LaTeX-mode-dollars)

Answer 1

From the documentation of font-lock-keywords:

MATCH-HIGHLIGHT should be of the form:

(SUBEXP FACENAME [OVERRIDE [LAXMATCH]])

OVERRIDE and LAXMATCH are flags. If OVERRIDE is t, existing fontification can be overwritten. If keep', only parts not already fontified are highlighted. Ifprepend' or `append', existing fontification is merged with the new, in which the new or existing fontification, respectively, takes precedence.

In other words, if you drop the t after 'success, it will no longer fontify dollar signs in comments and strings.

EDIT:

Apparently, the above solution is not sufficient in this situation, probably because dollar signs have been colored using another face earlier.

One way that might work is to not pass the HOW parameter (currently t) to font-lock-add-keywords. This means that they should be added to the end of the list. However, this might cause other things to stop working.

If we need a bigger hammer, you can write a bit more advanced rule that inspects the current fontification, and decides what to do upon this. For example, the following is used by Emacs to add a warning face to parentheses placed at column 0 in strings:

"^\\s("
  (0
   (if
       (memq
        (get-text-property
         (match-beginning 0)
         'face)
        '(font-lock-string-face font-lock-doc-face font-lock-comment-face))
           (list 'face font-lock-warning-face 'help-echo "Looks like a toplevel defun: escape the parenthesis"))
   prepend)

A third way to do this is to replace the regexp (rx "$") with the name of function that could search for $ and check that it appears in the correct context. One example of such font-lock rules can be found in the standard Emacs package cwarn.