MultilineRegexp

A word of warning before we start: Multi-line regular expressions rarely work as expected for FontLock. Why is that, you ask? Emacs basically knows which line you are editing, and font-lock works on these lines you are editing. Multi-line regular expressions would require Emacs to skip backwards multiple lines and see if any of the multi-line regular expression match. That either involves a lot of matching, unsolved questions (skip back to where – the beginning of the buffer in the worst case), or caching. So while multi-line regular expressions work fine when loading a file, font-lock can get confused when you edit the lines containing the beginning or the end of the expression. See MultilineFontLock.

With that out of the way, back to basics:

To match any number of characters, use this: .* – the problem is that . matches any character except newline. What many people propose next works, but is inefficient if you assume that newlines are not that common in your text: "\\(.\\|\n\\)". If you use ‘rx’ instead of a regexp string then you can use ‘anychar’ (or ‘anything’). That expands to "\\(.\\|\n\\)". An alternative is to use ".*\\(\n.*\\)*". Another alternative is "[\0-\377[:nonascii:]]*".

All options tend to suffer from “stack overflow” when used in “large” buffers, though some eat up the stack faster than others. An option which uses less stack space (proportional to the number of lines rather than the number of characters) could be \\(.*\n\\)*.*.

Incremental search

By default, in IncrementalSearch, “ “ (space) is interpreted to mean one or more of any whitespace character except newline. If you want to match a phrase which may have been wrapped by auto-fill-mode, this is inconvenient, but you can customize search-whitespace-regexp to something like “[ \t\r\n]+” as the documentation suggests, in order to include newlines. (You might have to set isearch-regexp-lax-whitespace to t in order to leverage this.)

For a literal newline in IncrementalSearch use ‘C-q C-j’. Typing “\n” does not work. You don’t have to use C-q C-m C-q C-j or similar variants for Windows or Mac files.

Parsing example

The following code snippet parses entries that look like this:

    A term :: a definition 
    which may span multiple lines 
    and is terminated by a blank line or
    the end of the file
    
    Another term :: You get the idea

Code:

    (defun parse-definitions ()
      (let ((dict nil))
        (while (re-search-forward
                (concat "^\\(.*?\\)"               ;; a term
                        "\\s-*::\\s-*"             ;; the separator
                        "\\(.*\\(?:\n.*\\)*?\\)"   ;; definition: to end of line,
                                                   ;; then maybe more lines
                                                   ;; (excludes any trailing \n)
                        "\\(?:\n\\s-*\n\\|\\'\\)") ;; blank line or EOF
                nil :no-error)
          (add-to-list 'dict (cons (match-string-no-properties 1)
                                   (match-string-no-properties 2))) )
        dict))

The regular expression matching the definition may need some more explaining.

What we have is this:

    "\\(.*\\(?:\n.*\\)*?\\)"

Taking it appart from the outside, we have a group. This is the second group; the first group matched the term. Therefore the later call to (match-string-no-properties 2) will return the definition matched.

    "\\( ... \\)"

Inside the group, we start with all characters except newline. This will match starting at point up to the end of the line. If we are at the end of the line, this will match the empty string before the newline.

    ".*"

Next we have a complex little bugger. On the outside, we have a group that is matched repeatedly, but as little as possible due to the *? at the end. Furthermore, this group will not record the matched substring (thus being faster). It is a so-called shy group due to the ?: at the very beginning of the group.

    "\\(?: ... \\)*?"

Inside the shy group, we match the newline, and all the characters following it except another newline. That is, we match the newline we did not match two paragraphs up, and the next line, ending again in front of a newline:

    "\n.*"

Associated XEmacs Bug

Above does not work for me in XEmacs 21.5-b22 or XEmacs 21.4.17 either.

If I change the third string to

                        "\\(.*?\\(?:\n.*\\)*?\\)"  ;; definition: to end of line,

instead of

                        "\\(.*\\(?:\n.*\\)*?\\)"   ;; definition: to end of line,

to make the initial wildcard match non-greedy, the match works!

As Alex points out below, “.” will never match a newline, but perhaps this observation of greediness will help diagnose this XEmacs bug.

This must be a bug in XEmacs.

Hm. can .* ever match \n? One of the main gotchas is that there is not (as far as I know) an equivalent of Perl’s /s modifier, so . can’t match \n [I was also testing in scratch]

You are correct. That’s the whole point of having the \n in there. I’ve taken in further appart. – AlexSchroeder


CategoryRegexp