UnitTesting

Unit tests are supplementary source code created to automatically test the functionality and correctness of code modules. They are a way of formalizing assumptions about the code’s behavior. With unit tests, one can validate code and code changes at any time, and know exactly when a new change breaks existing code.

This page discusses unit testing of EmacsLisp scripts and programs.

Terminology

A unit test is a piece of a code that checks the correctness of some other code. In Emacs Lisp, meaningful “units” to test are functions, commands and macros.

Various testing frameworks simplify the task of writing and running tests. The following terms are commonly encountered:

The precise meanings of these terms may vary, though there is some commonality among xUnit-like frameworks (like SUnit for Smalltalk, JUnit for Java, and lisp-unit for Common Lisp).

Frameworks

The following unit test frameworks exist for Emacs Lisp.

Included with Emacs

ErtTestLibrary – Elisp Regression Testing (ERT) by Christian M. Ohler is now included in Emacs. The framework provides facilities for defining and running test suites, reporting the results, and debugging test failures interactively.

xUnit-like

ElkTest by Nikolaj Schumacher is inspired by JUnit for Java. It provides visual feedback with fringe-helper.el and a number of specialized assertion macros like assert-equal, assert-nil, etc.

ElUnit is an experimental framework by PhilHagelberg. It is inspired by regress.el, Ruby’s Test::Unit framework and xUnit, and provides test suites and fixtures. It is deprecated in favor of ert.el.[1]

test-simple.el is intended to be

I use this my Debugger front-end https://github.com/rocky/emacs-dbgr

Behavior Driven Development

el-expectations.el by rubikitch is a small framework focusing on simplicity and readability. It is modeled after Ruby expectations (for example, (expect 4 (+ 2 2)) verifies “2 + 2 = 4”) and works with el-mock.el, a DSL-based mock/stub framework.

ETest (Emacs Testing Framework) by Phil Jackson allows arbitrary nesting of tests. It also works with el-mock.el. Manual.

PhilHagelberg describes a BDD framework with expectations-like syntax. Description.

ecukes is a cucumber-like framework for writing tests. It allows you to write human-readable feature tests.

Others

El4r (Emacs Lisp for Ruby) leverages Ruby’s Test::Unit framework by using the EmacsRuby extension language. You write unit tests in EmacsRuby, while the code is in EmacsLisp. In other words, you can treat the EmacsLisp code as a black box and feed it inputs and check the outputs using EmacsRuby.

Emtest is a modular, extensible testing framework. It supersedes regress.el, also by Tom Breton. It lives in a git repo on Savannah.

test.el is developed for testing operations on a buffer (but can also be used to test operations unrelated to buffers). Screenshot.

unit-test.el by MarkTriggs reports the pass/fail status of your unit tests (in any language). You need to define a function that runs your unit tests and returns non-nil if they pass. Depending on the output, it will display green or red “light” graphic icon (xpm) on the Emacs window mode line.

Without a framework

To sprinkle your code with tests, wrap them in eval-when-compile:

    (eval-when-compile
      ;; unit test code
      ...)

Then, when you byte-compile the file, the test code is executed; when you load the file, it is not. A more flexible approach is to define a run-my-tests variable and set it to t during compilation:

    (defvar run-my-tests nil)
    (eval-when-compile
     (setq run-my-tests t))
    (when run-my-tests
      ;; unit test code
      ...)

That compiles the test code too, and the run-my-tests variable is evaluated a lot when loading the file. But since all the tests are skipped when loading, the compiled .elc files are still much faster than the uncompiled .el files.

Assertions

Assertions can be made with the assert macro from cl.el:

    (assert (= (+ 2 2) 4))
    (assert (/= (* 2 3) 5))

Remember to load cl.el with (require 'cl).

Tests

Assertions can be grouped into tests with defun:

    (defun my-test-arithmetic ()
      (assert (= (+ 2 2) 4))
      (assert (/= (* 2 3) 5)))

Test suites

Tests can be grouped into test suites with add-hook:

    (defvar my-test-suite)
    (add-hook 'my-test-suite 'my-test-arithmetic)

To run the suite:

    (when run-my-tests
      (run-hooks 'my-test-suite))

Mock functions

Functions can be temporarily rebound with flet from cl.el:

    (defun my-test-division ()
      (flet ((/ (dd ds) 0))
        (assert (= (/ 1 0) 0))))

The above will suppress an arith-error when dividing by 0.

For more complex scenarios, mocker.el is a layer on top of flet that supports programmable mock functions.

Fixtures

While fixtures are a great syntactic simplification in other languages, they are not very useful in Lisp, where higher-order functions and unwind-protect are available. One approach is

    (unwind-protect
        (progn
          ;; Set up ...
          (my-test-arithmetic))
      ;; ... tear down
      )

The documentation of ert.el sketches out a more general solution.

Automated testing

External links

Discussion

That there is so much interest in this looks great, but could please someone start making a comparison of the different approaches? What does they support? How easy are they to use? Could more simple packages be built upon more general packages? Etc.

Indeed. Given that ERT is now part of GNU Emacs, I think particularly helpful would be if people who know and like other test frameworks summarised the differences, what they percieve as missing from ERT etc., so other people can make up their mind as to whether to stick with it or not, or push for improvements to ERT being accepted by the maintainers.

One question: do any of these packages support TAP (the Test Anything Protocol)? I don’t see a single reference to it.

In short order I added optional TAP output to test-simple.el. However after doing that I realize you are not going to be able to use prove unless one creates a custom Test::Harness. Open an issue on the github page if this is worth pursuing.

Does anybody know any mock or stub frameworks in Emacs Lisp? Because many Emacs Lisp functions have side-effects, a mock / stub framework is essential to unittest in Emacs Lisp. – rubikitch

I intended to create mock.el, but I didn’t yet … Matsuyama


CategoryCode CategoryExtensions