Unit tests are supplementary source code created to automatically test the functionality and correctness of code modules. They are a way of formalizing assumptions about the code’s behavior. With unit tests, one can validate code and code changes at any time, and know exactly when a new change breaks existing code.
This page discusses unit testing of EmacsLisp scripts and programs.
A unit test is a piece of a code that checks the correctness of some other code. In Emacs Lisp, meaningful “units” to test are functions, commands and macros.
Various testing frameworks simplify the task of writing and running tests. The following terms are commonly encountered:
(assert (= (+ 2 2) 4))verifies that “2 + 2 = 4”; if it wasn’t (the comparison returned
assertwould signal an error.
download-webpage” may return an appropriate value without actually connecting to the Web.
The precise meanings of these terms may vary, though there is some commonality among xUnit-like frameworks (like SUnit for Smalltalk, JUnit for Java, and lisp-unit for Common Lisp).
The following unit test frameworks exist for Emacs Lisp.
ErtTestLibrary – Elisp Regression Testing (ERT) by Christian M. Ohler is now included in Emacs. The framework provides facilities for defining and running test suites, reporting the results, and debugging test failures interactively.
ElUnit is an experimental framework by PhilHagelberg. It is inspired by regress.el, Ruby’s
Test::Unit framework and xUnit, and provides test suites and fixtures. It is deprecated in favor of
test-simple.el is intended to be
I use this my Debugger front-end https://github.com/rocky/emacs-dbgr
el-expectations.el by rubikitch is a small framework focusing on simplicity and readability. It is modeled after Ruby expectations (for example,
(expect 4 (+ 2 2)) verifies “2 + 2 = 4”) and works with el-mock.el, a DSL-based mock/stub framework.
ecukes is a cucumber-like framework for writing tests. It allows you to write human-readable feature tests.
El4r (Emacs Lisp for Ruby) leverages Ruby’s
Test::Unit framework by using the EmacsRuby extension language. You write unit tests in EmacsRuby, while the code is in EmacsLisp. In other words, you can treat the EmacsLisp code as a black box and feed it inputs and check the outputs using EmacsRuby.
unit-test.el by MarkTriggs reports the pass/fail status of your unit tests (in any language). You need to define a function that runs your unit tests and returns non-
nil if they pass. Depending on the output, it will display green or red “light” graphic icon (
xpm) on the Emacs window mode line.
To sprinkle your code with tests, wrap them in
(eval-when-compile ;; unit test code ...)
Then, when you byte-compile the file, the test code is executed; when you load the file, it is not. A more flexible approach is to define a
run-my-tests variable and set it to
t during compilation:
(defvar run-my-tests nil)
(eval-when-compile (setq run-my-tests t))
(when run-my-tests ;; unit test code ...)
That compiles the test code too, and the
run-my-tests variable is evaluated a lot when loading the file. But since all the tests are skipped when loading, the compiled
.elc files are still much faster than the uncompiled
Assertions can be made with the
assert macro from cl.el:
(assert (= (+ 2 2) 4)) (assert (/= (* 2 3) 5))
Remember to load
Assertions can be grouped into tests with
(defun my-test-arithmetic () (assert (= (+ 2 2) 4)) (assert (/= (* 2 3) 5)))
Tests can be grouped into test suites with
(defvar my-test-suite) (add-hook 'my-test-suite 'my-test-arithmetic)
To run the suite:
(when run-my-tests (run-hooks 'my-test-suite))
Functions can be temporarily rebound with
flet from cl.el:
(defun my-test-division () (flet ((/ (dd ds) 0)) (assert (= (/ 1 0) 0))))
The above will suppress an
arith-error when dividing by
For more complex scenarios, mocker.el is a layer on top of
flet that supports programmable mock functions.
While fixtures are a great syntactic simplification in other languages, they are not very useful in Lisp, where higher-order functions and
unwind-protect are available. One approach is
(unwind-protect (progn ;; Set up ... (my-test-arithmetic)) ;; ... tear down )
The documentation of ert.el sketches out a more general solution.
That there is so much interest in this looks great, but could please someone start making a comparison of the different approaches? What does they support? How easy are they to use? Could more simple packages be built upon more general packages? Etc.
Indeed. Given that ERT is now part of GNU Emacs, I think particularly helpful would be if people who know and like other test frameworks summarised the differences, what they percieve as missing from ERT etc., so other people can make up their mind as to whether to stick with it or not, or push for improvements to ERT being accepted by the maintainers.
One question: do any of these packages support TAP (the Test Anything Protocol)? I don’t see a single reference to it.
In short order I added optional TAP output to test-simple.el. However after doing that I realize you are not going to be able to use prove unless one creates a custom Test::Harness. Open an issue on the github page if this is worth pursuing.
Does anybody know any mock or stub frameworks in Emacs Lisp? Because many Emacs Lisp functions have side-effects, a mock / stub framework is essential to unittest in Emacs Lisp. – rubikitch
I intended to create
mock.el, but I didn’t yet … Matsuyama