Starting with GNU Emacs 23, there is relatively complete Unicode support. See the Emacs manual for more information. Some starter tips about entering (inserting) Unicode characters:
‘ucs-insert’ (‘C-x 8 RET’), which lets you either enter a Unicode code point or complete against the Unicode character name. It can be slow, because there are many candidate characters to choose from.‘ucsc-insert’, a replacement for ‘ucs-insert’ — bind it to ‘C-x 8 RET’. It does the same thing, except that with a negative prefix argument (for which ‘ucs-insert’ does nothing) it automatically creates such a command.‘ucs-make-commands’, which defines a whole set of such commands. You pass it a regexp, which is matched against all Unicode character names (in ‘ucs-names’). An insertion command is created for each of the characters whose name matches.‘SPC’ characters in the character names are replaced by hyphens (‘-’), and the command names are lowercase, not uppercase like the character names.GNU Emacs 21 has built-in Unicode support which is sufficient for editing, for example, UTF-8-encoded Western technical text from the Basic Multilingual Plane. It doesn’t support most Far Eastern characters from the BMP, or combining characters (which can easily be added), but some support for them has been installed in the development sources, probably for GNU Emacs 22.1. On version 21.4, it means, for example, that reading a Japanese UTF-8 encoded text will not be displayed correctly but in three bytes instead of a single character. It also means that if you enter a text with a Japanese input method, it will be displayed correctly but you can’t save the buffer in utf-8 (see section Saving CJK into Unicode): if your text mixes Arabic, Russian, West-European character and Japanese characters, all but the Japanese will be displayed and saved correctly. Also, saving and pasting from other applications, will not work (see Saving section). The GNU Emacs 21.2 PROBLEMS file is somewhat misleading about Unicode support. GNU Emacs 21.3 contains support for UTF-16 encoding of the same characters as UTF-8 and its utf-8 coding system will encode characters from more Emacs charsets than in 21.2.
Various Unicode-related extensions for GNU Emacs 21.2 are available from http://www.loveshack.ukfsn.org/emacs/Mule/. Beware that these currently have some bugs compared with the equivalent code installed in the Emacs development sources.
In future (probably Emacs 22), Mule will use an internal encoding which is a UTF-8 encoding of a superset of Unicode. The basic implementation of that is done, but needs plenty more work.
Note that the internal encoding is largely orthogonal to whether Emacs can satisfactorily edit unicoded documents, as the GNU Emacs 21 implementations demonstrate. You do not need a Unicode-based Emacs to use Unicode. UTF-8 is just another coding system; the only fundamental issue is whether all the relevant characters can be represented.
Assume you have an UTF-8 file, but your locale uses ISO-8859-1 (Latin-1). Typically when you open the file, it will be garbled, depending on its exact contents and your setup. Auto-detection of UTF-8 is effectively disabled by default in GNU Emacs 21.3 and below. You can prefer it just below your preferred coding system by specifying utf-8 with ‘M-x prefer-coding-system’ and then repeating the command to replace your most preferred coding system at the front of the priority list (‘coding-category-list’).
Otherwise, you need to tell Emacs that the file is in UTF-8 before reading it by prefixing your ‘C-x C-f’ or ‘C-x C-v’ command with ‘C-x C-m c utf-8 RET’. It should display correctly.
To edit the file, just type. If you need to insert characters that your keyboard does not have, you need an InputMethod. Choose one using ‘M-x set-input-method’. Once you have an input method active, toggle it on and off using `C-\’.
To describe input methods, use ‘C-h I’.
I also bind ‘set-input-method’ as follows:
(global-set-key (kbd "C-x C-m i") 'set-input-method)
C-\ can also set input method. --XueFuqiao
If you always prefer UTF-8 to ISO-8859-1, you can use this:
(prefer-coding-system 'utf-8)
This function, bound to a key, will allow you to enter unicode characters by code point: handy if you do not know which input method to use, but can find the code point of the character:
(defun unicode-insert (char)
"Read a unicode code point and insert said character.
Input uses `read-quoted-char-radix'. If you want to copy
the values from the Unicode charts, you should set it to 16."
(interactive (list (read-quoted-char "Char: ")))
(ucs-insert char))You can find charts of the characters here: http://www.unicode.org/charts/
If you use them, you might want to use the following setting to enter hexadecimal:
(setq read-quoted-char-radix 16)
‘ucs-insert’, which I wrote for this job?If you know the emacs code point of the character, you can enter it with C-q instead.
You need ‘M-x customize-option RET auto-coding-regexp-alist RET’. You can then insert items such as
<meta\b[^>]*\bcontent="text/html; charset=UTF-8"[^>]*> utf-8 <\?xml\b[^>]*\bencoding="utf-8"[^>]*\?> utf-8
There may be a better way – these patterns are a little too rigid for reliable identification of all Unicode documents. At least you can edit them to cope with the set of documents you normally deal with.
You can use ‘file-coding-system-alist’ to specify that, say, .xml and .utf8 files are utf-8-encoded.
But how can I get emacs to recognize utf-16 automatically? This is the encoding used on Windows for many files, including *.xml files. So using the file extension is not the correct approach. I am running gnu emacs version 21.2.1 on Windows XP. I tried the suggestion at http://lists.gnu.org/archive/html/help-emacs-windows/2002-09/msg00011.html i.e. require utf-16.el but without success. In any case I want to be able to say it after I am looking at a file in a buffer, not need to do it before hand. Even better I want emacs to automatically recognize this. This should be easy because the first two bytes in the file are the byte order mark. See http://www.unicode.org/faq/utf_bom.html#BOM This shows that the file is utf-16 and also tells whether it is little endian or big endian. Steve Tolkin
‘C-x C-m c’ to specify the coding system when you find the file. You can’t sensibly do anything after finding it. You need something like the following to auto-detect utf-16 with signature. Note that we were confused about the correct names for the utf-16 coding systems, and they have changed after Emacs 21.3. (setq coding-category-utf-16-le 'utf-16-le)
(push 'coding-category-utf-16-le coding-category-list)I (Steven Tolkin, who asked the question) added the above lines, but this was not enough. In fact it seemed at one time to cause this error: “Symbol’s function definition is void: utf-16-le-pre-write-conversion”. The fix, from http://www.gatago.com/gnu/emacs/help/20222624.html is to afterwards also add the following lines. This now makes emacs 21.3.1 work on Windows XP, i.e. it correctly opens an UTF-16 file that starts with a BOM.
;; Detect endianness of UTF-16 containing a Byte Order Mark U+FEFF
(add-to-list 'auto-coding-regexp-alist '("^\xFF\xFE" . utf-16-le) t)
(add-to-list 'auto-coding-regexp-alist '("^\xFE\xFF" . utf-16-be) t)
;; Add missing support functions
(defun utf-16-le-pre-write-conversion (start end) nil)
(defun utf-16-be-pre-write-conversion (start end) nil)
A further refinement is to add support for auto detecting the line ending. The following code will handle line ending differences:
;; Detect endianness of UTF-16 containing a Byte Order Mark U+FEFF
;; Detect EOL mode by looking for CR/LF on the first line
(add-to-list 'auto-coding-regexp-alist '("^\xFF\xFE.*\x0D\x00$" . utf-16-le-dos) t)
(add-to-list 'auto-coding-regexp-alist '("^\xFE\xFF.*\x0D\x00$" . utf-16-be-dos) t)
(add-to-list 'auto-coding-regexp-alist '("^\xFF\xFE" . utf-16-le) t)
(add-to-list 'auto-coding-regexp-alist '("^\xFE\xFF" . utf-16-be) t)
Some CJK (Chinese, Japanese, Korean) characters have been “unified” in the Unicode standard. What does that mean? The Unicode FAQ by Markus Kuhn [1] says:
Some compilations of Chinese characters alone can already take 50000 positions. Putting them all in the BMP would not leave enough space for all the other characters there, so Unicode unified sets of character shapes that have the same meaning but vary in inessential detail. The Han Unification in Unicode article by Otfried Cheong [2] uses the following two characters as an example to illustrate what kinds of characters have been unified, even though this particular pair has not been unified:
footnote: In Fact, the radical on the left is simplified form. Simplified Chinese Character are not always simpler than other variants,it’s more of a unified standard form! Anyway, there are indeed a lot of variants which cause communication difficulty in some extend in Chinese world. – A reader from Beijing
For some users, this unification was not acceptable, however. Otfried Cheong says:
Mokurai 黙雷 Cherlin says:
GNU Emacs 21 can read JIS (Japanese) encoded files, but in order to save such a file as UTF-8, you need a mapping from the JIS codepoints to the corresponding Unicode codepoints. You can do this using ‘utf-translate-cjk-mode’ in Emacs CVS, or by loading mule-ucs in GNU Emacs 21, or by using external character encoding conversion tools. (The problem otherwise is that Emacs doesn’t have character values to represent the relevant Unicodes in a char table for the CCL translation mechanism; Mule-UCS uses a complicated system of vectors to do the translation, and ‘utf-translate-cjk-mode’ uses a new feature allowing hash tables to do the job.) In your InitFile:
(utf-translate-cjk-mode 1)
Now you should be able to save the Japanese, Chinese, and Korean part of the etc/HELLO file using UTF-8. To test it, copy a few CJK characters from the etc/HELLO file and save them in a UTF-8 file using `C-x C-m c utf-8 RET C-x C-s filename RET’.
Maybe worth noting that Windows, Mac, and Linux all have character selection tools designed to make cut/paste with special characters a little easier (e.g., linux utilities include gucharmap and kcharselect)?
If you have trouble pasting from other applications: Emacs by default requests a selection of type COMPOUND_TEXT. It seems that xterm (and other applications) ignore this sometimes and reply using UTF-8 encoded text.
Here’s how to tell Emacs to request UTF-8 first:
(setq x-select-request-type '(UTF8_STRING COMPOUND_TEXT TEXT STRING))
Note that this was introduced post-Emacs 21.3, i.e. isn’t released at the time of writing. Also UTF8_STRING isn’t documented yet by X, and the behaviour of XFree86 and GNOME, Mozilla &c for these things has been quite unstable over different releases, sigh. The situation for selections is particularly complicated by X being a networked window system which may involve exchanging selections between old non-XFree86 or X.org releases…
I wrote the support, and I’m unclear what this is about. It seems confused about characters and coding systems. ‘utf-translate-cjk-mode’ isn’t released at least before Emacs 22.1. It mitigates Han unification for most users because the charsets it prefers depend on the language environment from which it was invoked. It doesn’t allow the whole of HELLO to be encoded. In a released Emacs, you probably want to use Mule-UCS for CJK Unicode as above. – fx
I’ve tried to rewrite it. Better now? – AlexSchroeder
I modified your introduction in order to make it clear that displaying and saving Asian texts is not working, even under my 21.4 version. Benefit of UTF-8 is when you edit mixed text including Arabic (yes it is not right to left on the screen but you can read the characters at least), Russian, Western European characters; but if you input a Japanese sentence (with any Japanese input method) you can’t save it. Am I wrong ? – NicolasBrouard?
I read some xml comment that makes emacs open that file utf-8 converted similar to the bash annotation:
# -*- coding: utf-8 -*-
and without configuration in emacs. But i cant find it anymore. some ideas?
Do you simply mean <!-- -*- coding: utf-8 -*- --> ?
‘file-coding-system-alist’ post-21.3. Then the coding cookie is only necessary if the encoding isn’t utf-8, but a new mechanism (‘auto-coding-functions’) tries to figure that out without the cookie. As far as I remember, it only works if the charset specification in the XML declaration matches a coding system exactly, modulo case. (The function ‘locale-charset-to-coding-system’ in the emacs-unicode CVS branch is designed to cope with things like `ISO8859_1’ versus ‘iso-8859-1’.)Say you have opened an utf-8 encoded file and the encoding is wrong, what are, if any, the steps to being able to edit it correctly without having to reload the file? If it is impossible, can it be documented on this page?
‘C-x C-m c utf-8’.It seems Emacs does not support some combining characters. For instance if you paste or read from a file the word страни́ца (as seen in Wiktionary) you will see страни´ца. – DanielClemente, Apr2012, 24.1.50.4 in X.org in Debian