Innehållsförteckning RecentChanges News ElispArea HowTo Problems Suggestions

UnicodeEncoding

GNU Emacs 23 and Later

Starting with GNU Emacs 23, there is relatively complete Unicode support. See the Emacs manual for more information. Some starter tips about entering (inserting) Unicode characters:

In both cases, the commands created have the same names as the characters they insert, except that ‘SPC’ characters in the character names are replaced by hyphens (‘-’), and the command names are lowercase, not uppercase like the character names.

Using Unicode with Emacs Releases Prior to Emacs 23

GNU Emacs 21 has built-in Unicode support which is sufficient for editing, for example, UTF-8-encoded Western technical text from the Basic Multilingual Plane. It doesn’t support most Far Eastern characters from the BMP, or combining characters (which can easily be added), but some support for them has been installed in the development sources, probably for GNU Emacs 22.1. On version 21.4, it means, for example, that reading a Japanese UTF-8 encoded text will not be displayed correctly but in three bytes instead of a single character. It also means that if you enter a text with a Japanese input method, it will be displayed correctly but you can’t save the buffer in utf-8 (see section Saving CJK into Unicode): if your text mixes Arabic, Russian, West-European character and Japanese characters, all but the Japanese will be displayed and saved correctly. Also, saving and pasting from other applications, will not work (see Saving section). The GNU Emacs 21.2 PROBLEMS file is somewhat misleading about Unicode support. GNU Emacs 21.3 contains support for UTF-16 encoding of the same characters as UTF-8 and its utf-8 coding system will encode characters from more Emacs charsets than in 21.2.

Various Unicode-related extensions for GNU Emacs 21.2 are available from http://www.loveshack.ukfsn.org/emacs/Mule/. Beware that these currently have some bugs compared with the equivalent code installed in the Emacs development sources.

In future (probably Emacs 22), Mule will use an internal encoding which is a UTF-8 encoding of a superset of Unicode. The basic implementation of that is done, but needs plenty more work.

Note that the internal encoding is largely orthogonal to whether Emacs can satisfactorily edit unicoded documents, as the GNU Emacs 21 implementations demonstrate. You do not need a Unicode-based Emacs to use Unicode. UTF-8 is just another coding system; the only fundamental issue is whether all the relevant characters can be represented.

Input Characters Using an Input Method

Assume you have an UTF-8 file, but your locale uses ISO-8859-1 (Latin-1). Typically when you open the file, it will be garbled, depending on its exact contents and your setup. Auto-detection of UTF-8 is effectively disabled by default in GNU Emacs 21.3 and below. You can prefer it just below your preferred coding system by specifying utf-8 with ‘M-x prefer-coding-system’ and then repeating the command to replace your most preferred coding system at the front of the priority list (‘coding-category-list’).

Otherwise, you need to tell Emacs that the file is in UTF-8 before reading it by prefixing your ‘C-x C-f’ or ‘C-x C-v’ command with ‘C-x C-m c utf-8 RET’. It should display correctly.

To edit the file, just type. If you need to insert characters that your keyboard does not have, you need an InputMethod. Choose one using ‘M-x set-input-method’. Once you have an input method active, toggle it on and off using `C-\’.

To describe input methods, use ‘C-h I’.

I also bind ‘set-input-method’ as follows:

    (global-set-key (kbd "C-x C-m i") 'set-input-method)

C-\ can also set input method. --XueFuqiao

If you always prefer UTF-8 to ISO-8859-1, you can use this:

    (prefer-coding-system 'utf-8)
You might just want to use Emacs 21.3’s UTF-8 language environment.

Input Characters by Code Point

This function, bound to a key, will allow you to enter unicode characters by code point: handy if you do not know which input method to use, but can find the code point of the character:

    (defun unicode-insert (char)
      "Read a unicode code point and insert said character.
    Input uses `read-quoted-char-radix'.  If you want to copy
    the values from the Unicode charts, you should set it to 16."
      (interactive (list (read-quoted-char "Char: ")))
      (ucs-insert char))

You can find charts of the characters here: http://www.unicode.org/charts/

If you use them, you might want to use the following setting to enter hexadecimal:

    (setq read-quoted-char-radix 16)
How is this better than just using ‘ucs-insert’, which I wrote for this job?

If you know the emacs code point of the character, you can enter it with C-q instead.

`auto-coding-regexp-alist'

You need ‘M-x customize-option RET auto-coding-regexp-alist RET’. You can then insert items such as

  <meta\b[^>]*\bcontent="text/html; charset=UTF-8"[^>]*>
  utf-8
  
  <\?xml\b[^>]*\bencoding="utf-8"[^>]*\?>
  utf-8

There may be a better way – these patterns are a little too rigid for reliable identification of all Unicode documents. At least you can edit them to cope with the set of documents you normally deal with.

You can use ‘file-coding-system-alist’ to specify that, say, .xml and .utf8 files are utf-8-encoded.

But how can I get emacs to recognize utf-16 automatically? This is the encoding used on Windows for many files, including *.xml files. So using the file extension is not the correct approach. I am running gnu emacs version 21.2.1 on Windows XP. I tried the suggestion at http://lists.gnu.org/archive/html/help-emacs-windows/2002-09/msg00011.html i.e. require utf-16.el but without success. In any case I want to be able to say it after I am looking at a file in a buffer, not need to do it before hand. Even better I want emacs to automatically recognize this. This should be easy because the first two bytes in the file are the byte order mark. See http://www.unicode.org/faq/utf_bom.html#BOM This shows that the file is utf-16 and also tells whether it is little endian or big endian. Steve Tolkin

With the coding systems defined by utf-16.el, just use ‘C-x C-m c’ to specify the coding system when you find the file. You can’t sensibly do anything after finding it. You need something like the following to auto-detect utf-16 with signature. Note that we were confused about the correct names for the utf-16 coding systems, and they have changed after Emacs 21.3.
    (setq coding-category-utf-16-le 'utf-16-le)
    (push 'coding-category-utf-16-le coding-category-list)

I (Steven Tolkin, who asked the question) added the above lines, but this was not enough. In fact it seemed at one time to cause this error: “Symbol’s function definition is void: utf-16-le-pre-write-conversion”. The fix, from http://www.gatago.com/gnu/emacs/help/20222624.html is to afterwards also add the following lines. This now makes emacs 21.3.1 work on Windows XP, i.e. it correctly opens an UTF-16 file that starts with a BOM.

    ;; Detect endianness of UTF-16 containing a Byte Order Mark U+FEFF 
    (add-to-list 'auto-coding-regexp-alist '("^\xFF\xFE" . utf-16-le) t)
    (add-to-list 'auto-coding-regexp-alist '("^\xFE\xFF" . utf-16-be) t)
    ;; Add missing support functions
    (defun utf-16-le-pre-write-conversion (start end) nil)
    (defun utf-16-be-pre-write-conversion (start end) nil)

A further refinement is to add support for auto detecting the line ending. The following code will handle line ending differences:

    ;; Detect endianness of UTF-16 containing a Byte Order Mark U+FEFF
    ;; Detect EOL mode by looking for CR/LF on the first line
    (add-to-list 'auto-coding-regexp-alist '("^\xFF\xFE.*\x0D\x00$" . utf-16-le-dos) t)
    (add-to-list 'auto-coding-regexp-alist '("^\xFE\xFF.*\x0D\x00$" . utf-16-be-dos) t)
    (add-to-list 'auto-coding-regexp-alist '("^\xFF\xFE" . utf-16-le) t)
    (add-to-list 'auto-coding-regexp-alist '("^\xFE\xFF" . utf-16-be) t)

Saving CJK as Unicode

Some CJK (Chinese, Japanese, Korean) characters have been “unified” in the Unicode standard. What does that mean? The Unicode FAQ by Markus Kuhn [1] says:

ISO 10646 defines formally a 31-bit character set. The most commonly used characters, including all those found in older encoding standards, have been placed in one of the first 65534 positions (0x0000 to 0xFFFD). This 16-bit subset of UCS is called the Basic Multilingual Plane (BMP) or Plane 0. Initially, 29,000 CJK characters were placed in the CJK Unified Ideographs block starting at U+4E00. Since then three more blocks have been allocated: CJK Unified Ideographs Extension A, starting at U+3400; CJK Compatibility Ideographs, starting at U+F900; and CJK Unified Ideographs Extension B, in Plane 1, starting at U+020000. Further additions are planned, and there is an entire plane of more than 65,000 code points reserved for CJK, which is expected to reach somewhere between 100,000 and 150,000 code points.

Some compilations of Chinese characters alone can already take 50000 positions. Putting them all in the BMP would not leave enough space for all the other characters there, so Unicode unified sets of character shapes that have the same meaning but vary in inessential detail. The Han Unification in Unicode article by Otfried Cheong [2] uses the following two characters as an example to illustrate what kinds of characters have been unified, even though this particular pair has not been unified:

Another example of a character element that appears in two variants is the “black” radical. It can be written either in its traditional form as U+9ed1 (with two little dots), or in its simplified form as U+9ed2 (where the two dots have been replaced by a single stroke):
黑 黒

footnote: In Fact, the radical on the left is simplified form. Simplified Chinese Character are not always simpler than other variants,it’s more of a unified standard form! Anyway, there are indeed a lot of variants which cause communication difficulty in some extend in Chinese world. – A reader from Beijing

For some users, this unification was not acceptable, however. Otfried Cheong says:

I believe the JIS standard actually prescribes the shapes of the glyphs for each character, and this is perhaps exactly the grief that Japanese have with Unicode. If you are used to think about a codepoint being associated with a well-defined shape, the loose view that Unicode takes seems rather careless.
Note that the situation is similar for maths characters. E.g. the black letter I ࠿ (U+2111) and script I ࠾ (U+2110) mean very different things to a physicist, and Unicode actually defines many mathematical shape letters as separate characters.

Mokurai 黙雷 Cherlin says:

Historically, however, shapes of Unified characters that many Japanese now consider distinct were taught as being equivalent even in Japan. There are only a few characters about which there is any fuss, and all of them fall into this category, where there was a difference between Chinese and Japanese calligraphy, which continues to be respected in fonts specifically meant for one language, but not in generic Unicode fonts.
Another point of contention is the Japanese modification of ASCII, JIS X 0201, to have the Yen sign (now U+00A5) in the location of the backslash, or REVERSE SOLIDUS, ‘\’ (ASCII 0x5C, U+005C), and to substitute overline ‘¯’, now U+00AF, for tilde ‘~’ at 0x7E. This created inherent ambiguity in programming for Windows, where ‘\’ is the separator in file paths, except in systems where ‘\’ has been moved to 80h, thus breaking compatibility with the ISO 8859 encodings. Some Japanese programmers claim that Unicode is broken for not permitting the practice of overloading 0x5C to continue, and insisting that Japanese programmers should either label their files with the proper legacy encoding, or translate them properly to Unicode. This may require manual intervention to resolve the ambiguity, throughout a huge code base. Note that some Microsoft Windows fonts for Japan and Korea that claim to be Unicode-compliant have Yen sign or Korean Won sign at this code point.

GNU Emacs 21 can read JIS (Japanese) encoded files, but in order to save such a file as UTF-8, you need a mapping from the JIS codepoints to the corresponding Unicode codepoints. You can do this using ‘utf-translate-cjk-mode’ in Emacs CVS, or by loading mule-ucs in GNU Emacs 21, or by using external character encoding conversion tools. (The problem otherwise is that Emacs doesn’t have character values to represent the relevant Unicodes in a char table for the CCL translation mechanism; Mule-UCS uses a complicated system of vectors to do the translation, and ‘utf-translate-cjk-mode’ uses a new feature allowing hash tables to do the job.) In your InitFile:

    (utf-translate-cjk-mode 1)

Now you should be able to save the Japanese, Chinese, and Korean part of the etc/HELLO file using UTF-8. To test it, copy a few CJK characters from the etc/HELLO file and save them in a UTF-8 file using `C-x C-m c utf-8 RET C-x C-s filename RET’.

Pasting

Maybe worth noting that Windows, Mac, and Linux all have character selection tools designed to make cut/paste with special characters a little easier (e.g., linux utilities include gucharmap and kcharselect)?

If you have trouble pasting from other applications: Emacs by default requests a selection of type COMPOUND_TEXT. It seems that xterm (and other applications) ignore this sometimes and reply using UTF-8 encoded text.

Here’s how to tell Emacs to request UTF-8 first:

   (setq x-select-request-type '(UTF8_STRING COMPOUND_TEXT TEXT STRING))

Note that this was introduced post-Emacs 21.3, i.e. isn’t released at the time of writing. Also UTF8_STRING isn’t documented yet by X, and the behaviour of XFree86 and GNOME, Mozilla &c for these things has been quite unstable over different releases, sigh. The situation for selections is particularly complicated by X being a networked window system which may involve exchanging selections between old non-XFree86 or X.org releases…

Comments

I wrote the support, and I’m unclear what this is about. It seems confused about characters and coding systems. ‘utf-translate-cjk-mode’ isn’t released at least before Emacs 22.1. It mitigates Han unification for most users because the charsets it prefers depend on the language environment from which it was invoked. It doesn’t allow the whole of HELLO to be encoded. In a released Emacs, you probably want to use Mule-UCS for CJK Unicode as above. – fx

I’ve tried to rewrite it. Better now? – AlexSchroeder

I modified your introduction in order to make it clear that displaying and saving Asian texts is not working, even under my 21.4 version. Benefit of UTF-8 is when you edit mixed text including Arabic (yes it is not right to left on the screen but you can read the characters at least), Russian, Western European characters; but if you input a Japanese sentence (with any Japanese input method) you can’t save it. Am I wrong ? – NicolasBrouard?

I read some xml comment that makes emacs open that file utf-8 converted similar to the bash annotation:

    # -*- coding: utf-8 -*-

and without configuration in emacs. But i cant find it anymore. some ideas?

Do you simply mean <!-- -*- coding: utf-8 -*- --> ?

XML is utf-8 by default, so an entry was added to ‘file-coding-system-alist’ post-21.3. Then the coding cookie is only necessary if the encoding isn’t utf-8, but a new mechanism (‘auto-coding-functions’) tries to figure that out without the cookie. As far as I remember, it only works if the charset specification in the XML declaration matches a coding system exactly, modulo case. (The function ‘locale-charset-to-coding-system’ in the emacs-unicode CVS branch is designed to cope with things like `ISO8859_1’ versus ‘iso-8859-1’.)
Emacs 22 should get xml right on its own, and for Emacs 21 (as a bit of shameless self-promotion) try XmlCoding. – KevinRyde, Jan08

Say you have opened an utf-8 encoded file and the encoding is wrong, what are, if any, the steps to being able to edit it correctly without having to reload the file? If it is impossible, can it be documented on this page?

That isn’t generally possible—incorrect decoding loses information. The correct thing to do is just to re-find it with ‘C-x C-m c utf-8’.
How is it lost? See ChangingEncodings
It depends on what you mean by “the encoding is wrong”. If you mean (for example) “the file is encoded in latin-1 on disk but that is wrong, it should be utf-8 instead (and Emacs is displaying the non-ascii characters correctly)” then yes, see ChangingEncodings. If you mean “the file is encoded in utf-8 on disk but Emacs guessed the encoding wrong (it interpreted it as latin-1, so the non-ascii characters are displayed as gibberish)” you should revisit it with the right coding system. I guess in this particular case no information was technically “lost” by the incorrect decoding (you could encode the gibberish back to bytes using latin-1 and decode the bytes using utf-8 and the result would make sense) but I am not sure if that is the case for all coding systems (it depends on what they do with invalid input (including corner cases like utf-8 “overlong forms” (see http://en.wikipedia.org/wiki/UTF-8) where simply leaving them as-is may not even be what the user wants!)

Combining characters

It seems Emacs does not support some combining characters. For instance if you paste or read from a file the word страни́ца (as seen in Wiktionary) you will see страни´ца. – DanielClemente, Apr2012, 24.1.50.4 in X.org in Debian

Combining characters work for me with the Freemono font. Interestingly, no other (monospace) fonts seem to work.

CategoryInternationalization