Chapter 10. Character Set Support
Prev |  | Next

## Chapter 10. Character Set Support

Improved support for character set handling was added to MySQL in version 4.1. This support enables you to store data using a variety of character sets and perform comparisons according to a variety of collations. You can specify character sets at the server, database, table, and column level. MySQL supports the use of character sets for the `MyISAM`, `MEMORY`, and (as of MySQL 4.1.2) `InnoDB` storage engines. The `ISAM` storage engine does not include character set support; there are no plans to change this, because `ISAM` is deprecated.

Note: The `NDBCluster` storage engine in MySQL 4.1 (available beginning with MySQL 4.1.3-Max) provides limited character set and collation support; see Section 15.11, “Known Limitations of MySQL Cluster”.

This chapter discusses the following topics:

- What are character sets and collations?

- The multiple-level default system for character set assignment

- Syntax for specifying character sets and collations

- Affected functions and operations

- Unicode support

- The character sets and collations that are available, with notes

Character set issues affect data storage, but also communication between client programs and the MySQL server. If you want the client program to communicate with the server using a character set different from the default, you'll need to indicate which one. For example, to use the `utf8` Unicode character set, issue this statement after connecting to the server:

    SET NAMES 'utf8';

For more information about character set-related issues in client/server communication, see Section 10.4, “Connection Character Sets and Collations”.

## 10.1. Character Sets and Collations in General

A character set is a set of symbols and encodings. A collation is a set of rules for comparing characters in a character set. Let's make the distinction clear with an example of an imaginary character set.

Suppose that we have an alphabet with four letters: ‘`A`’, ‘`B`’, ‘`a`’, ‘`b`’. We give each letter a number: ‘`A`’ = 0, ‘`B`’ = 1, ‘`a`’ = 2, ‘`b`’ = 3. The letter ‘`A`’ is a symbol, the number 0 is the encoding for ‘`A`’, and the combination of all four letters and their encodings is a character set.

Suppose that we want to compare two string values, ‘`A`’ and ‘`B`’. The simplest way to do this is to look at the encodings: 0 for ‘`A`’ and 1 for ‘`B`’. Because 0 is less than 1, we say ‘`A`’ is less than ‘`B`’. What we've just done is apply a collation to our character set. The collation is a set of rules (only one rule in this case): “compare the encodings.” We call this simplest of all possible collations a binary collation.

But what if we want to say that the lowercase and uppercase letters are equivalent? Then we would have at least two rules: (1) treat the lowercase letters ‘`a`’ and ‘`b`’ as equivalent to ‘`A`’ and ‘`B`’; (2) then compare the encodings. We call this a case-insensitive collation. It's a little more complex than a binary collation.

In real life, most character sets have many characters: not just ‘`A`’ and ‘`B`’ but whole alphabets, sometimes multiple alphabets or eastern writing systems with thousands of characters, along with many special symbols and punctuation marks. Also in real life, most collations have many rules, not just for whether to distinguish lettercase, but also for whether to distinguish accents (an “accent” is a mark attached to a character as in German <squo;`Ö`’), and for multiple-character mappings (such as the rule that ‘`Ö`’ = ‘`OE`’ in one of the two German collations).

MySQL 4.1 can do these things for you:

- Store strings using a variety of character sets

- Compare strings using a variety of collations

- Mix strings with different character sets or collations in the same server, the same database, or even the same table

- Allow specification of character set and collation at any level

In these respects, not only is MySQL 4.1 far more flexible than MySQL 4.0, it also is far ahead of most other database management systems. However, to use these features effectively, you need to know what character sets and collations are available, how to change the defaults, and how they affect the behavior of string operators and functions.

## 10.2. Character Sets and Collations in MySQL

The MySQL server can support multiple character sets. To list the available character sets, use the `SHOW CHARACTER SET` statement. A partial listing follows. For more complete information, see Section 10.10, “Character Sets and Collations That MySQL Supports”.

    mysql> SHOW CHARACTER SET;
    +----------+-----------------------------+---------------------+--------+
    | Charset  | Description                 | Default collation   | Maxlen |
    +----------+-----------------------------+---------------------+--------+
    | big5     | Big5 Traditional Chinese    | big5_chinese_ci     |      2 |
    | dec8     | DEC West European           | dec8_swedish_ci     |      1 |
    | cp850    | DOS West European           | cp850_general_ci    |      1 |
    | hp8      | HP West European            | hp8_english_ci      |      1 |
    | koi8r    | KOI8-R Relcom Russian       | koi8r_general_ci    |      1 |
    | latin1   | cp1252 West European        | latin1_swedish_ci   |      1 |
    | latin2   | ISO 8859-2 Central European | latin2_general_ci   |      1 |
    | swe7     | 7bit Swedish                | swe7_swedish_ci     |      1 |
    | ascii    | US ASCII                    | ascii_general_ci    |      1 |
    | ujis     | EUC-JP Japanese             | ujis_japanese_ci    |      3 |
    | sjis     | Shift-JIS Japanese          | sjis_japanese_ci    |      2 |
    | hebrew   | ISO 8859-8 Hebrew           | hebrew_general_ci   |      1 |
    | tis620   | TIS620 Thai                 | tis620_thai_ci      |      1 |
    | euckr    | EUC-KR Korean               | euckr_korean_ci     |      2 |
    | koi8u    | KOI8-U Ukrainian            | koi8u_general_ci    |      1 |
    | gb2312   | GB2312 Simplified Chinese   | gb2312_chinese_ci   |      2 |
    | greek    | ISO 8859-7 Greek            | greek_general_ci    |      1 |
    | cp1250   | Windows Central European    | cp1250_general_ci   |      1 |
    | gbk      | GBK Simplified Chinese      | gbk_chinese_ci      |      2 |
    | latin5   | ISO 8859-9 Turkish          | latin5_turkish_ci   |      1 |
    ...

Any given character set always has at least one collation. It may have several collations. To list the collations for a character set, use the `SHOW COLLATION` statement. For example, to see the collations for the `latin1` (cp1252 West European) character set, use this statement to find those collation names that begin with `latin1`:

    mysql> SHOW COLLATION LIKE 'latin1%';
    +---------------------+---------+----+---------+----------+---------+
    | Collation           | Charset | Id | Default | Compiled | Sortlen |
    +---------------------+---------+----+---------+----------+---------+
    | latin1_german1_ci   | latin1  |  5 |         |          |       0 |
    | latin1_swedish_ci   | latin1  |  8 | Yes     | Yes      |       1 |
    | latin1_danish_ci    | latin1  | 15 |         |          |       0 |
    | latin1_german2_ci   | latin1  | 31 |         | Yes      |       2 |
    | latin1_bin          | latin1  | 47 |         | Yes      |       1 |
    | latin1_general_ci   | latin1  | 48 |         |          |       0 |
    | latin1_general_cs   | latin1  | 49 |         |          |       0 |
    | latin1_spanish_ci   | latin1  | 94 |         |          |       0 |
    +---------------------+---------+----+---------+----------+---------+

The `latin1` collations have the following meanings:

Collation | Meaning
`latin1_german1_ci` | German DIN-1
`latin1_swedish_ci` | Swedish/Finnish
`latin1_danish_ci` | Danish/Norwegian
`latin1_german2_ci` | German DIN-2
`latin1_bin` | Binary according to `latin1` encoding
`latin1_general_ci` | Multilingual (Western European)
`latin1_general_cs` | Multilingual (ISO Western European), case sensitive
`latin1_spanish_ci` | Modern Spanish

Collations have these general characteristics:

- Two different character sets cannot have the same collation.

- Each character set has one collation that is the default collation. For example, the default collation for `latin1` is `latin1_swedish_ci`. The output for `SHOW CHARACTER SET` indicates which collation is the default for each displayed character set.

- There is a convention for collation names: They start with the name of the character set with which they are associated, they usually include a language name, and they end with `_ci` (case insensitive), `_cs` (case sensitive), or `_bin` (binary).

## 10.3. Specifying Character Sets and Collations

There are default settings for character sets and collations at four levels: server, database, table, and column. The following description may appear complex, but it has been found in practice that multiple-level defaulting leads to natural and obvious results.

`CHARACTER SET` is used in clauses that specify a character set. `CHARSET` may be used as a synonym for `CHARACTER SET`.

### 10.3.1. Server Character Set and Collation

MySQL Server has a server character set and a server collation. These can be set at server startup and changed at runtime.

Initially, the server character set and collation depend on the options that you use when you start mysqld. You can use `--character-set-server` for the character set. Along with it, you can add `--collation-server` for the collation. If you don't specify a character set, that is the same as saying `--character-set-server=latin1`. If you specify only a character set (for example, `latin1`) but not a collation, that is the same as saying `--character-set-server=latin1` `--collation-server=latin1_swedish_ci` because `latin1_swedish_ci` is the default collation for `latin1`. Therefore, the following three commands all have the same effect:

    shell> mysqld
    shell> mysqld --character-set-server=latin1
    shell> mysqld --character-set-server=latin1 \
               --collation-server=latin1_swedish_ci

One way to change the settings is by recompiling. If you want to change the default server character set and collation when building from sources, use: `--with-charset` and `--with-collation` as arguments for configure. For example:

    shell> ./configure --with-charset=latin1

Or:

    shell> ./configure --with-charset=latin1 \
               --with-collation=latin1_german1_ci

Both mysqld and configure verify that the character set/collation combination is valid. If not, each program displays an error message and terminates.

The current server character set and collation can be determined from the values of the `character_set_server` and `collation_server` system variables. These variables can be changed at runtime.

### 10.3.2. Database Character Set and Collation

Every database has a database character set and a database collation. The `CREATE DATABASE` and `ALTER DATABASE` statements have optional clauses for specifying the database character set and collation:

    CREATE DATABASE db_name
        [[DEFAULT] CHARACTER SET charset_name]
        [[DEFAULT] COLLATE collation_name]

    ALTER DATABASE db_name
        [[DEFAULT] CHARACTER SET charset_name]
        [[DEFAULT] COLLATE collation_name]

All database options are stored in a text file named `db.opt` that can be found in the database directory.

The `CHARACTER SET` and `COLLATE` clauses make it possible to create databases with different character sets and collations on the same MySQL server.

Example:

    CREATE DATABASE db_name CHARACTER SET latin1 COLLATE latin1_swedish_ci;

MySQL chooses the database character set and database collation in the following manner:

- If both `CHARACTER SET `X`` and `COLLATE `Y`` were specified, then character set `X` and collation `Y`.

- If `CHARACTER SET `X`` was specified without `COLLATE`, then character set `X` and its default collation.

- If `COLLATE `Y`` was specified without `CHARACTER SET`, then the character set associated with `Y` and collation `Y`.

- Otherwise, the server character set and server collation.

The database character set and collation are used as default values if the table character set and collation are not specified in `CREATE TABLE` statements. They have no other purpose.

The character set and collation for the default database can be determined from the values of the `character_set_database` and `collation_database` system variables. The server sets these variables whenever the default database changes. If there is no default database, the variables have the same value as the corresponding server-level system variables, `character_set_server` and `collation_server`.

### 10.3.3. Table Character Set and Collation

Every table has a table character set and a table collation. The `CREATE TABLE` and `ALTER TABLE` statements have optional clauses for specifying the table character set and collation:

    CREATE TABLE tbl_name (column_list)
        [[DEFAULT] CHARACTER SET charset_name] [COLLATE collation_name]]

    ALTER TABLE tbl_name
        [[DEFAULT] CHARACTER SET charset_name] [COLLATE collation_name]

Example:

    CREATE TABLE t1 ( ... ) CHARACTER SET latin1 COLLATE latin1_danish_ci;

MySQL chooses the table character set and collation in the following manner:

- If both `CHARACTER SET `X`` and `COLLATE `Y`` were specified, then character set `X` and collation `Y`.

- If `CHARACTER SET `X`` was specified without `COLLATE`, then character set `X` and its default collation.

- If `COLLATE `Y`` was specified without `CHARACTER SET`, then the character set associated with `Y` and collation `Y`.

- Otherwise, the database character set and collation.

The table character set and collation are used as default values if the column character set and collation are not specified in individual column definitions. The table character set and collation are MySQL extensions; there are no such things in standard SQL.

### 10.3.4. Column Character Set and Collation

Every “character” column (that is, a column of type `CHAR`, `VARCHAR`, or `TEXT`) has a column character set and a column collation. Column definition syntax has optional clauses for specifying the column character set and collation:

    col_name {CHAR | VARCHAR | TEXT} (col_length)
        [CHARACTER SET charset_name] [COLLATE collation_name]

Example:

    CREATE TABLE Table1
    (
        column1 VARCHAR(5) CHARACTER SET latin1 COLLATE latin1_german1_ci
    );

MySQL chooses the column character set and collation in the following manner:

- If both `CHARACTER SET `X`` and `COLLATE `Y`` were specified, then character set `X` and collation `Y` are used.

- If `CHARACTER SET `X`` was specified without `COLLATE`, then character set `X` and its default collation are used.

- If `COLLATE `Y`` was specified without `CHARACTER SET`, then the character set associated with `Y` and collation `Y`.

- Otherwise, the table character set and collation are used.

The `CHARACTER SET` and `COLLATE` clauses are standard SQL.

### 10.3.5. Character String Literal Character Set and Collation

Every character string literal has a character set and a collation.

A character string literal may have an optional character set introducer and `COLLATE` clause:

    [_charset_name]'string' [COLLATE collation_name]

Examples:

    SELECT 'string';
    SELECT _latin1'string';
    SELECT _latin1'string' COLLATE latin1_danish_ci;

For the simple statement `SELECT '`string`'`, the string has the character set and collation defined by the `character_set_connection` and `collation_connection` system variables.

The `_`charset_name`` expression is formally called an introducer. It tells the parser, “the string that is about to follow uses character set `X`.” Because this has confused people in the past, we emphasize that an introducer does not cause any conversion; it is strictly a signal that does not change the string's value. An introducer is also legal before standard hex literal and numeric hex literal notation (`x'`literal`'` and `0x`nnnn``)>.

Examples:

    SELECT _latin1 x'AABBCC';
    SELECT _latin1 0xAABBCC;

MySQL determines a literal's character set and collation in the following manner:

- If both `_X` and `COLLATE `Y`` were specified, then character set `X` and collation `Y` are used.

- If `_X` is specified but `COLLATE` is not specified, then character set `X` and its default collation are used.

- Otherwise, the character set and collation given by the `character_set_connection` and `collation_connection` system variables are used.

Examples:

- A string with `latin1` character set and `latin1_german1_ci` collation:

      SELECT _latin1'Müller' COLLATE latin1_german1_ci;

- A string with `latin1` character set and its default collation (that is, `latin1_swedish_ci`):

      SELECT _latin1'Müller';

- A string with the connection default character set and collation:

      SELECT 'Müller';

Character set introducers and the `COLLATE` clause are implemented according to standard SQL specifications.

An introducer indicates the character set for the following string, but does not change now how the parser performs escape processing within the string. Escapes are always interpreted by the parser according to the character set given by `character_set_connection`.

The following examples show that escape processsing occurs using `character_set_connection` even in the presence of an introducer. The examples use `SET NAMES` (which changes `character_set_connection`, as discussed in Section 10.4, “Connection Character Sets and Collations”), and display the resulting strings using the `HEX()` function so that the exact string contents can be seen.

Example 1:

    mysql> SET NAMES latin1;
    Query OK, 0 rows affected (0.01 sec)

    mysql> SELECT HEX('à\n'), HEX(_sjis'à\n');
    +------------+-----------------+
    | HEX('à\n') | HEX(_sjis'à\n') |
    +------------+-----------------+
    | E00A       | E00A            |
    +------------+-----------------+
    1 row in set (0.00 sec)

Here, ‘`à`’ (hex value `E0`) is followed by ‘`\n`’, the escape sequence for newline. The escape sequence is interpreted using the `character_set_connection` value of `latin1` to produce a literal newline (hex value `0A`). This happens even for the second string. That is, the introducer of `_sjis` does not affect the parser's escape processing.

Example 2:

    mysql> SET NAMES sjis;
    Query OK, 0 rows affected (0.00 sec)

    mysql> SELECT HEX('à\n'), HEX(_latin1'à\n');
    +------------+-------------------+
    | HEX('à\n') | HEX(_latin1'à\n') |
    +------------+-------------------+
    | E05C6E     | E05C6E            |
    +------------+-------------------+
    1 row in set (0.04 sec)

Here, `character_set_connection` is `sjis`, a character set in which the sequence of ‘`à`’ followed by ‘`\`’ (hex values `05` and `5C`) is a valid multi-byte character. Hence, the first two bytes of the string are interpreted as a single `sjis` character, and the ‘`\`’ is not intrepreted as an escape character. The following ‘`n`’ (hex value `6E`) is not interpreted as part of an escape sequence. This is true even for the second string; the introducer of `_latin1` does not affect escape processing.

### 10.3.6. National Character Set

Before MySQL 4.1, `NCHAR` and `CHAR` were synonymous. Standard SQL defines `NCHAR` or `NATIONAL CHAR` as a way to indicate that a `CHAR` column should use some predefined character set. MySQL 4.1 and up uses `utf8` as that predefined character set. For example, these data type declarations are equivalent:

    CHAR(10) CHARACTER SET utf8
    NATIONAL CHARACTER(10)
    NCHAR(10)

As are these:

    VARCHAR(10) CHARACTER SET utf8
    NATIONAL VARCHAR(10)
    NCHAR VARCHAR(10)
    NATIONAL CHARACTER VARYING(10)
    NATIONAL CHAR VARYING(10)

You can use `N'`literal`'` to create a string in the national character set. These two statements are equivalent:

    SELECT N'some text';
    SELECT _utf8'some text';

### 10.3.7. Examples of Character Set and Collation Assignment

The following examples show how MySQL determines default character set and collation values.

Example 1: Table and Column Definition

    CREATE TABLE t1
    (
        c1 CHAR(10) CHARACTER SET latin1 COLLATE latin1_german1_ci
    ) DEFAULT CHARACTER SET latin2 COLLATE latin2_bin;

Here we have a column with a `latin1` character set and a `latin1_german1_ci` collation. The definition is explicit, so that's straightforward. Notice that there is no problem with storing a `latin1` column in a `latin2` table.

Example 2: Table and Column Definition

    CREATE TABLE t1
    (
        c1 CHAR(10) CHARACTER SET latin1
    ) DEFAULT CHARACTER SET latin1 COLLATE latin1_danish_ci;

This time we have a column with a `latin1` character set and a default collation. Although it might seem natural, the default collation is not taken from the table level. Instead, because the default collation for `latin1` is always `latin1_swedish_ci`, column `c1` has a collation of `latin1_swedish_ci` (not `latin1_danish_ci`).

Example 3: Table and Column Definition

    CREATE TABLE t1
    (
        c1 CHAR(10)
    ) DEFAULT CHARACTER SET latin1 COLLATE latin1_danish_ci;

We have a column with a default character set and a default collation. In this circumstance, MySQL checks the table level to determine the column character set and collation. Consequently, the character set for column `c1` is `latin1` and its collation is `latin1_danish_ci`.

Example 4: Database, Table, and Column Definition

    CREATE DATABASE d1
        DEFAULT CHARACTER SET latin2 COLLATE latin2_czech_ci;
    USE d1;
    CREATE TABLE t1
    (
        c1 CHAR(10)
    );

We create a column without specifying its character set and collation. We're also not specifying a character set and a collation at the table level. In this circumstance, MySQL checks the database level to determine the table settings, which thereafter become the column settings.) Consequently, the character set for column `c1` is `latin2` and its collation is `latin2_czech_ci`.

### 10.3.8. Compatibility with Other DBMSs

For MaxDB compatibility these two statements are the same:

    CREATE TABLE t1 (f1 CHAR(N) UNICODE);
    CREATE TABLE t1 (f1 CHAR(N) CHARACTER SET ucs2);

## 10.4. Connection Character Sets and Collations

Several character set and collation system variables relate to a client's interaction with the server. Some of these have been mentioned in earlier sections:

- The server character set and collation can be determined from the values of the `character_set_server` and `collation_server` system variables.

- The character set and collation of the default database can be determined from the values of the `character_set_database` and `collation_database` system variables.

Additional character set and collation system variables are involved in handling traffic for the connection between a client and the server. Every client has connection-related character set and collation system variables.

Consider what a “connection” is: It's what you make when you connect to the server. The client sends SQL statements, such as queries, over the connection to the server. The server sends responses, such as result sets, over the connection back to the client. This leads to several questions about character set and collation handling for client connections, each of which can be answered in terms of system variables:

- What character set is the statement in when it leaves the client?

  The server takes the `character_set_client` system variable to be the character set in which statements are sent by the client.

- What character set should the server translate a statement to after receiving it?

  For this, the server uses the `character_set_connection` and `collation_connection` system variables. It converts statements sent by the client from `character_set_client` to `character_set_connection` (except for string literals that have an introducer such as `_latin1` or `_utf8`). `collation_connection` is important for comparisons of literal strings. For comparisons of strings with column values, `collation_connection` does not matter because columns have their own collation, which has a higher collation precedence.

- What character set should the server translate to before shipping result sets or error messages back to the client?

  The `character_set_results` system variable indicates the character set in which the server returns query results to the client. This includes result data such as column values, and result metadata such as column names.

You can fine-tune the settings for these variables, or you can depend on the defaults (in which case, you can skip the rest of this section).

There are two statements that affect the connection character sets:

    SET NAMES 'charset_name'
    SET CHARACTER SET charset_name

`SET NAMES` indicates what character set the client will use to send SQL statements to the server. Thus, `SET NAMES 'cp1251'` tells the server “future incoming messages from this client are in character set `cp1251`.” It also specifies the character set that the server should use for sending results back to the client. (For example, it indicates what character set to use for column values if you use a `SELECT` statement.)

A `SET NAMES '`x`'` statement is equivalent to these three statements:

    SET character_set_client = x;
    SET character_set_results = x;
    SET character_set_connection = x;

Setting `character_set_connection` to `x` also sets `collation_connection` to the default collation for `x`. It is not necessary to set that collation explicitly. To specify a particular collation for the character sets, use the optional `COLLATE` clause:

    SET NAMES 'charset_name' COLLATE 'collation_name'

`SET CHARACTER SET` is similar to `SET NAMES` but sets `character_set_connection` and `collation_connection` to `character_set_database` and `collation_database`. A `SET CHARACTER SET `x`` statement is equivalent to these three statements:

    SET character_set_client = x;
    SET character_set_results = x;
    SET collation_connection = @@collation_database;

Setting `collation_connection` also sets `character_set_connection` to the character set associated with the collation (equivalent to executing `SET character_set_connection = @@character_set_database`). It is not necessary to set `character_set_connection` explicitly.

When a client connects, it sends to the server the name of the character set that it wants to use. The server uses the name to set the `character_set_client`, `character_set_results`, and `character_set_connection` system variables. In effect, the server performs a `SET NAMES` operation using the character set name.

With the mysql client, it is not necessary to execute `SET NAMES` every time you start up if you want to use a character set different from the default. You can add the `--default-character-set` option setting to your mysql statement line, or in your option file. For example, the following option file setting changes the three character set variables set to `koi8r` each time you invoke mysql:

    [mysql]
    default-character-set=koi8r

Example: Suppose that `column1` is defined as `CHAR(5) CHARACTER SET latin2`. If you do not say `SET NAMES` or `SET CHARACTER SET`, then for `SELECT column1 FROM t`, the server sends back all the values for `column1` using the character set that the client specified when it connected. On the other hand, if you say `SET NAMES 'latin1'` or `SET CHARACTER SET latin1` before issuing the `SELECT` statement, the server converts the `latin2` values to `latin1` just before sending results back. Conversion may be lossy if there are characters that are not in both character sets.

If you do not want the server to perform any conversion of result sets, set `character_set_results` to `NULL`:

    SET character_set_results = NULL;

Note: Currently, UCS-2 cannot be used as a client character set, which means that `SET NAMES 'ucs2'` does not work.

To see the values of the character set and collation system variables that apply to your connection, use these statements:

    SHOW VARIABLES LIKE 'character_set%';
    SHOW VARIABLES LIKE 'collation%';

## 10.5. Collation Issues

The following sections various aspects of character set collations.

### 10.5.1. Using `COLLATE` in SQL Statements

With the `COLLATE` clause, you can override whatever the default collation is for a comparison. `COLLATE` may be used in various parts of SQL statements. Here are some examples:

- With `ORDER BY`:

      SELECT k
      FROM t1
      ORDER BY k COLLATE latin1_german2_ci;

- With `AS`:

      SELECT k COLLATE latin1_german2_ci AS k1
      FROM t1
      ORDER BY k1;

- With `GROUP BY`:

      SELECT k
      FROM t1
      GROUP BY k COLLATE latin1_german2_ci;

- With aggregate functions:

      SELECT MAX(k COLLATE latin1_german2_ci)
      FROM t1;

- With `DISTINCT`:

      SELECT DISTINCT k COLLATE latin1_german2_ci
      FROM t1;

- With `WHERE`:

           SELECT *
           FROM t1
           WHERE _latin1 'Müller' COLLATE latin1_german2_ci = k;

           SELECT *
           FROM t1
           WHERE k LIKE _latin1 'Müller' COLLATE latin1_german2_ci;

- With `HAVING`:

      SELECT k
      FROM t1
      GROUP BY k
      HAVING k = _latin1 'Müller' COLLATE latin1_german2_ci;

### 10.5.2. `COLLATE` Clause Precedence

The `COLLATE` clause has high precedence (higher than `||`), so the following two expressions are equivalent:

    x || y COLLATE z
    x || (y COLLATE z)

### 10.5.3. `BINARY` Operator

The `BINARY` operator casts the string following it to a binary string. This is an easy way to force a comparison to be done byte by byte rather than character by character. `BINARY` also causes trailing spaces to be significant.

    mysql> SELECT 'a' = 'A';
            -> 1
    mysql> SELECT BINARY 'a' = 'A';
            -> 0
    mysql> SELECT 'a' = 'a ';
            -> 1
    mysql> SELECT BINARY 'a' = 'a ';
            -> 0

`BINARY `str`` is shorthand for `CAST(`str` AS BINARY)`.

The `BINARY` attribute in character column definitions has a different effect. A character column defined with the `BINARY` attribute is assigned the binary collation of the column's character set. Every character set has a binary collation. For example, the binary collation for the `latin1` character set is `latin1_bin`, so if the table default character set is `latin1`, these two column definitions are equivalent:

    CHAR(10) BINARY
    CHAR(10) CHARACTER SET latin1 COLLATE latin1_bin

The effect of `BINARY` as a column attribute differs from its effect prior to MySQL 4.1. Formerly, `BINARY` resulted in a column that was treated as a binary string. A binary string is a string of bytes that has no character set or collation, which differs from a non-binary character string that has a binary collation. For both types of strings, comparisons are based on the numeric values of the string unit, but for non-binary strings the unit is the character and some character sets allow multi-byte characters. Section 11.4.2, “The `BINARY` and `VARBINARY` Types”.

The use of `CHARACTER SET binary` in the definition of a `CHAR`, `VARCHAR`, or `TEXT` column causes the column to be treated as a binary data type. For example, the following pairs of definitions are equivalent:

    CHAR(10) CHARACTER SET binary
    BINARY(10)

    VARCHAR(10) CHARACTER SET binary
    VARBINARY(10)

    TEXT CHARACTER SET binary
    BLOB

### 10.5.4. Some Special Cases Where the Collation Determination Is Tricky

In the great majority of statements, it is obvious what collation MySQL uses to resolve a comparison operation. For example, in the following cases, it should be clear that the collation is the collation of column `x`:

    SELECT x FROM T ORDER BY x;
    SELECT x FROM T WHERE x = x;
    SELECT DISTINCT x FROM T;

However, when multiple operands are involved, there can be ambiguity. For example:

    SELECT x FROM T WHERE x = 'Y';

Should this query use the collation of the column `x`, or of the string literal `'Y'`?

Standard SQL resolves such questions using what used to be called “coercibility” rules. Basically, this means: Both `x` and `'Y'` have collations, so which collation takes precedence? This can be difficult to resolve, but the following rules cover most situations:

- An explicit `COLLATE` clause has a coercibility of 0. (Not coercible at all.)

- The concatenation of two strings with different collations has a coercibility of 1.

- A column's collation has a coercibility of 2.

- A “system constant” (the string returned by functions such as `USER()` or `VERSION()`) has a coercibility of 3.

- A literal's collation has a coercibility of 4.

- `NULL` or an expression that is derived from `NULL` has a coercibility of 5.

The preceding coercibility values are current as of MySQL 4.1.11. See the note later in this section for additional version-related information.

Those rules resolve ambiguities in the following manner:

- Use the collation with the lowest coercibility value.

- If both sides have the same coercibility, then it is an error if the collations aren't the same.

Examples:

`column1 = 'A'` | Use collation of `column1`
`column1 = 'A' COLLATE x` | Use collation of `'A'`
`column1 COLLATE x = 'A' COLLATE y` | Error

The `COERCIBILITY()` function can be used to determine the coercibility of a string expression:

    mysql> SELECT COERCIBILITY('A' COLLATE latin1_swedish_ci);
            -> 0
    mysql> SELECT COERCIBILITY(VERSION());
            -> 3
    mysql> SELECT COERCIBILITY('A');
            -> 4

See Section 12.9.3, “Information Functions”.

Before MySQL 4.1.11, there is no system constant or ignorable coercibility. Functions such as `USER()` have a coercibility of 2 rather than 3, and literals have a coercibility of 3 rather than 4.

### 10.5.5. Collations Must Be for the Right Character Set

Each character set has one or more collations, but each collation is associated with one and only one character set. Therefore, the following statement causes an error message because the `latin2_bin` collation is not legal with the `latin1` character set:

    mysql> SELECT _latin1 'x' COLLATE latin2_bin;
    ERROR 1253 (42000): COLLATION 'latin2_bin' is not valid
    for CHARACTER SET 'latin1'

In some cases, expressions that worked before MySQL 4.1 fail in early versions of MySQL 4.1 if you do not take character set and collation into account. For example, before 4.1, this statement works as is:

    mysql> SELECT SUBSTRING_INDEX(USER(),'@',1);
    +-------------------------------+
    | SUBSTRING_INDEX(USER(),'@',1) |
    +-------------------------------+
    | root                          |
    +-------------------------------+

The statement also works as is in MySQL 4.1 as of 4.1.8: In MySQL 4.1, usernames are stored using the `utf8` character set (see Section 10.8, “UTF-8 for Metadata”). The literal string `'@'` has the server character set (`latin1` by default). Although the character sets are different, MySQL can coerce the `latin1` string to the character set (and collation) of `USER()` without data loss. It does so, performs the substring operation, and returns a result that has a character set of `utf8`.

However, in versions of MySQL 4.1 before 4.1.8, the statement fails:

    mysql> SELECT SUBSTRING_INDEX(USER(),'@',1);
    ERROR 1267 (HY000): Illegal mix of collations
    (utf8_general_ci,IMPLICIT) and (latin1_swedish_ci,COERCIBLE)
    for operation 'substr_index'

This happens because the automatic character set conversion of `'@'` does not occur and the string operands have different character sets (and thus different collations):

    mysql> SELECT COLLATION(USER()), COLLATION('@');
    +-------------------+-------------------+
    | COLLATION(USER()) | COLLATION('@')    |
    +-------------------+-------------------+
    | utf8_general_ci   | latin1_swedish_ci |
    +-------------------+-------------------+

One way to deal with this is to upgrade to MySQL 4.1.8 or later. If that is not possible, you can tell MySQL to interpret the literal string as `utf8`:

    mysql> SELECT SUBSTRING_INDEX(USER(),_utf8'@',1);
    +------------------------------------+
    | SUBSTRING_INDEX(USER(),_utf8'@',1) |
    +------------------------------------+
    | root                               |
    +------------------------------------+

Another way is to change the connection character set and collation to `utf8`. You can do that with `SET NAMES 'utf8'` or by setting the `character_set_connection` and `collation_connection` system variables directly.

### 10.5.6. An Example of the Effect of Collation

Suppose that column `X` in table `T` has these `latin1` column values:

    Muffler
    Müller
    MX Systems
    MySQL

Suppose also that the column values are retrieved using the following statement:

    SELECT X FROM T ORDER BY X COLLATE collation_name;

The following table shows the resulting order of the values if we use `ORDER BY` with different collations:

`latin1_swedish_ci` | `latin1_german1_ci` | `latin1_german2_ci`
Muffler | Muffler | Müller
MX Systems | Müller | Muffler
Müller | MX Systems | MX Systems
MySQL | MySQL | MySQL

The character that causes the different sort orders in this example is the U with two dots over it (`ü`), which the Germans call “U-umlaut.”

- The first column shows the result of the `SELECT` using the Swedish/Finnish collating rule, which says that U-umlaut sorts with Y.

- The second column shows the result of the `SELECT` using the German DIN-1 rule, which says that U-umlaut sorts with U.

- The third column shows the result of the `SELECT` using the German DIN-2 rule, which says that U-umlaut sorts with UE.

## 10.6. Operations Affected by Character Set Support

This section describes operations that take character set information into account as of MySQL 4.1.

### 10.6.1. Result Strings

MySQL has many operators and functions that return a string. This section answers the question: What is the character set and collation of such a string?

For simple functions that take string input and return a string result as output, the output's character set and collation are the same as those of the principal input value. For example, `UPPER(`X`)` returns a string whose character string and collation are the same as that of `X`. The same applies for `INSTR()`, `LCASE()`, `LOWER()`, `LTRIM()`, `MID()`, `REPEAT()`, `REPLACE()`, `REVERSE()`, `RIGHT()`, `RPAD()`, `RTRIM()`, `SOUNDEX()`, `SUBSTRING()`, `TRIM()`, `UCASE()`, and `UPPER()`.

Note: The `REPLACE()` function, unlike all other functions, always ignores the collation of the string input and performs a case-sensitive comparison.

If a string input or function result is a binary string, the string has no character set or collation. This can be check by using the `CHARSET()` and `COLLATION()` functions, both of which return `binary` to indicate that their argument is a binary string:

    mysql> SELECT CHARSET(BINARY 'a'), COLLATION(BINARY 'a');
    +---------------------+-----------------------+
    | CHARSET(BINARY 'a') | COLLATION(BINARY 'a') |
    +---------------------+-----------------------+
    | binary              | binary                |
    +---------------------+-----------------------+

For operations that combine multiple string inputs and return a single string output, the “aggregation rules” of standard SQL apply for determining the collation of the result:

- If an explicit `COLLATE `X`` occurs, use `X`.

- If explicit `COLLATE `X`` and `COLLATE `Y`` occur, raise an error.

- Otherwise, if all collations are `X`, use `X`.

- Otherwise, the result has no collation.

For example, with `CASE ... WHEN a THEN b WHEN b THEN c COLLATE `X` END`, the resulting collation is `X`. The same applies for `UNION`, `||`, `CONCAT()`, `ELT()`, `GREATEST()`, `IF()`, and `LEAST()`.

For operations that convert to character data, the character set and collation of the strings that result from the operations are defined by the `character_set_connection` and `collation_connection` system variables. This applies only to `CAST()`, `CHAR()`, `CONV()`, `FORMAT()`, `HEX()`, and `SPACE()`.

If you are uncertain about the character set or collation of the result returned by a string function, you can use the `CHARSET()` or `COLLATE()` function to find out:

    mysql> SELECT USER(), CHARSET(USER()), COLLATION(USER());
    +----------------+-----------------+-------------------+
    | USER()         | CHARSET(USER()) | COLLATION(USER()) |
    +----------------+-----------------+-------------------+
    | test@localhost | utf8            | utf8_general_ci   |
    +----------------+-----------------+-------------------+

### 10.6.2. `CONVERT()` and `CAST()`

`CONVERT()` provides a way to convert data between different character sets. The syntax is:

    CONVERT(expr USING transcoding_name)

In MySQL, transcoding names are the same as the corresponding character set names.

Examples:

    SELECT CONVERT(_latin1'Müller' USING utf8);
    INSERT INTO utf8table (utf8column)
        SELECT CONVERT(latin1field USING utf8) FROM latin1table;

`CONVERT(... USING ...)` is implemented according to the standard SQL specification.

You may also use `CAST()` to convert a string to a different character set. The syntax is:

    CAST(character_string AS character_data_type CHARACTER SET charset_name)

Example:

    SELECT CAST(_latin1'test' AS CHAR CHARACTER SET utf8);

If you use `CAST()` without specifying `CHARACTER SET`, the resulting character set and collation are defined by the `character_set_connection` and `collation_connection` system variables. If you use `CAST()` with `CHARACTER SET X`, the resulting character set and collation are `X` and the default collation of `X`.

You may not use a `COLLATE` clause inside a `CAST()`, but you may use it outside. That is, `CAST(... COLLATE ...)` is illegal, but `CAST(...) COLLATE ...` is legal.

Example:

    SELECT CAST(_latin1'test' AS CHAR CHARACTER SET utf8) COLLATE utf8_bin;

### 10.6.3. `SHOW` Statements and `INFORMATION_SCHEMA`

Several `SHOW` statements are added or modified in MySQL 4.1 to provide additional character set information. `SHOW CHARACTER SET`, `SHOW COLLATION`, and `SHOW CREATE DATABASE` are new. `SHOW CREATE TABLE` and `SHOW COLUMNS` are modified. These statements are described here briefly. For more information, see Section 13.5.4, “`SHOW` Syntax”.

The `SHOW CHARACTER SET` command shows all available character sets. It takes an optional `LIKE` clause that indicates which character set names to match. For example:

    mysql> SHOW CHARACTER SET LIKE 'latin%';
    +---------+-----------------------------+-------------------+--------+
    | Charset | Description                 | Default collation | Maxlen |
    +---------+-----------------------------+-------------------+--------+
    | latin1  | cp1252 West European        | latin1_swedish_ci |      1 |
    | latin2  | ISO 8859-2 Central European | latin2_general_ci |      1 |
    | latin5  | ISO 8859-9 Turkish          | latin5_turkish_ci |      1 |
    | latin7  | ISO 8859-13 Baltic          | latin7_general_ci |      1 |
    +---------+-----------------------------+-------------------+--------+

The output from `SHOW COLLATION` includes all available character sets. It takes an optional `LIKE` clause that indicates which collation names to match. For example:

    mysql> SHOW COLLATION LIKE 'latin1%';
    +-------------------+---------+----+---------+----------+---------+
    | Collation         | Charset | Id | Default | Compiled | Sortlen |
    +-------------------+---------+----+---------+----------+---------+
    | latin1_german1_ci | latin1  |  5 |         |          |       0 |
    | latin1_swedish_ci | latin1  |  8 | Yes     | Yes      |       0 |
    | latin1_danish_ci  | latin1  | 15 |         |          |       0 |
    | latin1_german2_ci | latin1  | 31 |         | Yes      |       2 |
    | latin1_bin        | latin1  | 47 |         | Yes      |       0 |
    | latin1_general_ci | latin1  | 48 |         |          |       0 |
    | latin1_general_cs | latin1  | 49 |         |          |       0 |
    | latin1_spanish_ci | latin1  | 94 |         |          |       0 |
    +-------------------+---------+----+---------+----------+---------+

`SHOW CREATE DATABASE` displays the `CREATE DATABASE` statement that creates a given database:

    mysql> SHOW CREATE DATABASE test;
    +----------+-----------------------------------------------------------------+
    | Database | Create Database                                                 |
    +----------+-----------------------------------------------------------------+
    | test     | CREATE DATABASE `test` /*!40100 DEFAULT CHARACTER SET latin1 */ |
    +----------+-----------------------------------------------------------------+

If no `COLLATE` clause is shown, the default collation for the character set applies.

`SHOW CREATE TABLE` is similar, but displays the `CREATE TABLE` statement to create a given table. The column definitions indicate any character set specifications, and the table options include character set information.

The `SHOW COLUMNS` statement displays the collations of a table's columns when invoked as `SHOW FULL COLUMNS`. Columns with `CHAR`, `VARCHAR`, or `TEXT` data types have collations. Numeric and other non-character types have no collation (indicated by `NULL` as the `Collation` value). For example:

    mysql> SHOW FULL COLUMNS FROM person\G
    *************************** 1. row ***************************
         Field: id
          Type: smallint(5) unsigned
     Collation: NULL
          Null: NO
           Key: PRI
       Default: NULL
         Extra: auto_increment
    Privileges: select,insert,update,references
       Comment:
    *************************** 2. row ***************************
         Field: name
          Type: char(60)
     Collation: latin1_swedish_ci
          Null: NO
           Key:
       Default:
         Extra:
    Privileges: select,insert,update,references
       Comment:

The character set is not part of the display but is implied by the collation name.

## 10.7. Unicode Support

As of MySQL version 4.1, there are two new character sets for storing Unicode data:

- `ucs2`, the UCS-2 Unicode character set.

- `utf8`, the UTF-8 encoding of the Unicode character set.

In UCS-2 (binary Unicode representation), every character is represented by a two-byte Unicode code with the most significant byte first. For example: `LATIN CAPITAL LETTER A` has the code `0x0041` and it is stored as a two-byte sequence: `0x00 0x41`. `CYRILLIC SMALL LETTER YERU` (Unicode `0x044B`) is stored as a two-byte sequence: `0x04 0x4B`. For Unicode characters and their codes, please refer to the Unicode Home Page.

Currently, UCS-2 cannot be used as a client character set. That means that `SET NAMES 'ucs2'` does not work.

UTF-8 (Unicode Transform representation) is an alternative way to store Unicode data. It is implemented according to RFC 3629. The idea of UTF-8 is that various Unicode characters are encoded using byte sequences of different lengths:

- Basic Latin letters, digits, and punctuation signs use one byte.

- Most European and Middle East script letters fit into a two-byte sequence: extended Latin letters (with tilde, macron, acute, grave and other accents), Cyrillic, Greek, Armenian, Hebrew, Arabic, Syriac, and others.

- Korean, Chinese, and Japanese ideographs use three-byte sequences.

RFC 3629 describes encoding sequences that take from one to four bytes. Currently, MySQL support for UTF-8 does not include four-byte sequences. (An older standard for UTF-8 encoding is given by RFC 2279, which describes UTF-8 sequences that take from one to six bytes. RFC 3629 renders RFC 2279 obsolete; for this reason, sequences with five and six bytes are no longer used.)

Tip: To save space with UTF-8, use `VARCHAR` instead of `CHAR`. Otherwise, MySQL must reserve three bytes for each character in a `CHAR CHARACTER SET utf8` column because that is the maximum possible length. For example, MySQL must reserve 30 bytes for a `CHAR(10) CHARACTER SET utf8` column.

## 10.8. UTF-8 for Metadata

Metadata is “the data about the data.” Anything that describes the database — as opposed to being the contents of the database — is metadata. Thus column names, database names, usernames, version names, and most of the string results from `SHOW` are metadata.

Representation of metadata must satisfy these requirements:

- All metadata must be in the same character set. Otherwise, `SHOW` wouldn't work properly because different rows in the same column would be in different character sets.

- Metadata must include all characters in all languages. Otherwise, users would not be able to name columns and tables using their own languages.

To satisfy both requirements, MySQL stores metadata in a Unicode character set, namely UTF-8. This does not cause any disruption if you never use accented or non-Latin characters. But if you do, you should be aware that metadata is in UTF-8.

The metadata requirements mean that the return values of the `USER()`, `CURRENT_USER()`, `SESSION_USER()`, `SYSTEM_USER()`, `DATABASE()`, and `VERSION()` functions have the UTF-8 character set by default.

The server sets the `character_set_system` system variable to the name of the metadata character set:

    mysql> SHOW VARIABLES LIKE 'character_set_system';
    +----------------------+-------+
    | Variable_name        | Value |
    +----------------------+-------+
    | character_set_system | utf8  |
    +----------------------+-------+

Storage of metadata using Unicode does not mean that the server returns headers of columns and the results of `DESCRIBE` functions in the `character_set_system` character set by default. When you use `SELECT column1 FROM t`, the name `column1` itself is returned from the server to the client in the character set determined by the value of the `character_set_results` system variable, which has a default value of `latin1`. If you want the server to pass metadata results back in a different character set, use the `SET NAMES` statement to force the server to perform character set conversion. `SET NAMES` sets the `character_set_results` and other related system variables. (See Section 10.4, “Connection Character Sets and Collations”.) Alternatively, a client program can perform the conversion after receiving the result from the server. It is more efficient for the client perform the the conversion, but this option is not available for many clients until late in the MySQL 4.x product cycle.

If `character_set_results` is set to `NULL`, no conversion is performed and the server returns metadata using its original character set (the set indicated by `character_set_system`).

Beginning with MySQL 4.1.1, error messages returned from the server to the client are converted to the client character set automatically, as with metadata.

If you are using (for example) the `USER()` function for comparison or assignment within a single statement, don't worry. MySQL performs some automatic conversion for you.

    SELECT * FROM Table1 WHERE USER() = latin1_column;

This works because the contents of `latin1_column` are automatically converted to UTF-8 before the comparison.

    INSERT INTO Table1 (latin1_column) SELECT USER();

This works because the contents of `USER()` are automatically converted to `latin1` before the assignment. Automatic conversion is not fully implemented yet, but should work correctly in a later version.

Although automatic conversion is not in the SQL standard, the SQL standard document does say that every character set is (in terms of supported characters) a “subset” of Unicode. Because it is a well-known principle that “what applies to a superset can apply to a subset,” we believe that a collation for Unicode can apply for comparisons with non-Unicode strings.

## 10.9. Upgrading Character Sets from MySQL 4.0

What about upgrading from older versions of MySQL? MySQL 4.1 is almost upward compatible with MySQL 4.0 and earlier for the simple reason that almost all the features are new, so there's nothing in earlier versions to conflict with. However, there are some differences and a few things to be aware of.

It is important to note that the “MySQL 4.0 character set” contains both character set and collation information in one single entity. Beginning in MySQL 4.1, character sets and collations are separate entities. Though each collation corresponds to a particular character set, the two are not bundled together.

There is a special treatment of national character sets in MySQL 4.1. `NCHAR` is not the same as `CHAR`, and `N'...'` literals are not the same as `'...'` literals.

Finally, there is a different file format for storing information about character sets and collations. Make sure that you have reinstalled the `/share/mysql/charsets/` directory containing the new configuration files.

If you want to start mysqld from a 4.1.x distribution with data created by MySQL 4.0, you should start the server with the same character set and collation. In this case, you won't need to reindex your data.

There are two ways to do so:

    shell> ./configure --with-charset=... --with-collation=...
    shell> ./mysqld --default-character-set=... --default-collation=...

If you used mysqld with, for example, the MySQL 4.0 `danish` character set, you should use the `latin1` character set and the `latin1_danish_ci` collation:

    shell> ./configure --with-charset=latin1 \
               --with-collation=latin1_danish_ci
    shell> ./mysqld --default-character-set=latin1 \
               --default-collation=latin1_danish_ci

Use the table shown in Section 10.9.1, “4.0 Character Sets and Corresponding 4.1 Character Set/Collation Pairs”, to find old 4.0 character set names and their 4.1 character set/collation pair equivalents.

If you have non-`latin1` data stored in a 4.0 `latin1` table and want to convert the table column definitions to reflect the actual character set of the data, use the instructions in Section 10.9.2, “Converting 4.0 Character Columns to 4.1 Format”.

### 10.9.1. 4.0 Character Sets and Corresponding 4.1 Character Set/Collation Pairs

ID | 4.0 Character Set | 4.1 Character Set | 4.1 Collation
1 | `big5` | `big5` | `big5_chinese_ci`
2 | `czech` | `latin2` | `latin2_czech_ci`
3 | `dec8` | `dec8` | `dec8_swedish_ci`
4 | `dos` | `cp850` | `cp850_general_ci`
5 | `german1` | `latin1` | `latin1_german1_ci`
6 | `hp8` | `hp8` | `hp8_english_ci`
7 | `koi8_ru` | `koi8r` | `koi8r_general_ci`
8 | `latin1` | `latin1` | `latin1_swedish_ci`
9 | `latin2` | `latin2` | `latin2_general_ci`
10 | `swe7` | `swe7` | `swe7_swedish_ci`
11 | `usa7` | `ascii` | `ascii_general_ci`
12 | `ujis` | `ujis` | `ujis_japanese_ci`
13 | `sjis` | `sjis` | `sjis_japanese_ci`
14 | `cp1251` | `cp1251` | `cp1251_bulgarian_ci`
15 | `danish` | `latin1` | `latin1_danish_ci`
16 | `hebrew` | `hebrew` | `hebrew_general_ci`
17 | `win1251` | `(removed)` | `(removed)`
18 | `tis620` | `tis620` | `tis620_thai_ci`
19 | `euc_kr` | `euckr` | `euckr_korean_ci`
20 | `estonia` | `latin7` | `latin7_estonian_ci`
21 | `hungarian` | `latin2` | `latin2_hungarian_ci`
22 | `koi8_ukr` | `koi8u` | `koi8u_ukrainian_ci`
23 | `win1251ukr` | `cp1251` | `cp1251_ukrainian_ci`
24 | `gb2312` | `gb2312` | `gb2312_chinese_ci`
25 | `greek` | `greek` | `greek_general_ci`
26 | `win1250` | `cp1250` | `cp1250_general_ci`
27 | `croat` | `latin2` | `latin2_croatian_ci`
28 | `gbk` | `gbk` | `gbk_chinese_ci`
29 | `cp1257` | `cp1257` | `cp1257_lithuanian_ci`
30 | `latin5` | `latin5` | `latin5_turkish_ci`
31 | `latin1_de` | `latin1` | `latin1_german2_ci`

### 10.9.2. Converting 4.0 Character Columns to 4.1 Format

Normally, the server runs using the `latin1` character set by default. If you have been storing column data that actually is in some other character set that the 4.1 server supports directly, you can convert the column. However, you should avoid trying to convert directly from `latin1` to the "real" character set. This may result in data loss. Instead, convert the column to a binary data type, and then from the binary type to a non-binary type with the desired character set. Conversion to and from binary involves no attempt at character value conversion and preserves your data intact. For example, suppose that you have a 4.0 table with three columns that are used to store values represented in `latin1`, `latin2`, and `utf8`:

    CREATE TABLE t
    (
        latin1_col CHAR(50),
        latin2_col CHAR(100),
        utf8_col CHAR(150)
    );

For MySQL 4.1, you want to convert this table to leave `latin1_col` alone but change the `latin2_col` and `utf8_col` columns to have character sets of `latin2` and `utf8`. Before upgrading to 4.1, back up your table, then convert the columns as follows:

    ALTER TABLE t MODIFY latin2_col BINARY(100);
    ALTER TABLE t MODIFY utf8_col BINARY(150);

Then, after upgrading to 4.1, complete the conversion by issuing these statements:

    ALTER TABLE t MODIFY latin2_col CHAR(100) CHARACTER SET latin2;
    ALTER TABLE t MODIFY utf8_col CHAR(150) CHARACTER SET utf8;

The first two statements “remove” the character set information from the `latin2_col` and `utf8_col` columns. The second two statements assign the proper character sets to the two columns.

If you like, you can combine the to-binary conversions and from-binary conversions into single statements. In MySQL 4.0, do this:

    ALTER TABLE t
        MODIFY latin2_col BINARY(100),
        MODIFY utf8_col BINARY(150);

After upgrading to 4.1, do this:

    ALTER TABLE t
        MODIFY latin2_col CHAR(100) CHARACTER SET latin2,
        MODIFY utf8_col CHAR(150) CHARACTER SET utf8;

If you can ensure that the tables will not otherwise be modified before you perform the character set conversion, you can issue all of the `ALTER TABLE` statements after upgrading to MySQL 4.1.

If you specified attributes when creating a column initially, you should also specify them when altering the table with `ALTER TABLE`. For example, if you specified `NOT NULL` and an explicit `DEFAULT` value, you should also provide them in the `ALTER TABLE` statement. Otherwise, the resulting column definition will not include those attributes.

## 10.10. Character Sets and Collations That MySQL Supports

MySQL supports 70+ collations for 30+ character sets. This section indicates which character sets MySQL supports. There is one subsection for each group of related character sets. For each character set, the allowable collations are listed.

You can always list the available character sets and their default collations with the `SHOW CHARACTER SET` statement:

    mysql> SHOW CHARACTER SET;
    +----------+-----------------------------+---------------------+
    | Charset  | Description                 | Default collation   |
    +----------+-----------------------------+---------------------+
    | big5     | Big5 Traditional Chinese    | big5_chinese_ci     |
    | dec8     | DEC West European           | dec8_swedish_ci     |
    | cp850    | DOS West European           | cp850_general_ci    |
    | hp8      | HP West European            | hp8_english_ci      |
    | koi8r    | KOI8-R Relcom Russian       | koi8r_general_ci    |
    | latin1   | cp1252 West European        | latin1_swedish_ci   |
    | latin2   | ISO 8859-2 Central European | latin2_general_ci   |
    | swe7     | 7bit Swedish                | swe7_swedish_ci     |
    | ascii    | US ASCII                    | ascii_general_ci    |
    | ujis     | EUC-JP Japanese             | ujis_japanese_ci    |
    | sjis     | Shift-JIS Japanese          | sjis_japanese_ci    |
    | hebrew   | ISO 8859-8 Hebrew           | hebrew_general_ci   |
    | tis620   | TIS620 Thai                 | tis620_thai_ci      |
    | euckr    | EUC-KR Korean               | euckr_korean_ci     |
    | koi8u    | KOI8-U Ukrainian            | koi8u_general_ci    |
    | gb2312   | GB2312 Simplified Chinese   | gb2312_chinese_ci   |
    | greek    | ISO 8859-7 Greek            | greek_general_ci    |
    | cp1250   | Windows Central European    | cp1250_general_ci   |
    | gbk      | GBK Simplified Chinese      | gbk_chinese_ci      |
    | latin5   | ISO 8859-9 Turkish          | latin5_turkish_ci   |
    | armscii8 | ARMSCII-8 Armenian          | armscii8_general_ci |
    | utf8     | UTF-8 Unicode               | utf8_general_ci     |
    | ucs2     | UCS-2 Unicode               | ucs2_general_ci     |
    | cp866    | DOS Russian                 | cp866_general_ci    |
    | keybcs2  | DOS Kamenicky Czech-Slovak  | keybcs2_general_ci  |
    | macce    | Mac Central European        | macce_general_ci    |
    | macroman | Mac West European           | macroman_general_ci |
    | cp852    | DOS Central European        | cp852_general_ci    |
    | latin7   | ISO 8859-13 Baltic          | latin7_general_ci   |
    | cp1251   | Windows Cyrillic            | cp1251_general_ci   |
    | cp1256   | Windows Arabic              | cp1256_general_ci   |
    | cp1257   | Windows Baltic              | cp1257_general_ci   |
    | binary   | Binary pseudo charset       | binary              |
    | geostd8  | GEOSTD8 Georgian            | geostd8_general_ci  |
    | cp932    | SJIS for Windows Japanese   | cp932_japanese_ci   |
    | eucjpms  | UJIS for Windows Japanese   | eucjpms_japanese_ci |
    +----------+-----------------------------+---------------------+

### 10.10.1. Unicode Character Sets

MySQL has two Unicode character sets. You can store text in about 650 languages using these character sets.

- `ucs2` (UCS-2 Unicode) collations:

  - `ucs2_bin`

  - `ucs2_czech_ci`

  - `ucs2_danish_ci`

  - `ucs2_estonian_ci`

  - `ucs2_general_ci` (default)

  - `ucs2_icelandic_ci`

  - `ucs2_latvian_ci`

  - `ucs2_lithuanian_ci`

  - `ucs2_persian_ci`

  - `ucs2_polish_ci`

  - `ucs2_roman_ci`

  - `ucs2_romanian_ci`

  - `ucs2_slovak_ci`

  - `ucs2_slovenian_ci`

  - `ucs2_spanish2_ci`

  - `ucs2_spanish_ci`

  - `ucs2_swedish_ci`

  - `ucs2_turkish_ci`

  - `ucs2_unicode_ci`

- `utf8` (UTF-8 Unicode) collations:

  - `utf8_bin`

  - `utf8_czech_ci`

  - `utf8_danish_ci`

  - `utf8_estonian_ci`

  - `utf8_general_ci` (default)

  - `utf8_icelandic_ci`

  - `utf8_latvian_ci`

  - `utf8_lithuanian_ci`

  - `utf8_persian_ci`

  - `utf8_polish_ci`

  - `utf8_roman_ci`

  - `utf8_romanian_ci`

  - `utf8_slovak_ci`

  - `utf8_slovenian_ci`

  - `utf8_spanish2_ci`

  - `utf8_spanish_ci`

  - `utf8_swedish_ci`

  - `utf8_turkish_ci`

  - `utf8_unicode_ci`

Note that in the `ucs2_roman_ci` and `utf8_roman_ci` collations, `I` and `J` are given the same precedence, and `U` and `V` are given the same precedence.

There is a limitation in MySQL 4.1 that results in two characters not being correctly handled when a user tries to change their case using `LOWER()` or `UPPER()`:

- LATIN SMALL LETTER DOTLESS i

- LATIN CAPITAL LETTER I WITH DOT ABOVE

Here are two workarounds for MySQL 4.1:

1. Use UCS2 if you have Turkish data.

2. Use these function calls:

       CONVERT(LOWER(CONVERT(col USING ucs2)) USING utf8)

MySQL implements the `utf8_unicode_ci` collation according to the Unicode Collation Algorithm (UCA) described at http://www.unicode.org/reports/tr10/. The collation uses the version-4.0.0 UCA weight keys: http://www.unicode.org/Public/UCA/4.0.0/allkeys-4.0.0.txt. The following discussion uses `utf8_unicode_ci`, but it is also true for `ucs2_unicode_ci`.

Currently, the `utf8_unicode_ci` collation has only partial support for the Unicode Collation Algorithm. Some characters are not supported yet. Also, combining marks are not fully supported. This affects primarily Vietnamese and some minority languages in Russia such as Udmurt, Tatar, Bashkir, and Mari.

The most significant feature in `utf8_unicode_ci` is that it supports expansions; that is, when one character compares as equal to combinations of other characters. For example, in German and some other languages ‘`ß`’ is equal to ‘`ss`’.

`utf8_general_ci` is a legacy collation that does not support expansions. It can make only one-to-one comparisons between characters. This means that comparisons for the `utf8_general_ci` collation are faster, but slightly less correct, than comparisons for `utf8_unicode_ci`.

For example, the following equalities hold in both `utf8_general_ci` and `utf8_unicode_ci`:

    Ä = A
    Ö = O
    Ü = U

A difference between the collations is that this is true for `utf8_general_ci`:

    ß = s

Whereas this is true for `utf8_unicode_ci`:

    ß = ss

MySQL implements language-specific collations for the `utf8` character set only if the ordering with `utf8_unicode_ci` does not work well for a language. For example, `utf8_unicode_ci` works fine for German and French, so there is no need to create special `utf8` collations for these two languages.

`utf8_general_ci` also is satisfactory for both German and French, except that ‘`ß`’ is equal to ‘`s`’, and not to ‘`ss`’. If this is acceptable for your application, then you should use `utf8_general_ci` because it is faster. Otherwise, use `utf8_unicode_ci` because it is more accurate.

`utf8_swedish_ci`, like other `utf8` language-specific collations, is derived from `utf8_unicode_ci` with additional language rules. For example, in Swedish, the following relationship holds, which is not something expected by a German or French speaker:

    Ü = Y < Ö

The `utf8_spanish_ci` and `utf8_spanish2_ci` collations correspond to modern Spanish and traditional Spanish, respectively. In both collations, ‘`ñ`’ (n-tilde) is a separate letter between ‘`n`’ and ‘`o`’. In addition, for traditional Spanish, ‘`ch`’ is a separate letter between ‘`c`’ and ‘`d`’, and ‘`ll`’ is a separate letter between ‘`l`’ and ‘`m`’

### 10.10.2. West European Character Sets

Western European character sets cover most West European languages, such as French, Spanish, Catalan, Basque, Portuguese, Italian, Albanian, Dutch, German, Danish, Swedish, Norwegian, Finnish, Faroese, Icelandic, Irish, Scottish, and English.

- `ascii` (US ASCII) collations:

  - `ascii_bin`

  - `ascii_general_ci` (default)

- `cp850` (DOS West European) collations:

  - `cp850_bin`

  - `cp850_general_ci` (default)

- `dec8` (DEC Western European) collations:

  - `dec8_bin`

  - `dec8_swedish_ci` (default)

- `hp8` (HP Western European) collations:

  - `hp8_bin`

  - `hp8_english_ci` (default)

- `latin1` (cp1252 West European) collations:

  - `latin1_bin`

  - `latin1_danish_ci`

  - `latin1_general_ci`

  - `latin1_general_cs`

  - `latin1_german1_ci`

  - `latin1_german2_ci`

  - `latin1_spanish_ci`

  - `latin1_swedish_ci` (default)

  `latin1` is the default character set. MySQL's `latin1` is the same as the Windows `cp1252` character set. This means it is the same as the official `ISO 8859-1` or IANA (Internet Assigned Numbers Authority) `latin1`, but IANA `latin1` treats the code points between `0x80` and `0x9f` as “undefined,” whereas `cp1252`, and therefore MySQL's `latin1`, assign characters for those positions. For example, `0x80` is the Euro sign. For the “undefined” entries in `cp1252`, MySQL translates `0x81` to Unicode `0x0081`, `0x8d` to `0x008d`, `0x8f` to `0x008f`, `0x90` to `0x0090`, and `0x9d` to `0x009d`.

  The `latin1_swedish_ci` collation is the default that probably is used by the majority of MySQL customers. Although it is frequently said that it is based on the Swedish/Finnish collation rules, there are Swedes and Finns who disagree with this statement.

  The `latin1_german1_ci` and `latin1_german2_ci` collations are based on the DIN-1 and DIN-2 standards, where DIN stands for Deutsches Institut für Normung (the German equivalent of ANSI). DIN-1 is called the “dictionary collation” and DIN-2 is called the “phone book collation.”

  - `latin1_german1_ci` (dictionary) rules:

        Ä = A
        Ö = O
        Ü = U
        ß = s

  - `latin1_german2_ci` (phone-book) rules:

        Ä = AE
        Ö = OE
        Ü = UE
        ß = ss

  In the `latin1_spanish_ci` collation, ‘`ñ`’ (n-tilde) is a separate letter between ‘`n`’ and ‘`o`’.

- `macroman` (Mac West European) collations:

  - `macroman_bin`

  - `macroman_general_ci` (default)

- `swe7` (7bit Swedish) collations:

  - `swe7_bin`

  - `swe7_swedish_ci` (default)

### 10.10.3. Central European Character Sets

MySQL provides some support for character sets used in the Czech Republic, Slovakia, Hungary, Romania, Slovenia, Croatia, Poland, and Serbia (Latin).

- `cp1250` (Windows Central European) collations:

  - `cp1250_bin`

  - `cp1250_croatian_ci`

  - `cp1250_czech_cs`

  - `cp1250_general_ci` (default)

- `cp852` (DOS Central European) collations:

  - `cp852_bin`

  - `cp852_general_ci` (default)

- `keybcs2` (DOS Kamenicky Czech-Slovak) collations:

  - `keybcs2_bin`

  - `keybcs2_general_ci` (default)

- `latin2` (ISO 8859-2 Central European) collations:

  - `latin2_bin`

  - `latin2_croatian_ci`

  - `latin2_czech_cs`

  - `latin2_general_ci` (default)

  - `latin2_hungarian_ci`

- `macce` (Mac Central European) collations:

  - `macce_bin`

  - `macce_general_ci` (default)

### 10.10.4. South European and Middle East Character Sets

South European and Middle Eastern character sets supported by MySQL include Armenian, Arabic, Georgian, Greek, Hebrew, and Turkish.

- `armscii8` (ARMSCII-8 Armenian) collations:

  - `armscii8_bin`

  - `armscii8_general_ci` (default)

- `cp1256` (Windows Arabic) collations:

  - `cp1256_bin`

  - `cp1256_general_ci` (default)

- `geostd8` (GEOSTD8 Georgian) collations:

  - `geostd8_bin`

  - `geostd8_general_ci` (default)

- `greek` (ISO 8859-7 Greek) collations:

  - `greek_bin`

  - `greek_general_ci` (default)

- `hebrew` (ISO 8859-8 Hebrew) collations:

  - `hebrew_bin`

  - `hebrew_general_ci` (default)

- `latin5` (ISO 8859-9 Turkish) collations:

  - `latin5_bin`

  - `latin5_turkish_ci` (default)

### 10.10.5. Baltic Character Sets

The Baltic character sets cover Estonian, Latvian, and Lithuanian languages.

- `cp1257` (Windows Baltic) collations:

  - `cp1257_bin`

  - `cp1257_general_ci` (default)

  - `cp1257_lithuanian_ci`

- `latin7` (ISO 8859-13 Baltic) collations:

  - `latin7_bin`

  - `latin7_estonian_cs`

  - `latin7_general_ci` (default)

  - `latin7_general_cs`

### 10.10.6. Cyrillic Character Sets

The Cyrillic character sets and collations are for use with Belarusian, Bulgarian, Russian, Ukrainian, and Serbian (Cyrillic) languages.

- `cp1251` (Windows Cyrillic) collations:

  - `cp1251_bin`

  - `cp1251_bulgarian_ci`

  - `cp1251_general_ci` (default)

  - `cp1251_general_cs`

  - `cp1251_ukrainian_ci`

- `cp866` (DOS Russian) collations:

  - `cp866_bin`

  - `cp866_general_ci` (default)

- `koi8r` (KOI8-R Relcom Russian) collations:

  - `koi8r_bin`

  - `koi8r_general_ci` (default)

- `koi8u` (KOI8-U Ukrainian) collations:

  - `koi8u_bin`

  - `koi8u_general_ci` (default)

### 10.10.7. Asian Character Sets

The Asian character sets that we support include Chinese, Japanese, Korean, and Thai. These can be complicated. For example, the Chinese sets must allow for thousands of different characters. See Section 10.10.7.1, “The `cp932` Character Set”, for additional information about the `cp932` and `sjis` character sets.

- `big5` (Big5 Traditional Chinese) collations:

  - `big5_bin`

  - `big5_chinese_ci` (default)

- `cp932` (SJIS for Windows Japanese) collations:

  - `cp932_bin`

  - `cp932_japanese_ci` (default)

- `eucjpms` (UJIS for Windows Japanese) collations:

  - `eucjpms_bin`

  - `eucjpms_japanese_ci` (default)

- `euckr` (EUC-KR Korean) collations:

  - `euckr_bin`

  - `euckr_korean_ci` (default)

- `gb2312` (GB2312 Simplified Chinese) collations:

  - `gb2312_bin`

  - `gb2312_chinese_ci` (default)

- `gbk` (GBK Simplified Chinese) collations:

  - `gbk_bin`

  - `gbk_chinese_ci` (default)

- `sjis` (Shift-JIS Japanese) collations:

  - `sjis_bin`

  - `sjis_japanese_ci` (default)

- `tis620` (TIS620 Thai) collations:

  - `tis620_bin`

  - `tis620_thai_ci` (default)

- `ujis` (EUC-JP Japanese) collations:

  - `ujis_bin`

  - `ujis_japanese_ci` (default)

#### 10.10.7.1. The `cp932` Character Set

Why is `cp932` needed?

In MySQL, the `sjis` character set corresponds to the `Shift_JIS` character set defined by IANA, which supports JIS X0201 and JIS X0208 characters. (See http://www.iana.org/assignments/character-sets.)

However, the meaning of “SHIFT JIS” as a descriptive term has become very vague and it often includes the extensions to `Shift_JIS` that are defined by various vendors.

For example, “SHIFT JIS” used in Japanese Windows environments is a Microsoft extension of `Shift_JIS` and its exact name is `Microsoft Windows Codepage : 932` or `cp932`. In addition to the characters supported by `Shift_JIS`, `cp932` supports extension characters such as NEC special characters, NEC selected — IBM extended characters, and IBM extended characters.

Since MySQL 4.1, many Japanese users have experienced problems using these extension characters. These problems stem from the following factors:

- MySQL automatically converts character sets.

- Character sets are converted via Unicode (`ucs2`).

- The `sjis` character set does not support the conversion of these extension characters.

- There are several conversion rules from so-called “SHIFT JIS” to Unicode, and some characters are converted to Unicode differently depending on the conversion rule. MySQL supports only one of these rules (described later).

The MySQL `cp932` character set is designed to solve these problems. It is available as of MySQL 4.1.12.

Before MySQL 4.1, it was safe to use any version of “SHIFT JIS” in conjunction with the `sjis` character set. However, because MySQL supports character set conversion beginning with 4.1, it is important to separate IANA `Shift_JIS` and `cp932` into two different character sets because they provide different conversion rules.

How does `cp932` differ from `sjis`?

The `cp932` character set differs from `sjis` in the following ways:

- `cp932` supports NEC special characters, NEC selected — IBM extended characters, and IBM selected characters.

- Some `cp932` characters have two different code points, both of which convert to the same Unicode code point. When converting from Unicode back to `cp932`, one of the code points must be selected. For this “round trip conversion,” the rule recommended by Microsoft is used. (See http://support.microsoft.com/kb/170559/EN-US/.)

  The conversion rule works like this:

  - If the character is in both JIS X 0208 and NEC special characters, use the code point of JIS X 0208.

  - If the character is in both NEC special characters and IBM selected characters, use the code point of NEC special characters.

  - If the character is in both IBM selected characters and NEC selected — IBM extended characters, use the code point of IBM extended characters.

  The table shown at http://www.microsoft.com/globaldev/reference/dbcs/932.htm provides information about the Unicode values of `cp932` characters. For `cp932` table entries with characters under which a four-digit number appears, the number represents the corresponding Unicode (`ucs2`) encoding. For table entries with an underlined two-digit value appears, there is a range of `cp932` character values that begin with those two digits. Clicking such a table entry takes you to a page that displays the Unicode value for each of the `cp932` characters that begin with those digits.

  The following links are of special interest. They correspond to the encodings for the following sets of characters:

  - NEC special characters:

        http://www.microsoft.com/globaldev/reference/dbcs/932/932_87.htm

  - NEC selected — IBM extended characters:

        http://www.microsoft.com/globaldev/reference/dbcs/932/932_ED.htm
        http://www.microsoft.com/globaldev/reference/dbcs/932/932_EE.htm

  - IBM selected characters:

        http://www.microsoft.com/globaldev/reference/dbcs/932/932_FA.htm
        http://www.microsoft.com/globaldev/reference/dbcs/932/932_FB.htm
        http://www.microsoft.com/globaldev/reference/dbcs/932/932_FC.htm

For some characters, conversion to and from `ucs2` is different for `sjis` and `cp932`. The following tables illustrate these differences.

Conversion to `ucs2`:

`sjis`/`cp932` Value | `sjis` -> `ucs2` Conversion | `cp932` -> `ucs2` Conversion
5C | 005C | 005C
7E | 007E | 007E
815C | 2015 | 2015
815F | 005C | FF3C
8160 | 301C | FF5E
8161 | 2016 | 2225
817C | 2212 | FF0D
8191 | 00A2 | FFE0
8192 | 00A3 | FFE1
81CA | 00AC | FFE2

Conversion from `ucs2`:

`ucs2` value | `ucs2` -> `sjis` Conversion | `ucs2` -> `cp932` Conversion
005C | 815F | 5C
007E | 7E | 7E
00A2 | 8191 | 3F
00A3 | 8192 | 3F
00AC | 81CA | 3F
2015 | 815C | 815C
2016 | 8161 | 3F
2212 | 817C | 3F
2225 | 3F | 8161
301C | 8160 | 3F
FF0D | 3F | 817C
FF3C | 3F | 815F
FF5E | 3F | 8160
FFE0 | 3F | 8191
FFE1 | 3F | 8192
FFE2 | 3F | 81CA

Users of any Japanese character sets should be aware that using `--character-set-client-handshake` (or `--skip-character-set-client-handshake`) has an important effect. See Section 5.2.1, “Command Options”.

Prev |  | Next
Chapter 9. Language Structure | Home | Chapter 11. Data Types