|
|
<html><head><title>pcre2unicode specification</title></head><body bgcolor="#FFFFFF" text="#00005A" link="#0066FF" alink="#3399FF" vlink="#2222BB"><h1>pcre2unicode man page</h1><p>Return to the <a href="index.html">PCRE2 index page</a>.</p><p>This page is part of the PCRE2 HTML documentation. It was generatedautomatically from the original man page. If there is any nonsense in it,please consult the man page, in case the conversion went wrong.<br><br><b>UNICODE AND UTF SUPPORT</b><br><P>PCRE2 is normally built with Unicode support, though if you do not need it, youcan build it without, in which case the library will be smaller. With Unicodesupport, PCRE2 has knowledge of Unicode character properties and can processstrings of text in UTF-8, UTF-16, and UTF-32 format (depending on the code unitwidth), but this is not the default. Unless specifically requested, PCRE2treats each code unit in a string as one character.</P><P>There are two ways of telling PCRE2 to switch to UTF mode, where characters mayconsist of more than one code unit and the range of values is constrained. Theprogram can call<a href="pcre2_compile.html"><b>pcre2_compile()</b></a>with the PCRE2_UTF option, or the pattern may start with the sequence (*UTF).However, the latter facility can be locked out by the PCRE2_NEVER_UTF option.That is, the programmer can prevent the supplier of the pattern from switchingto UTF mode.</P><P>Note that the PCRE2_MATCH_INVALID_UTF option (see<a href="#matchinvalid">below)</a>forces PCRE2_UTF to be set.</P><P>In UTF mode, both the pattern and any subject strings that are matched againstit are treated as UTF strings instead of strings of individual one-code-unitcharacters. There are also some other changes to the way characters arehandled, as documented below.</P><br><b>UNICODE PROPERTY SUPPORT</b><br><P>When PCRE2 is built with Unicode support, the escape sequences \p{..},\P{..}, and \X can be used. This is not dependent on the PCRE2_UTF setting.The Unicode properties that can be tested are a subset of those that Perlsupports. Currently they are limited to the general category properties such asLu for an upper case letter or Nd for a decimal number, the derived propertiesAny and LC (synonym L&), the Unicode script names such as Arabic or Han,Bidi_Class, Bidi_Control, and a few binary properties.</P><P>The full lists are given in the<a href="pcre2pattern.html"><b>pcre2pattern</b></a>and<a href="pcre2syntax.html"><b>pcre2syntax</b></a>documentation. In general, only the short names for properties are supported.For example, \p{L} matches a letter. Its longer synonym, \p{Letter}, is notsupported. Furthermore, in Perl, many properties may optionally be prefixed by"Is", for compatibility with Perl 5.6. PCRE2 does not support this.</P><br><b>WIDE CHARACTERS AND UTF MODES</b><br><P>Code points less than 256 can be specified in patterns by either braced orunbraced hexadecimal escape sequences (for example, \x{b3} or \xb3). Largervalues have to use braced sequences. Unbraced octal code points up to \777 arealso recognized; larger ones can be coded using \o{...}.</P><P>The escape sequence \N{U+<hex digits>} is recognized as another way ofspecifying a Unicode character by code point in a UTF mode. It is not allowedin non-UTF mode.</P><P>In UTF mode, repeat quantifiers apply to complete UTF characters, not toindividual code units.</P><P>In UTF mode, the dot metacharacter matches one UTF character instead of asingle code unit.</P><P>In UTF mode, capture group names are not restricted to ASCII, and may containany Unicode letters and decimal digits, as well as underscore.</P><P>The escape sequence \C can be used to match a single code unit in UTF mode,but its use can lead to some strange effects because it breaks up multi-unitcharacters (see the description of \C in the<a href="pcre2pattern.html"><b>pcre2pattern</b></a>documentation). For this reason, there is a build-time option that disablessupport for \C completely. There is also a less draconian compile-time optionfor locking out the use of \C when a pattern is compiled.</P><P>The use of \C is not supported by the alternative matching function<b>pcre2_dfa_match()</b> when in UTF-8 or UTF-16 mode, that is, when a charactermay consist of more than one code unit. The use of \C in these modes provokesa match-time error. Also, the JIT optimization does not support \C in thesemodes. If JIT optimization is requested for a UTF-8 or UTF-16 pattern thatcontains \C, it will not succeed, and so when <b>pcre2_match()</b> is called,the matching will be carried out by the interpretive function.</P><P>The character escapes \b, \B, \d, \D, \s, \S, \w, and \W correctly testcharacters of any code value, but, by default, the characters that PCRE2recognizes as digits, spaces, or word characters remain the same set as innon-UTF mode, all with code points less than 256. This remains true even whenPCRE2 is built to include Unicode support, because to do otherwise would slowdown matching in many common cases. Note that this also applies to \band \B, because they are defined in terms of \w and \W. If you wantto test for a wider sense of, say, "digit", you can use explicit Unicodeproperty tests such as \p{Nd}. Alternatively, if you set the PCRE2_UCP option,the way that the character escapes work is changed so that Unicode propertiesare used to determine which characters match, though there are some optionsthat suppress this for individual escapes. For details see the section on<a href="pcre2pattern.html#genericchartypes">generic character types</a>in the<a href="pcre2pattern.html"><b>pcre2pattern</b></a>documentation.</P><P>Like the escapes, characters that match the POSIX named character classes areall low-valued characters unless the PCRE2_UCP option is set, but there is anoption to override this.</P><P>In contrast to the character escapes and character classes, the specialhorizontal and vertical white space escapes (\h, \H, \v, and \V) do matchall the appropriate Unicode characters, whether or not PCRE2_UCP is set.</P><br><b>UNICODE CASE-EQUIVALENCE</b><br><P>If either PCRE2_UTF or PCRE2_UCP is set, upper/lower case processing makes useof Unicode properties except for characters whose code points are less than 128and that have at most two case-equivalent values. For these, a direct tablelookup is used for speed. A few Unicode characters such as Greek sigma havemore than two code points that are case-equivalent, and these are treatedspecially. Setting PCRE2_UCP without PCRE2_UTF allows Unicode-style caseprocessing for non-UTF character encodings such as UCS-2.</P><P>There are two ASCII characters (S and K) that, in addition to their ASCII lowercase equivalents, have a non-ASCII one as well (long S and Kelvin sign).Recognition of these non-ASCII characters as case-equivalent to their ASCIIcounterparts can be disabled by setting the PCRE2_EXTRA_CASELESS_RESTRICToption. When this is set, all characters in a case equivalence must either beASCII or non-ASCII; there can be no mixing.<a name="scriptruns"></a></P><br><b>SCRIPT RUNS</b><br><P>The pattern constructs (*script_run:...) and (*atomic_script_run:...), withsynonyms (*sr:...) and (*asr:...), verify that the string matched within theparentheses is a script run. In concept, a script run is a sequence ofcharacters that are all from the same Unicode script. However, because somescripts are commonly used together, and because some diacritical and othermarks are used with multiple scripts, it is not that simple.</P><P>Every Unicode character has a Script property, mostly with a valuecorresponding to the name of a script, such as Latin, Greek, or Cyrillic. Thereare also three special values:</P><P>"Unknown" is used for code points that have not been assigned, and also for thesurrogate code points. In the PCRE2 32-bit library, characters whose codepoints are greater than the Unicode maximum (U+10FFFF), which are accessibleonly in non-UTF mode, are assigned the Unknown script.</P><P>"Common" is used for characters that are used with many scripts. These includepunctuation, emoji, mathematical, musical, and currency symbols, and the ASCIIdigits 0 to 9.</P><P>"Inherited" is used for characters such as diacritical marks that modify aprevious character. These are considered to take on the script of the characterthat they modify.</P><P>Some Inherited characters are used with many scripts, but many of them are onlynormally used with a small number of scripts. For example, U+102E0 (CopticEpact thousands mark) is used only with Arabic and Coptic. In order to make itpossible to check this, a Unicode property called Script Extension exists. Itsvalue is a list of scripts that apply to the character. For the majority ofcharacters, the list contains just one script, the same one as the Scriptproperty. However, for characters such as U+102E0 more than one Script islisted. There are also some Common characters that have a single, non-Commonscript in their Script Extension list.</P><P>The next section describes the basic rules for deciding whether a given stringof characters is a script run. Note, however, that there are some special casesinvolving the Chinese Han script, and an additional constraint for decimaldigits. These are covered in subsequent sections.</P><br><b>Basic script run rules</b><br><P>A string that is less than two characters long is a script run. This is theonly case in which an Unknown character can be part of a script run. Longerstrings are checked using only the Script Extensions property, not the basicScript property.</P><P>If a character's Script Extension property is the single value "Inherited", itis always accepted as part of a script run. This is also true for the property"Common", subject to the checking of decimal digits described below. All theremaining characters in a script run must have at least one script in common intheir Script Extension lists. In set-theoretic terminology, the intersection ofall the sets of scripts must not be empty.</P><P>A simple example is an Internet name such as "google.com". The letters are allin the Latin script, and the dot is Common, so this string is a script run.However, the Cyrillic letter "o" looks exactly the same as the Latin "o"; astring that looks the same, but with Cyrillic "o"s is not a script run.</P><P>More interesting examples involve characters with more than one script in theirScript Extension. Consider the following characters:<pre> U+060C Arabic comma U+06D4 Arabic full stop</pre>The first has the Script Extension list Arabic, Hanifi Rohingya, Syriac, andThaana; the second has just Arabic and Hanifi Rohingya. Both of them couldappear in script runs of either Arabic or Hanifi Rohingya. The first could alsoappear in Syriac or Thaana script runs, but the second could not.</P><br><b>The Chinese Han script</b><br><P>The Chinese Han script is commonly used in conjunction with other scripts forwriting certain languages. Japanese uses the Hiragana and Katakana scriptstogether with Han; Korean uses Hangul and Han; Taiwanese Mandarin uses Bopomofoand Han. These three combinations are treated as special cases when checkingscript runs and are, in effect, "virtual scripts". Thus, a script run maycontain a mixture of Hiragana, Katakana, and Han, or a mixture of Hangul andHan, or a mixture of Bopomofo and Han, but not, for example, a mixture ofHangul and Bopomofo and Han. PCRE2 (like Perl) follows Unicode's TechnicalStandard 39 ("Unicode Security Mechanisms", http://unicode.org/reports/tr39/)in allowing such mixtures.</P><br><b>Decimal digits</b><br><P>Unicode contains many sets of 10 decimal digits in different scripts, and somescripts (including the Common script) contain more than one set. Some of thesedecimal digits them are visually indistinguishable from the common ASCIIdigits. In addition to the script checking described above, if a script runcontains any decimal digits, they must all come from the same set of 10adjacent characters.</P><br><b>VALIDITY OF UTF STRINGS</b><br><P>When the PCRE2_UTF option is set, the strings passed as patterns and subjectsare (by default) checked for validity on entry to the relevant functions. If aninvalid UTF string is passed, a negative error code is returned. The code unitoffset to the offending character can be extracted from the match data block bycalling <b>pcre2_get_startchar()</b>, which is used for this purpose after a UTFerror.</P><P>In some situations, you may already know that your strings are valid, andtherefore want to skip these checks in order to improve performance, forexample in the case of a long subject string that is being scanned repeatedly.If you set the PCRE2_NO_UTF_CHECK option at compile time or at match time,PCRE2 assumes that the pattern or subject it is given (respectively) containsonly valid UTF code unit sequences.</P><P>If you pass an invalid UTF string when PCRE2_NO_UTF_CHECK is set, the resultis undefined and your program may crash or loop indefinitely or give incorrectresults. There is, however, one mode of matching that can handle invalid UTFsubject strings. This is enabled by passing PCRE2_MATCH_INVALID_UTF to<b>pcre2_compile()</b> and is discussed below in the next section. The rest ofthis section covers the case when PCRE2_MATCH_INVALID_UTF is not set.</P><P>Passing PCRE2_NO_UTF_CHECK to <b>pcre2_compile()</b> just disables the UTF checkfor the pattern; it does not also apply to subject strings. If you want todisable the check for a subject string you must pass this same option to<b>pcre2_match()</b> or <b>pcre2_dfa_match()</b>.</P><P>UTF-16 and UTF-32 strings can indicate their endianness by special code knowsas a byte-order mark (BOM). The PCRE2 functions do not handle this, expectingstrings to be in host byte order.</P><P>Unless PCRE2_NO_UTF_CHECK is set, a UTF string is checked before any otherprocessing takes place. In the case of <b>pcre2_match()</b> and<b>pcre2_dfa_match()</b> calls with a non-zero starting offset, the check isapplied only to that part of the subject that could be inspected duringmatching, and there is a check that the starting offset points to the firstcode unit of a character or to the end of the subject. If there are nolookbehind assertions in the pattern, the check starts at the starting offset.Otherwise, it starts at the length of the longest lookbehind before thestarting offset, or at the start of the subject if there are not that manycharacters before the starting offset. Note that the sequences \b and \B areone-character lookbehinds.</P><P>In addition to checking the format of the string, there is a check to ensurethat all code points lie in the range U+0 to U+10FFFF, excluding the surrogatearea. The so-called "non-character" code points are not excluded becauseUnicode corrigendum #9 makes it clear that they should not be.</P><P>Characters in the "Surrogate Area" of Unicode are reserved for use by UTF-16,where they are used in pairs to encode code points with values greater than0xFFFF. The code points that are encoded by UTF-16 pairs are availableindependently in the UTF-8 and UTF-32 encodings. (In other words, the wholesurrogate thing is a fudge for UTF-16 which unfortunately messes up UTF-8 andUTF-32.)</P><P>Setting PCRE2_NO_UTF_CHECK at compile time does not disable the error that isgiven if an escape sequence for an invalid Unicode code point is encountered inthe pattern. If you want to allow escape sequences such as \x{d800} (asurrogate code point) you can set the PCRE2_EXTRA_ALLOW_SURROGATE_ESCAPES extraoption. However, this is possible only in UTF-8 and UTF-32 modes, because thesevalues are not representable in UTF-16.<a name="utf8strings"></a></P><br><b>Errors in UTF-8 strings</b><br><P>The following negative error codes are given for invalid UTF-8 strings:<pre> PCRE2_ERROR_UTF8_ERR1 PCRE2_ERROR_UTF8_ERR2 PCRE2_ERROR_UTF8_ERR3 PCRE2_ERROR_UTF8_ERR4 PCRE2_ERROR_UTF8_ERR5</pre>The string ends with a truncated UTF-8 character; the code specifies how manybytes are missing (1 to 5). Although RFC 3629 restricts UTF-8 characters to beno longer than 4 bytes, the encoding scheme (originally defined by RFC 2279)allows for up to 6 bytes, and this is checked first; hence the possibility of4 or 5 missing bytes.<pre> PCRE2_ERROR_UTF8_ERR6 PCRE2_ERROR_UTF8_ERR7 PCRE2_ERROR_UTF8_ERR8 PCRE2_ERROR_UTF8_ERR9 PCRE2_ERROR_UTF8_ERR10</pre>The two most significant bits of the 2nd, 3rd, 4th, 5th, or 6th byte of thecharacter do not have the binary value 0b10 (that is, either the mostsignificant bit is 0, or the next bit is 1).<pre> PCRE2_ERROR_UTF8_ERR11 PCRE2_ERROR_UTF8_ERR12</pre>A character that is valid by the RFC 2279 rules is either 5 or 6 bytes long;these code points are excluded by RFC 3629.<pre> PCRE2_ERROR_UTF8_ERR13</pre>A 4-byte character has a value greater than 0x10ffff; these code points areexcluded by RFC 3629.<pre> PCRE2_ERROR_UTF8_ERR14</pre>A 3-byte character has a value in the range 0xd800 to 0xdfff; this range ofcode points are reserved by RFC 3629 for use with UTF-16, and so are excludedfrom UTF-8.<pre> PCRE2_ERROR_UTF8_ERR15 PCRE2_ERROR_UTF8_ERR16 PCRE2_ERROR_UTF8_ERR17 PCRE2_ERROR_UTF8_ERR18 PCRE2_ERROR_UTF8_ERR19</pre>A 2-, 3-, 4-, 5-, or 6-byte character is "overlong", that is, it codes for avalue that can be represented by fewer bytes, which is invalid. For example,the two bytes 0xc0, 0xae give the value 0x2e, whose correct coding uses justone byte.<pre> PCRE2_ERROR_UTF8_ERR20</pre>The two most significant bits of the first byte of a character have the binaryvalue 0b10 (that is, the most significant bit is 1 and the second is 0). Such abyte can only validly occur as the second or subsequent byte of a multi-bytecharacter.<pre> PCRE2_ERROR_UTF8_ERR21</pre>The first byte of a character has the value 0xfe or 0xff. These values cannever occur in a valid UTF-8 string.<a name="utf16strings"></a></P><br><b>Errors in UTF-16 strings</b><br><P>The following negative error codes are given for invalid UTF-16 strings:<pre> PCRE2_ERROR_UTF16_ERR1 Missing low surrogate at end of string PCRE2_ERROR_UTF16_ERR2 Invalid low surrogate follows high surrogate PCRE2_ERROR_UTF16_ERR3 Isolated low surrogate
<a name="utf32strings"></a></PRE></P><br><b>Errors in UTF-32 strings</b><br><P>The following negative error codes are given for invalid UTF-32 strings:<pre> PCRE2_ERROR_UTF32_ERR1 Surrogate character (0xd800 to 0xdfff) PCRE2_ERROR_UTF32_ERR2 Code point is greater than 0x10ffff
<a name="matchinvalid"></a></PRE></P><br><b>MATCHING IN INVALID UTF STRINGS</b><br><P>You can run pattern matches on subject strings that may contain invalid UTFsequences if you call <b>pcre2_compile()</b> with the PCRE2_MATCH_INVALID_UTFoption. This is supported by <b>pcre2_match()</b>, including JIT matching, butnot by <b>pcre2_dfa_match()</b>. When PCRE2_MATCH_INVALID_UTF is set, it forcesPCRE2_UTF to be set as well. Note, however, that the pattern itself must be avalid UTF string.</P><P>If you do not set PCRE2_MATCH_INVALID_UTF when calling <b>pcre2_compile</b>, andyou are not certain that your subject strings are valid UTF sequences, youshould not make use of the JIT "fast path" function <b>pcre2_jit_match()</b>because it bypasses sanity checks, including the one for UTF validity. Aninvalid string may cause undefined behaviour, including looping, crashing, orgiving the wrong answer.</P><P>Setting PCRE2_MATCH_INVALID_UTF does not affect what <b>pcre2_compile()</b>generates, but if <b>pcre2_jit_compile()</b> is subsequently called, it doesgenerate different code. If JIT is not used, the option affects the behaviourof the interpretive code in <b>pcre2_match()</b>. When PCRE2_MATCH_INVALID_UTFis set at compile time, PCRE2_NO_UTF_CHECK is ignored at match time.</P><P>In this mode, an invalid code unit sequence in the subject never matches anypattern item. It does not match dot, it does not match \p{Any}, it does noteven match negative items such as [^X]. A lookbehind assertion fails if itencounters an invalid sequence while moving the current point backwards. Inother words, an invalid UTF code unit sequence acts as a barrier which no matchcan cross.</P><P>You can also think of this as the subject being split up into fragments ofvalid UTF, delimited internally by invalid code unit sequences. The pattern ismatched fragment by fragment. The result of a successful match, however, isgiven as code unit offsets in the entire subject string in the usual way. Thereare a few points to consider:</P><P>The internal boundaries are not interpreted as the beginnings or ends of linesand so do not match circumflex or dollar characters in the pattern.</P><P>If <b>pcre2_match()</b> is called with an offset that points to an invalidUTF-sequence, that sequence is skipped, and the match starts at the next validUTF character, or the end of the subject.</P><P>At internal fragment boundaries, \b and \B behave in the same way as at thebeginning and end of the subject. For example, a sequence such as \bWORD\bwould match an instance of WORD that is surrounded by invalid UTF code units.</P><P>Using PCRE2_MATCH_INVALID_UTF, an application can run matches on arbitrarydata, knowing that any matched strings that are returned are valid UTF. Thiscan be useful when searching for UTF text in executable or other binary files.</P><P>Note, however, that the 16-bit and 32-bit PCRE2 libraries process strings assequences of uint16_t or uint32_t code points. They cannot find valid UTFsequences within an arbitrary string of bytes unless such sequences aresuitably aligned.</P><br><b>AUTHOR</b><br><P>Philip Hazel<br>Retired from University Computing Service<br>Cambridge, England.<br></P><br><b>REVISION</b><br><P>Last updated: 12 October 2023<br>Copyright © 1997-2023 University of Cambridge.<br><p>Return to the <a href="index.html">PCRE2 index page</a>.</p>
|
