Unicode Support

This file contains various utility functions for manipulating UTF8-encoded strings. For the uninitiated, this is a method of representing arbitrary Unicode characters as a variable number of bytes, in such a way that characters in the ASCII range are unmodified, and a zero byte never appears within non-zero characters.

Returns true if the first len bytes of the string s are the same whether or not the string is encoded in UTF-8 (or UTF-EBCDIC on EBCDIC machines). That is, if they are invariant. On ASCII-ish machines, only ASCII characters fit this definition, hence the function's name.

If len is 0, it will be calculated using strlen(s).

See also "is_utf8_string"(), "is_utf8_string_loclen"(), and "is_utf8_string_loc"().

Adds the UTF-8 representation of the code point uv to the end of the string d; d should have at least UTF8_MAXBYTES+1 free bytes available. The return value is the pointer to the byte after the end of the new character. In other words,

    d = uvuni_to_utf8_flags(d, uv, flags);

or, in most cases,

    d = uvuni_to_utf8(d, uv);

(which is equivalent to)

    d = uvuni_to_utf8_flags(d, uv, 0);

This is the recommended Unicode-aware way of saying

    *(d++) = uv;

This function will convert to UTF-8 (and not warn) even code points that aren't legal Unicode or are problematic, unless flags contains one or more of the following flags:

If uv is a Unicode surrogate code point and UNICODE_WARN_SURROGATE is set, the function will raise a warning, provided UTF8 warnings are enabled. If instead UNICODE_DISALLOW_SURROGATE is set, the function will fail and return NULL. If both flags are set, the function will both warn and return NULL.

The UNICODE_WARN_NONCHAR and UNICODE_DISALLOW_NONCHAR flags correspondingly affect how the function handles a Unicode non-character. And, likewise for the UNICODE_WARN_SUPER and UNICODE_DISALLOW_SUPER flags, and code points that are above the Unicode maximum of 0x10FFFF. Code points above 0x7FFF_FFFF (which are even less portable) can be warned and/or disallowed even if other above-Unicode code points are accepted by the UNICODE_WARN_FE_FF and UNICODE_DISALLOW_FE_FF flags.

And finally, the flag UNICODE_WARN_ILLEGAL_INTERCHANGE selects all four of the above WARN flags; and UNICODE_DISALLOW_ILLEGAL_INTERCHANGE selects all four DISALLOW flags.

Returns the number of bytes that comprise the first UTF-8 encoded character in buffer buf. buf_end should point to one position beyond the end of the buffer. 0 is returned if buf does not point to a complete, valid UTF-8 encoded character.

Note that an INVARIANT character (i.e. ASCII on non-EBCDIC machines) is a valid UTF-8 character.

DEPRECATED!

Tests if some arbitrary number of bytes begins in a valid UTF-8 character. Note that an INVARIANT (i.e. ASCII on non-EBCDIC machines) character is a valid UTF-8 character. The actual number of bytes in the UTF-8 character will be returned if it is valid, otherwise 0.

This function is deprecated due to the possibility that malformed input could cause reading beyond the end of the input buffer. Use "is_utf8_char_buf" instead.

Returns true if the first len bytes of string s form a valid UTF-8 string, false otherwise. If len is 0, it will be calculated using strlen(s) (which means if you use this option, that s has to have a terminating NUL byte). Note that all characters being ASCII constitute 'a valid UTF-8 string'.

See also "is_ascii_string"(), "is_utf8_string_loclen"(), and "is_utf8_string_loc"().

Like "is_utf8_string" but stores the location of the failure (in the case of "utf8ness failure") or the location s+len (in the case of "utf8ness success") in the ep.

See also "is_utf8_string_loclen"() and "is_utf8_string"().

Like "is_utf8_string"() but stores the location of the failure (in the case of "utf8ness failure") or the location s+len (in the case of "utf8ness success") in the ep, and the number of UTF-8 encoded characters in the el.

See also "is_utf8_string_loc"() and "is_utf8_string"().

Bottom level UTF-8 decode routine. Returns the code point value of the first character in the string s, which is assumed to be in UTF-8 (or UTF-EBCDIC) encoding, and no longer than curlen bytes; *retlen (if retlen isn't NULL) will be set to the length, in bytes, of that character.

The value of flags determines the behavior when s does not point to a well-formed UTF-8 character. If flags is 0, when a malformation is found, zero is returned and *retlen is set so that (s + *retlen) is the next possible position in s that could begin a non-malformed character. Also, if UTF-8 warnings haven't been lexically disabled, a warning is raised.

Various ALLOW flags can be set in flags to allow (and not warn on) individual types of malformations, such as the sequence being overlong (that is, when there is a shorter sequence that can express the same code point; overlong sequences are expressly forbidden in the UTF-8 standard due to potential security issues). Another malformation example is the first byte of a character not being a legal first byte. See utf8.h for the list of such flags. For allowed 0 length strings, this function returns 0; for allowed overlong sequences, the computed code point is returned; for all other allowed malformations, the Unicode REPLACEMENT CHARACTER is returned, as these have no determinable reasonable value.

The UTF8_CHECK_ONLY flag overrides the behavior when a non-allowed (by other flags) malformation is found. If this flag is set, the routine assumes that the caller will raise a warning, and this function will silently just set retlen to -1 and return zero.

Certain code points are considered problematic. These are Unicode surrogates, Unicode non-characters, and code points above the Unicode maximum of 0x10FFFF. By default these are considered regular code points, but certain situations warrant special handling for them. If flags contains UTF8_DISALLOW_ILLEGAL_INTERCHANGE, all three classes are treated as malformations and handled as such. The flags UTF8_DISALLOW_SURROGATE, UTF8_DISALLOW_NONCHAR, and UTF8_DISALLOW_SUPER (meaning above the legal Unicode maximum) can be set to disallow these categories individually.

The flags UTF8_WARN_ILLEGAL_INTERCHANGE, UTF8_WARN_SURROGATE, UTF8_WARN_NONCHAR, and UTF8_WARN_SUPER will cause warning messages to be raised for their respective categories, but otherwise the code points are considered valid (not malformations). To get a category to both be treated as a malformation and raise a warning, specify both the WARN and DISALLOW flags. (But note that warnings are not raised if lexically disabled nor if UTF8_CHECK_ONLY is also specified.)

Very large code points (above 0x7FFF_FFFF) are considered more problematic than the others that are above the Unicode legal maximum. There are several reasons: they requre at least 32 bits to represent them on ASCII platforms, are not representable at all on EBCDIC platforms, and the original UTF-8 specification never went above this number (the current 0x10FFFF limit was imposed later). (The smaller ones, those that fit into 32 bits, are representable by a UV on ASCII platforms, but not by an IV, which means that the number of operations that can be performed on them is quite restricted.) The UTF-8 encoding on ASCII platforms for these large code points begins with a byte containing 0xFE or 0xFF. The UTF8_DISALLOW_FE_FF flag will cause them to be treated as malformations, while allowing smaller above-Unicode code points. (Of course UTF8_DISALLOW_SUPER will treat all above-Unicode code points, including these, as malformations.) Similarly, UTF8_WARN_FE_FF acts just like the other WARN flags, but applies just to these code points.

All other code points corresponding to Unicode characters, including private use and those yet to be assigned, are never considered malformed and never warn.

Most code should use "utf8_to_uvchr_buf"() rather than call this directly.

Returns the native code point of the first character in the string s which is assumed to be in UTF-8 encoding; send points to 1 beyond the end of s. *retlen will be set to the length, in bytes, of that character.

If s does not point to a well-formed UTF-8 character and UTF8 warnings are enabled, zero is returned and *retlen is set (if retlen isn't NULL) to -1. If those warnings are off, the computed value if well-defined (or the Unicode REPLACEMENT CHARACTER, if not) is silently returned, and *retlen is set (if retlen isn't NULL) so that (s + *retlen) is the next possible position in s that could begin a non-malformed character. See "utf8n_to_uvuni" for details on when the REPLACEMENT CHARACTER is returned.

DEPRECATED!

Returns the native code point of the first character in the string s which is assumed to be in UTF-8 encoding; retlen will be set to the length, in bytes, of that character.

Some, but not all, UTF-8 malformations are detected, and in fact, some malformed input could cause reading beyond the end of the input buffer, which is why this function is deprecated. Use "utf8_to_uvchr_buf" instead.

If s points to one of the detected malformations, and UTF8 warnings are enabled, zero is returned and *retlen is set (if retlen isn't NULL) to -1. If those warnings are off, the computed value if well-defined (or the Unicode REPLACEMENT CHARACTER, if not) is silently returned, and *retlen is set (if retlen isn't NULL) so that (s + *retlen) is the next possible position in s that could begin a non-malformed character. See "utf8n_to_uvuni" for details on when the REPLACEMENT CHARACTER is returned.

Returns the Unicode code point of the first character in the string s which is assumed to be in UTF-8 encoding; send points to 1 beyond the end of s. retlen will be set to the length, in bytes, of that character.

This function should only be used when the returned UV is considered an index into the Unicode semantic tables (e.g. swashes).

If s does not point to a well-formed UTF-8 character and UTF8 warnings are enabled, zero is returned and *retlen is set (if retlen isn't NULL) to -1. If those warnings are off, the computed value if well-defined (or the Unicode REPLACEMENT CHARACTER, if not) is silently returned, and *retlen is set (if retlen isn't NULL) so that (s + *retlen) is the next possible position in s that could begin a non-malformed character. See "utf8n_to_uvuni" for details on when the REPLACEMENT CHARACTER is returned.

DEPRECATED!

Returns the Unicode code point of the first character in the string s which is assumed to be in UTF-8 encoding; retlen will be set to the length, in bytes, of that character.

This function should only be used when the returned UV is considered an index into the Unicode semantic tables (e.g. swashes).

Some, but not all, UTF-8 malformations are detected, and in fact, some malformed input could cause reading beyond the end of the input buffer, which is why this function is deprecated. Use "utf8_to_uvuni_buf" instead.

If s points to one of the detected malformations, and UTF8 warnings are enabled, zero is returned and *retlen is set (if retlen doesn't point to NULL) to -1. If those warnings are off, the computed value if well-defined (or the Unicode REPLACEMENT CHARACTER, if not) is silently returned, and *retlen is set (if retlen isn't NULL) so that (s + *retlen) is the next possible position in s that could begin a non-malformed character. See "utf8n_to_uvuni" for details on when the REPLACEMENT CHARACTER is returned.

Return the length of the UTF-8 char encoded string s in characters. Stops at e (inclusive). If e < s or if the scan would end up past e, croaks.

Returns the number of UTF-8 characters between the UTF-8 pointers a and b.

WARNING: use only if you *know* that the pointers point inside the same UTF-8 buffer.

Return the UTF-8 pointer s displaced by off characters, either forward or backward.

WARNING: do not use the following unless you *know* off is within the UTF-8 data pointed to by s *and* that on entry s is aligned on the first byte of character or just after the last byte of a character.

Compares the sequence of characters (stored as octets) in b, blen with the sequence of characters (stored as UTF-8) in u, ulen. Returns 0 if they are equal, -1 or -2 if the first string is less than the second string, +1 or +2 if the first string is greater than the second string.

-1 or +1 is returned if the shorter string was identical to the start of the longer string. -2 or +2 is returned if the was a difference between characters within the strings.

Converts a string s of length len from UTF-8 into native byte encoding. Unlike "bytes_to_utf8", this over-writes the original string, and updates len to contain the new length. Returns zero on failure, setting len to -1.

If you need a copy of the string, see "bytes_from_utf8".

Converts a string s of length len from UTF-8 into native byte encoding. Unlike "utf8_to_bytes" but like "bytes_to_utf8", returns a pointer to the newly-created string, and updates len to contain the new length. Returns the original string if no conversion occurs, len is unchanged. Do nothing if is_utf8 points to 0. Sets is_utf8 to 0 if s is converted or consisted entirely of characters that are invariant in utf8 (i.e., US-ASCII on non-EBCDIC machines).

Converts a string s of length len bytes from the native encoding into UTF-8. Returns a pointer to the newly-created string, and sets len to reflect the new length in bytes.

A NUL character will be written after the end of the string.

If you want to convert to UTF-8 from encodings other than the native (Latin1 or EBCDIC), see "sv_recode_to_utf8"().

The p contains the pointer to the UTF-8 string encoding the character that is being converted. This routine assumes that the character at p is well-formed.

The ustrp is a pointer to the character buffer to put the conversion result to. The lenp is a pointer to the length of the result.

The swashp is a pointer to the swash to use.

Both the special and normal mappings are stored in lib/unicore/To/Foo.pl, and loaded by SWASHNEW, using lib/utf8_heavy.pl. The special (usually, but not always, a multicharacter mapping), is tried first.

The special is a string like "utf8::ToSpecLower", which means the hash %utf8::ToSpecLower. The access to the hash is through Perl_to_utf8_case().

The normal is a string like "ToLower" which means the swash %utf8::ToLower.

Convert the UTF-8 encoded character at p to its uppercase version and store that in UTF-8 in ustrp and its length in bytes in lenp. Note that the ustrp needs to be at least UTF8_MAXBYTES_CASE+1 bytes since the uppercase version may be longer than the original character.

The first character of the uppercased version is returned (but note, as explained above, that there may be more.)

The character at p is assumed by this routine to be well-formed.

Convert the UTF-8 encoded character at p to its titlecase version and store that in UTF-8 in ustrp and its length in bytes in lenp. Note that the ustrp needs to be at least UTF8_MAXBYTES_CASE+1 bytes since the titlecase version may be longer than the original character.

The first character of the titlecased version is returned (but note, as explained above, that there may be more.)

The character at p is assumed by this routine to be well-formed.

Convert the UTF-8 encoded character at p to its lowercase version and store that in UTF-8 in ustrp and its length in bytes in lenp. Note that the ustrp needs to be at least UTF8_MAXBYTES_CASE+1 bytes since the lowercase version may be longer than the original character.

The first character of the lowercased version is returned (but note, as explained above, that there may be more.)

The character at p is assumed by this routine to be well-formed.

Convert the UTF-8 encoded character at p to its foldcase version and store that in UTF-8 in ustrp and its length in bytes in lenp. Note that the ustrp needs to be at least UTF8_MAXBYTES_CASE+1 bytes since the foldcase version may be longer than the original character (up to three characters).

The first character of the foldcased version is returned (but note, as explained above, that there may be more.)

The character at p is assumed by this routine to be well-formed.

Adds the UTF-8 representation of the Native code point uv to the end of the string d; d should have at least UTF8_MAXBYTES+1 free bytes available. The return value is the pointer to the byte after the end of the new character. In other words,

    d = uvchr_to_utf8(d, uv);

is the recommended wide native character-aware way of saying

    *(d++) = uv;

Returns the native character value of the first character in the string s which is assumed to be in UTF-8 encoding; retlen will be set to the length, in bytes, of that character.

length and flags are the same as "utf8n_to_uvuni"().

Build to the scalar dsv a displayable version of the string spv, length len, the displayable version being at most pvlim bytes long (if longer, the rest is truncated and "..." will be appended).

The flags argument can have UNI_DISPLAY_ISPRINT set to display isPRINT()able characters as themselves, UNI_DISPLAY_BACKSLASH to display the \\[nrfta\\] as the backslashed versions (like '\n') (UNI_DISPLAY_BACKSLASH is preferred over UNI_DISPLAY_ISPRINT for \\). UNI_DISPLAY_QQ (and its alias UNI_DISPLAY_REGEX) have both UNI_DISPLAY_BACKSLASH and UNI_DISPLAY_ISPRINT turned on.

The pointer to the PV of the dsv is returned.

Build to the scalar dsv a displayable version of the scalar sv, the displayable version being at most pvlim bytes long (if longer, the rest is truncated and "..." will be appended).

The flags argument is as in "pv_uni_display"().

The pointer to the PV of the dsv is returned.

Returns true if the leading portions of the strings s1 and s2 (either or both of which may be in UTF-8) are the same case-insensitively; false otherwise. How far into the strings to compare is determined by other input parameters.

If u1 is true, the string s1 is assumed to be in UTF-8-encoded Unicode; otherwise it is assumed to be in native 8-bit encoding. Correspondingly for u2 with respect to s2.

If the byte length l1 is non-zero, it says how far into s1 to check for fold equality. In other words, s1+l1 will be used as a goal to reach. The scan will not be considered to be a match unless the goal is reached, and scanning won't continue past that goal. Correspondingly for l2 with respect to s2.

If pe1 is non-NULL and the pointer it points to is not NULL, that pointer is considered an end pointer beyond which scanning of s1 will not continue under any circumstances. This means that if both l1 and pe1 are specified, and pe1 is less than s1+l1, the match will never be successful because it can never get as far as its goal (and in fact is asserted against). Correspondingly for pe2 with respect to s2.

At least one of s1 and s2 must have a goal (at least one of l1 and l2 must be non-zero), and if both do, both have to be reached for a successful match. Also, if the fold of a character is multiple characters, all of them must be matched (see tr21 reference below for 'folding').

Upon a successful match, if pe1 is non-NULL, it will be set to point to the beginning of the next character of s1 beyond what was matched. Correspondingly for pe2 and s2.

For case-insensitiveness, the "casefolding" of Unicode is used instead of upper/lowercasing both the characters, see http://www.unicode.org/unicode/reports/tr21/ (Case Mappings).

cpanm sort

perl -MCPAN -e shell
install sort