ShiftJIS::Collate - collation of ShiftJIS strings
use ShiftJIS::Collate; @sorted = ShiftJIS::Collate->new(%tailoring)->sort(@not_sorted);
This module provides some functions to compare and sort strings in the ShiftJIS encoding using the collation of Japanese character strings.
This module is an implementation of JIS X 4061:1996 and the collation rules are based on that standard. See "Conformance on the Standard".
The new method returns a collator object.
new
$Collator = ShiftJIS::Collate->new( ignoreChar => $regexIgnoredChar, kanji => $kanji_class, katakana_before_hiragana => $bool, level => $collationLevel, position_in_bytes => $bool, tounicode => \&sjis_to_unicode, preprocess => \&preprocess, upper_before_lower => $bool, ); # if %tailoring is false (empty), # $Collator should do the default collation.
If specified as a regex, any characters that match it are ignored on collation.
e.g. If you want to ignore KATAKANA PROLONGED SOUND MARK and its halfwidth form, say
ignoreChar => '^(?:\x81\x5B|\xB0)',
By default, hiragana is before katakana.
If the parameter is true, this is reversed.
Set the kanji class. See "Kanji Classes".
Level 1: 'saisho' (minimal) Level 2: 'kihon' (basic) Level 3: 'kakucho' (extended)
The kanji class is specified as 1, 2, or 3. If omitted, class 2 is applied.
This module does not provide collation of 'kakucho' kanji class since the repertory Shift_JIS does not define all the Unicode CJK unified ideographs.
But if the kanji class 3 is specified, you can collate kanji in the unicode order. In this case you must provide tounicode coderef which gives a unicode codepoint from a Shift_JIS character.
Set the maximum level. See "Collation Levels". Any higher levels than the specified one are ignored.
Level 1: alphabetic ordering Level 2: diacritic ordering Level 3: case ordering Level 4: script ordering Level 5: width ordering
The collation level is specified as a number between 1 and 5. If omitted, level 4 is applied.
If you want to collate kanji in the unicode order, specify a coderef which gives a unicode codepoint from a Shift_JIS character.
Such a subroutine should map a string comprising of a kanji of level 1 and 2 in Shift_JIS to a codepoint in the range between 0x4E00 and 0x9FFF.
By default, the index method returns its results in characters.
index
If this parameter is true, it returns the results in bytes.
If specified, the coderef is used to preprocess before the formation of sort keys.
By default, lowercase is before uppercase.
$result = $Collator->cmp($a, $b)
Returns 1 (when $a is greater than $b) or 0 (when $a is equal to $b) or -1 (when $a is lesser than $b).
$a
$b
$result = $Collator->eq($a, $b)
$result = $Collator->ne($a, $b)
$result = $Collator->gt($a, $b)
$result = $Collator->ge($a, $b)
$result = $Collator->lt($a, $b)
$result = $Collator->le($a, $b)
They works like the same name operators as theirs.
@sorted = $Collator->sort(@not_sorted)
Sorts a list of strings by tanjun shogo: 'the simple collation'.
@sorted = $Collator->sortYomi(@not_sorted)
Sorts a list of references to arrays of (spell, reading) by yomi-hyoki shogo: 'the collation using readings and spells'.
@sorted = $Collator->sortDaihyo(@not_sorted)
Sorts a list of references to arrays of (spell, reading) by kan'i-daihyo-yomi shogo: 'the simplified representative reading collation'.
$sortKey = $Collator->getSortKey($string)
Returns a sort key.
You compare the sort keys using a binary comparison and get the result of the comparison of the strings.
$Collator->getSortKey($a) cmp $Collator->getSortKey($b) is equivalent to $Collator->cmp($a, $b)
$position = $Collator->index($string, $substring)
($position, $length) = $Collator->index($string, $substring)
If $substring matches a part of $string, returns the position of the first occurrence of the matching part in scalar context; in list context, returns a two-element list of the position and the length of the matching part.
$substring
$string
Notice that the length of the matching part may differ from the length of $substring.
If $substring does not match any part of $string, returns -1 in scalar context and an empty list in list context.
-1
e.g. you say
use ShiftJIS::Collate; use ShiftJIS::String qw(substr); my $Col = ShiftJIS::Collate->new( level => $level ); my $str = "* ‚Ђ炪‚ȂƃJƒ^ƒJƒi‚̓Œƒxƒ‹‚R‚Å‚Í“™‚µ‚¢‚©‚È�B"; my $sub = "‚©‚È"; my $match; if(my @tmp = $Col->index($str, $sub)){ $match = substr($str, $tmp[0], $tmp[1]); }
If $level is 1, you get "‚ª‚È"; if $level is 2 or 3, you get "ƒJƒi"; if $level is 4 or 5, you get "‚©‚È".
$level
"‚ª‚È"
"ƒJƒi"
"‚©‚È"
If your substr function is not oriented to Shift_JIS, specify true as position_in_bytes. See "Constructor and Tailoring".
substr
position_in_bytes
The following criteria are considered in order until the collation order is determined. By default, Levels 1 to 4 are applied and Level 5 is ignored (as JIS does).
The character class early appeared in the following list is smaller.
Space characters, Symbols and Punctuations, Digits, Greek Letters, Cyrillic Letters, Latin letters, Kana letters, Kanji ideographs, and Geta mark.
In the class, alphabets are collated alphabetically; kana letters are AIUEO-betically (in the Gozyuon order); kanji are in the JIS X 0208 order.
Characters that do not belong to any character class are ignored and skipped for collation.
In kana, the order is as shown the following list.
A voiceless kana, the voiced, then the semi-voiced (if exists). (eg. Ka < Ga; Ha < Ba < Pa)
A small Latin is lesser than the corresponding Capital.
In kana, the order is as shown the following list. see "Replacement of PROLONGED SOUND MARK and ITERATION MARKs".
Replaced PROLONGED SOUND MARK (U+30FC); Small Kana; Replaced ITERATION MARK (U+309D, U+309E, U+30FD or U+30FE); then, Normal Kana.
Any hiragana is lesser than the corresponding katakana.
A character that belongs to the block Halfwidth and Fullwidth Forms is greater than the corresponding normal character.
Halfwidth and Fullwidth Forms
BN: According to the JIS standard, the level 5 should be ignored.
There are three kanji classes. This modules provides the Classes 1 and 2.
It comprises five kanji-like characters, i.e. U+3003, U+3005, U+4EDD, U+3006, U+3007. Any kanji except U+4EDD are ignored on collation.
It comprises JIS level 1 and 2 kanji in addition to the minimal kanji class. Sorted in the JIS codepoint order. Any kanji excepting those defined by JIS X 0208 are ignored on collation.
All the CJK Unified Ideographs in addition to the minimal kanji class. Sorted in the Unicode codepoint order.
RFC1345 UCS [*5] U+309D HIRAGANA ITERATION MARK [+5] U+309E HIRAGANA VOICED ITERATION MARK [-6] U+30FC KATAKANA-HIRAGANA PROLONGED SOUND MARK [*6] U+30FD KATAKANA ITERATION MARK [+6] U+30FE KATAKANA VOICED ITERATION MARK
To represent Japanese characters, RFC 1345 Mnemonic characters enclosed by brackets are used below.
These characters, if replaced, are secondary equal to the replacing kana, while ternary not equal to.
The PROLONGED MARK is repleced to a normal vowel or nasal katakana corresponding to the preceding kana if exists.
eg. [Ka][-6] to [Ka][A6] [bi][-6] to [bi][I6] [Pi][YU][-6] to [Pi][YU][U6] [N6][-6] to [N6][N6]
The ITERATION MARKs (VOICELESS) are repleced to a normal kana corresponding to the preceding kana if exists.
eg. [Ka][*6] to [Ka][Ka] [Do][*5] to [Do][to] [n5][*5] to [n5][n5] [Pu][*6] to [Pu][Hu] [Pi][YU][*6] to [Pi][YU][Yu]
The VOICED ITERATION MARKs are repleced to a voiced kana corresponding to the preceding kana if exists.
eg. [ha][+5] to [ha][ba] [Pu][+5] to [Pu][bu] [Ko][+6] to [Ko][Go] [U6][+6] to [U6][Vu]
Otherwise, no replacement occurs. Especially in the cases when these marks follow any character except kana.
The unreplaced characters are primary greater than any kana.
eg. CJK Ideograph followed by PROLONGED SOUND MARK Digit followed by ITERATION MARK [A6][+6] ([A6] has no voiced variant)
For example, the Japanese string [Pa][-6][Ru] (Perl in kana) has three collation elements: KATAKANA PA, PROLONGED SOUND MARK replaced by KATAKANA A, and KATAKANA RU.
[Pa][-6][Ru]
Perl
KATAKANA PA
PROLONGED SOUND MARK replaced by KATAKANA A
KATAKANA RU
[Pa][-6][Ru] is converted to [Pa][A6][Ru] by replacement. primary equal to [ha][a5][ru]. secondary equal to [pa][a5][ru], greater than [ha][a5][ru]. tertiary equal to [pa][-6][ru], lesser than [Pa][A6][Ru]. quartenary greater than [pa][-6][ru].
[cf. the article 6.2, JIS X 4061] (1) charset: Shift_JIS. (2) No limit of the number of characters in the string considered to collate. (3) No character class is added. (4) The following characters are added as collation elements. IDEOGRAPHIC SPACE in the space class. ACUTE ACCENT, GRAVE ACCENT, DIAERESIS, CIRCUMFLEX ACCENT in the class of descriptive symbols. APOSTROPHE, QUOTATION MARK in the class of parentheses. HYPHEN-MINUS in the class of mathematical symbols. (5) Collation of Latin alphabets with macron and with circumflex is not supported. (6) L<Kanji Classes>: the minimal kanji class (Five kanji-like chars). the basic kanji class (Levels 1 and 2 kanji, JIS)..
Tomoyuki SADAHIRO
bqw10602@nifty.com http://homepage1.nifty.com/nomenclator/perl/ This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
JIS X 4061 [Collation of Japanese character strings]
JIS X 0208 [7-bits and 8-bits double byte coded Kanji sets for information interchange]
JIS X 0221 [Information technology - Universal Multiple-Octet Coded Character Set (UCS) - part 1 : Architectute and Basic Multilingual Plane]. This is a translation of ISO/IEC 10646-1.
RFC 1345 [Character Mnemonics & Character Sets]
ShiftJIS::String
ShiftJIS::Regexp
1 POD Error
The following errors were encountered while parsing the POD:
Non-ASCII character seen before =encoding in '‚Ђ炪‚ȂƃJƒ^ƒJƒi‚̓Œƒxƒ‹‚R‚Å‚Í“™‚µ‚¢‚©‚È�B";'. Assuming CP1252
To install ShiftJIS::Collate, copy and paste the appropriate command in to your terminal.
cpanm
cpanm ShiftJIS::Collate
CPAN shell
perl -MCPAN -e shell install ShiftJIS::Collate
For more information on module installation, please visit the detailed CPAN module installation guide.