ShiftJIS::Regexp - Shift_JIS-oriented regexps on the byte-oriented perl
use ShiftJIS::Regexp qw(:all); match('‚ ‚¨‚P‚Q', '\p{Hiragana}{2}\p{Digit}{2}'); # that is equivalant to this: match('‚ ‚¨‚P‚Q', '\pH{2}\pD{2}'); match('‚ ‚¢‚¢‚¤‚¤‚¤', '^‚ ‚¢+‚¤{3}$'); replace($str, 'A', '‚`', 'g');
This module provides some functions to use Shift_JIS-oriented regexps on the byte-oriented perl.
The legal Shift_JIS character in this module must match the following regexp:
[\x00-\x7F\xA1-\xDF]|[\x81-\x9F\xE0-\xFC][\x40-\x7E\x80-\xFC]
re(PATTERN)
re(PATTERN, MODIFIER)
Returns regexp parsable by the byte-oriented perl.
PATTERN is specified as a string.
PATTERN
MODIFIER is specified as a string.
MODIFIER
i case-insensitive pattern (only for ascii alphabets) I case-insensitive pattern (greek, cyrillic, fullwidth latin) j hiragana-katakana-insensitive pattern s treat string as single line m treat string as multiple lines x ignore whitespace (i.e. [ \n\r\t\f], but not comments!) unless backslashed or inside a character class o once parsed (not compiled!) and the result is cached internally.
re('^ƒRƒ“ƒsƒ…�[ƒ^�[?$') matches 'ƒRƒ“ƒsƒ…�[ƒ^�[' or 'ƒRƒ“ƒsƒ…�[ƒ^'.
re('^ƒRƒ“ƒsƒ…�[ƒ^�[?$')
'ƒRƒ“ƒsƒ…�[ƒ^�['
'ƒRƒ“ƒsƒ…�[ƒ^'
re('^‚ç‚‚¾$','j') matches '‚ç‚‚¾', 'ƒ‰ƒNƒ_', '‚çƒN‚¾', etc.
re('^‚ç‚‚¾$','j')
'‚ç‚‚¾'
'ƒ‰ƒNƒ_'
'‚çƒN‚¾'
o modifier
o
while(<DATA>){ print replace($_, '(perl)', '<strong>$1</strong>', 'igo'); } is more efficient than while(<DATA>){ print replace($_, '(perl)', '<strong>$1</strong>', 'ig'); } because in the latter case the pattern is parsed every time whenever the function is called.
match(STRING, PATTERN)
match(STRING, PATTERN, MODIFIER)
emulation of m// operator for the Shift_JIS encoding.
m//
i case-insensitive pattern (only for ascii alphabets) I case-insensitive pattern (greek, cyrillic, fullwidth latin) j hiragana-katakana-insensitive pattern s treat string as single line m treat string as multiple lines x ignore whitespace (i.e. [ \n\r\t\f], but not comments!) unless backslashed or inside a character class g match globally z tell the function the pattern matches zero-length substring (sorry, due to the poor auto-detection) o once parsed (not compiled!) and the result is cached internally.
replace(STRING or SCALAR REF, PATTERN, REPLACEMENT)
replace(STRING or SCALAR REF, PATTERN, REPLACEMENT, MODIFIER)
emulation of s/// operator for the Shift_JIS encoding.
s///
If a reference of scalar variable is specified as the first argument, returns the number of substitutions made. If a string is specified as the first argument, returns the substituted string and the specified string is unaffected.
my $str = '‹à‚P‚T‚R‚O‚O‚O‚O‰~'; 1 while replace(\$str, '(\pD)(\pD{3})(?!\pD)', '$1�C$2'); print $str; # ‹à‚P�C‚T‚R‚O�C‚O‚O‚O‰~
i case-insensitive pattern (only for ascii alphabets) I case-insensitive pattern (greek, cyrillic, fullwidth latin) j hiragana-katakana-insensitive pattern s treat string as single line treat string as single line m treat string as multiple lines x ignore whitespace (i.e. [ \n\r\t\f], but not comments!) unless backslashed or inside a character class g match globally z tell the function the pattern matches zero-length substring (sorry, due to the poor auto-detection) o once parsed (not compiled!) and the result is cached internally.
jsplit(PATTERN or ARRAY REF of [PATTERN, MODIFIER], STRING)
jsplit(PATTERN or ARRAY REF of [PATTERN, MODIFIER], STRING, LIMIT)
This function emulates CORE::split.
CORE::split
If not in list context, these functions do only return the number of fields found, but do not split into the @_ array.
@_
But ' ' as PATTERN has no special meaning; when you want to split the string on whitespace, you can use splitspace() function.
' '
splitspace()
jsplit('�^', '‚ ‚¢‚¤�^‚¦‚¨ƒ�^');
If you want to pass pattern with modifiers, specify an arrayref of [PATTERN, MODIFIER] as the first argument.
[PATTERN, MODIFIER]
jsplit([ '‚ ', 'jo' ], '01234‚ ‚¢‚¤‚¦‚¨ƒAƒCƒEƒGƒI');
Or you can say (see "Embedded Modifiers"):
jsplit('(?jo)‚ ', '01234‚ ‚¢‚¤‚¦‚¨ƒAƒCƒEƒGƒI');
i do case-insensitive pattern matching (only for ascii alphabets) I do case-insensitive pattern matching (greek, cyrillic, fullwidth latin) j do hiragana-katakana-insensitive pattern matching s treat string as single line m treat string as multiple lines x ignore whitespace (i.e. [ \n\r\t\f], but not comments!) unless backslashed or inside a character class o once parsed (not compiled!) and the result is cached internally.
splitspace(STRING)
splitspace(STRING, LIMIT)
This function emulates CORE::split ' ', STRING and returns the array given by split on whitespace including IDEOGRAPHIC SPACE. Leading whitespace characters do not produce any field.
CORE::split ' ', STRING
splitchar(STRING)
splitchar(STRING, LIMIT)
This function emulates CORE::split //, STRING and returns the array given by split of the specified string into characters.
CORE::split //, STRING
regexp meaning ^ match the start of the string match the start of any line with 'm' modifier $ match the end of the string, or before newline at the end match the end of any line with 'm' modifier . match any character except \n match any character with 's' modifier \A only at beginning of string \Z at the end of the string, or before newline at the end \z only at the end of the string (eq. '(?!\n)\Z') \C match a single C char (octet), i.e. [\0-\xFF] in perl. \j match any character, i.e. [\0-\x{FCFC}] in this module. \J match any character except \n, i.e. [^\n] in this module. * \j and \J are extensions by this module. e.g. match($_, '(\j{5})\z') returns last five chars including \n at the end match($_, '(\J{5})\Z') returns last five chars excluding \n at the end \a alarm (BEL) \b backspace (BS) * within character classes * \e escape (ESC) \f form feed (FF) \n newline (LF, NL) \r return (CR) \t tab (HT, TAB) \0 null (NUL) \ooo octal single-byte character \xhh hexadecimal single-byte character \x{hhhh} hexadecimal double-byte character \c[ control character e.g. \012 \123 \x5c \x5C \x{824F} \x{9Fae} \cA \cZ \c^ \c?
\d [\d] [0-9] \D [\D] [^0-9] \w [\w] [0-9A-Z_a-z] \W [\W] [^0-9A-Z_a-z] \s [\s] [\t\n\r\f ] \S [\S] [^\t\n\r\f ] \p{Xdigit} \pX [[:xdigit:]] [0-9A-Fa-f] \p{Digit} \pD [[:digit:]] [0-9‚O-‚X] \p{Upper} \pU [[:upper:]] [A-Z‚`-‚y] \p{Lower} \pL [[:lower:]] [a-z‚�-‚š] \p{Alpha} \pA [[:alpha:]] [A-Za-z‚`-‚y‚�-‚š] \p{Alnum} [\pA\pD] [[:alnum:]] [0-9A-Za-z‚O-‚X‚`-‚y‚�-‚š] \p{Word} \pW [[:word:]] [_\p{Digit}\p{European}\p{Kana}\p{Kanji}] \p{Punct} \pP [[:punct:]] [!-/:-@[-`{-~¡-¥�A-�I�L-�Q�\-�¬�¸-�¿�È-�Î�Ú-�è�ð-�÷�ü„Ÿ-„¾] \p{Space} \pS [[:space:]] [\t\n\r\f\x20\x{8140}] \p{Graph} \pG [[:graph:]] [^\0-\x20\x7F\x{8140}] \p{Print} [\pS\pG] [[:print:]] [^\0-\x08\x0B\x0E-\x1F\x7F] \p{Cntrl} \pC [[:cntrl:]] [\x00-\x1F] \p{Roman} \pR [[:roman:]] [\x00-\x7F] \p{ASCII} [[:ascii:]] [\p{Roman}] \p{Hankaku} [[:hankaku:]] [\xA1-\xDF] \p{Zenkaku} \pZ [[:zenkaku:]] [\x{8140}-\x{FCFC}] \p{X0201} [[:x0201:]] [\x00-\x7F\xA1-\xDF] \p{X0208} [[:x0208:]] [\x{8140}-�¬�¸-�¿�È-�Î�Ú-�è�ð-�÷�ü‚O-‚X‚`-‚y‚�-‚š‚Ÿ-‚ñƒ@-ƒ–ƒŸ-ƒ¶ƒ¿-ƒÖ„@-„`„p-„‘„Ÿ-„¾ˆŸ-˜r˜Ÿ-ê¤] \p{JIS} \pJ [[:jis:]] [\p{X0201}\p{X0208}] \p{NEC} \pN [[:nec:]] [\x{8740}-\x{875D}\x{875f}-\x{8775}\x{877E}-\x{879c}\x{ed40}-\x{eeec}\x{eeef}-\x{eefc}] \p{IBM} \pI [[:ibm:]] [\x{fa40}-\x{fc4b}] \p{Vendor} \pV [[:vendor:]] [\p{NEC}\p{IBM}] \p{MSWin} \pM [[:mswin:]] [\p{JIS}\p{NEC}\p{IBM}] \p{Latin} [[:latin:]] [A-Za-z] \p{FullLatin} [[:fulllatin:]] [‚`-‚y‚�-‚š] \p{Greek} [[:greek:]] [ƒŸ-ƒ¶ƒ¿-ƒÖ] \p{Cyrillic} [[:cyrillic:]] [„@-„`„p-„‘] \p{European} \pE [[:european:]] [A-Za-z‚`-‚y‚�-‚šƒŸ-ƒ¶ƒ¿-ƒÖ„@-„`„p-„‘] \p{HalfKana} [[:halfkana:]] [¦-ß] \p{Hiragana} \pH [[:hiragana:]] [‚Ÿ-‚ñ�J�K�T�U] \p{Katakana} \pK [[:katakana:]] [ƒ@-ƒ–�[�R�S] \p{FullKana} [\pH\pK] [[:fullkana:]] [‚Ÿ-‚ñƒ@-ƒ–�J�K�[�T�U�R�S] \p{Kana} [[:kana:]] [¦-ß‚Ÿ-‚ñƒ@-ƒ–�J�K�[�T�U�R�S] \p{Kanji0} \p0 [[:kanji0:]] [�V-�Z] \p{Kanji1} \p1 [[:kanji1:]] [ˆŸ-˜r] \p{Kanji2} \p2 [[:kanji2:]] [˜Ÿ-ê¤] \p{Kanji} [\p0\p1\p2] [[:kanji:]] [�V-�ZˆŸ-˜r˜Ÿ-ê¤] \p{BoxDrawing} \pB [[:boxdrawing:]] [„Ÿ-„¾]
\p{NEC} matches an NEC special character or an NEC-selected IBM extended character.
\p{NEC}
\p{IBM} matches an IBM extended character.
\p{IBM}
\p{Vendor} matches a character of vendor-defined characters in Microsoft CP932, i.e. equivalent to [\p{NEC}\p{IBM}].
\p{Vendor}
[\p{NEC}\p{IBM}]
\p{MSWin} matches a character of Microsoft CP932.
\p{MSWin}
\p{Kanji0} matches a kanji of the minimum kanji class of JIS X 4061; \p{Kanji1}, of the level 1 kanji of JIS X 0208; \p{Kanji2}, of the level 2 kanji of JIS X 0208; \p{Kanji}, of the basic kanji class of JIS X 4061.
\p{Kanji0}
\p{Kanji1}
\p{Kanji2}
\p{Kanji}
\p{Prop}, \P{^Prop}, [\p{Prop}], etc. are equivalent to each other; and their complements are \P{Prop}, \p{^Prop}, [\P{Prop}], [^\p{Prop}], etc.
\p{Prop}
\P{^Prop}
[\p{Prop}]
\P{Prop}
\p{^Prop}
[\P{Prop}]
[^\p{Prop}]
\pP, \P^P, [\pP], etc. are equivalent to each other; and their complements are \PP, \p^P, [\PP], [^\pP], etc.
\pP
\P^P
[\pP]
\PP
\p^P
[\PP]
[^\pP]
[[:class:]]is equivalent to [^[:^class:]]; and their complements are [[:^class:]] or [^[:class:]].
[[:class:]]
[^[:^class:]]
[[:^class:]]
[^[:class:]]
In \p{Prop}, \P{Prop}, [:class:] expressions, Prop and class are case-insensitive (e.g. \p{digit}, [:BoxDrawings:]).
[:class:]
Prop
class
\p{digit}
[:BoxDrawings:]
Prefixes Is and In for \p{Prop} and \P{Prop} (e.g. \p{IsProp}, \P{InProp}, etc.) are optional. But \p{isProp}, \p{ISProp}, etc. are not ok, as Is and In are not case-insensitive. Using of Is and In is deplicated since they may conflict with a property name beginning with 'is' or 'in' in future.
Is
In
\p{IsProp}
\P{InProp}
\p{isProp}
\p{ISProp}
'is'
'in'
Ranges in character class are supported.
The order of Shift_JIS characters is: 0x00 .. 0x7F, 0xA1 .. 0xDF, 0x8140 .. 0x9FFC, 0xE040 .. 0xFCFC.
0x00 .. 0x7F, 0xA1 .. 0xDF, 0x8140 .. 0x9FFC, 0xE040 .. 0xFCFC
So [\0-\x{fcfc}] matches any one Shift_JIS character.
[\0-\x{fcfc}]
In character classes, any character or byte sequence that does not match any one Shift_JIS character, e.g. re('[\xA0-\xFF]'), is croaked.
re('[\xA0-\xFF]')
Character classes that match non-Shift_JIS substring are not supported (use \C or alternation).
\C
Since the version 0.13, the POSIX character equivalent classes [=cc=] are supported. e.g. [[=‚ =]] is identical to [‚Ÿƒ@§‚ ƒA±]; [[=P=]] to [pP‚�‚o]; [[=4=]] to [4‚S]. They are used in a character class, like [[=cc=]], [[=p=][=e=][=r=][=l=]].
[=cc=]
[[=‚ =]]
[‚Ÿƒ@§‚ ƒA±]
[[=P=]]
[pP‚�‚o]
[[=4=]]
[4‚S]
[[=cc=]]
[[=p=][=e=][=r=][=l=]]
As cc in [=cc=], any character literal or meta chatacter (\xhh, \x{hhhh}) that belongs to the character equivalents can be used. e.g. [=‚ =], [=ƒA=], [=\x{82A0}=], [=\xB1=], etc. have identical meanings.
cc
\xhh
\x{hhhh}
[=‚ =]
[=ƒA=]
[=\x{82A0}=]
[=\xB1=]
[[=‚©=]] matches '‚©', 'ƒJ', '¶', '‚ª', 'ƒK', '¶Þ', 'ƒ•' ('¶Þ' is a two-character string, but one collation element, HALFWIDTH FORM FOR KATAKANA LETTER GA.
[[=‚©=]]
'‚©'
'ƒJ'
'¶'
'‚ª'
'ƒK'
'¶Þ'
'ĥ'
HALFWIDTH FORM FOR KATAKANA LETTER GA
[[===]] matches EQUALS SIGN or FULLWIDTH EQUALS SIGN; [[=[=]] matches LEFT SQUARE BRACKET or FULLWIDTH LEFT SQUARE BRACKET; [[=]=]] matches RIGHT SQUARE BRACKET or FULLWIDTH RIGHT SQUARE BRACKET; [[=\=]] matches YEN SIGN or FULLWIDTH YEN SIGN.
[[===]]
EQUALS SIGN
FULLWIDTH EQUALS SIGN
[[=[=]]
LEFT SQUARE BRACKET
FULLWIDTH LEFT SQUARE BRACKET
[[=]=]]
RIGHT SQUARE BRACKET
FULLWIDTH RIGHT SQUARE BRACKET
[[=\=]]
YEN SIGN
FULLWIDTH YEN SIGN
Parsing (?{ ... }) or (??{ ... }) assertions is carried out without any special care of double-byte characters.
(?{ ... })
(??{ ... })
(?{ ... }) assertions are disallowed in match() or replace() functions by perl due to security concerns. Use them via re() function inside your scope.
match()
replace()
re()
use ShiftJIS::Regexp qw(:all); use re 'eval'; $::res = 0; $_ = 'ƒ|' x 8; my $regex = re(q/ \j*? (?{ $cnt = 0 }) ( ƒ| (?{ local $cnt = $cnt + 1; }) )* ƒ|ƒ|ƒ| (?{ $::res = $cnt }) /, 'x'); /$regex/; print $::res; # 5
Since version 0.15, embedded modifiers are extended.
An embedded modifier, (?iIjsmxo), that appears at the beginning of the 'regexp' or that follows one of regexps ^, \A, or \G at the beginning of the 'regexp' is allowed to contain I, j, o modifiers.
(?iIjsmxo)
^
\A
\G
I
j
e.g. (?sm)pattern ^(?i)pattern \G(?j)pattern \A(?ijo)pattern
And match('ƒG', '(?i)ƒg') returns false (Good result) even on Perl below 5.005, since it works like match('ƒG', 'ƒg', 'i').
match('ƒG', '(?i)ƒg')
match('ƒG', 'ƒg', 'i')
A legal Shift_JIS character in this module must match the following regexp:
Any string from external resource should be checked by issjis() function of ShiftJIS::String, excepting you know it is surely encoded in Shift_JIS.
issjis()
ShiftJIS::String
Use of an illegal Shift_JIS string may lead to odd results.
Some Shift_JIS double-byte characters have one of [\x40-\x7E] as the trail byte.
[\x40-\x7E]
@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~
The Perl lexer doesn't take any care to these characters, so they sometimes make trouble. e.g. the quoted literal "•\" causes fatal error, since its trail byte 0x5C backslashes the closing quote.
"•\"
0x5C
Such a problem doesn't arise when the string is gotten from any external resource. But writing the script containing Shift_JIS double-byte characters needs the greatest care.
The use of single-quoted heredoc, << '', or \xhh meta characters is recommended in order to define a Shift_JIS string literal.
<< ''
The safe ASCII-graphic characters, [\x21-\x3F], are:
[\x21-\x3F]
!"#$%&'()*+,-./0123456789:;<=>?
They are preferred as the delimiter of quote-like operators.
The \U, \L, \Q, \E, and interpolation are not considered. If necessary, use them in "" (or qq//) operators in the argument list.
\U
\L
\Q
\E
""
qq//
The regexps of the word boundary, \b and \B, don't work correctly.
\b
\B
Never pass any regexp containing '(?i)' on perl below 5.005. Pass 'i' modifier as the second argument. (On Perl 5.005 or later, '(?i)' is allowed because '(?-i:RE)' prevents it from wrong matching)
'(?i)'
'i'
'(?-i:RE)'
e.g.
match('ƒG', '(?i)ƒg') returns true on Perl below 5.005 (Wrong). match('ƒG', '(?i)ƒg') returns false on Perl 5.005 or later (Good). match('ƒG', 'ƒg', 'i') returns false, ok. # The trail byte of 'ƒG' is 'G' and that of 'ƒg' is 'g';
(see also "Embedded Modifiers")
The i, I and j modifiers are invalid to \p{}, \P{}, and POSIX [: :]. (e.g. \p{IsLower}, [:lower:], etc). So use re('\p{IsAlpha}') instead of re('\p{IsLower}', 'iI').
i
\p{}
\P{}
[: :]
\p{IsLower}
[:lower:]
re('\p{IsAlpha}')
re('\p{IsLower}', 'iI')
The look-behind assertion like (?<=[A-Z]) is not prevented from matching trail byte of the previous double byte character: e.g. match("ƒAƒCƒE", '(?<=[A-Z])(\p{InKana})') returns ('ƒC').
(?<=[A-Z])
match("ƒAƒCƒE", '(?<=[A-Z])(\p{InKana})')
('ƒC')
Use of not greedy regexp, which can match empty string, such as .?? and \d*?, as the PATTERN in jsplit(), may cause failure to the emulation of CORE::split.
.??
\d*?
jsplit()
Tomoyuki SADAHIRO
bqw10602@nifty.com http://homepage1.nifty.com/nomenclator/perl/ This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
1 POD Error
The following errors were encountered while parsing the POD:
Non-ASCII character seen before =encoding in 'match('‚ ‚¨‚P‚Q','. Assuming CP1252
To install ShiftJIS::Regexp, copy and paste the appropriate command in to your terminal.
cpanm
cpanm ShiftJIS::Regexp
CPAN shell
perl -MCPAN -e shell install ShiftJIS::Regexp
For more information on module installation, please visit the detailed CPAN module installation guide.