The Perl Toolchain Summit needs more sponsors. If your company depends on Perl, please support this very important event.
 / 
ShiftJIS-Regexp-0.16
River stage zero No dependents
1 ++
/ ShiftJIS::Regexp

NAME

ShiftJIS::Regexp - Shift_JIS-oriented regexps on the byte-oriented perl

SYNOPSIS

  use ShiftJIS::Regexp qw(:all);

  match('‚ ‚¨‚P‚Q', '\p{InHiragana}{2}\p{IsDigit}{2}');
  match('‚ ‚¢‚¢‚¤‚¤‚¤', '^‚ ‚¢+‚¤{3}$');
  replace($str, 'A', '‚`', 'g');

DESCRIPTION

This module provides some functions to use Shift_JIS-oriented regexps on the byte-oriented perl.

The legal Shift_JIS character in this module must match the following regexp:

    [\x00-\x7F\xA1-\xDF]|[\x81-\x9F\xE0-\xFC][\x40-\x7E\x80-\xFC]

Functions

re(PATTERN)
re(PATTERN, MODIFIER)

Returns regexp parsable by the byte-oriented perl.

PATTERN is specified as a string.

MODIFIER is specified as a string.

     i  case-insensitive pattern (only for ascii alphabets)
     I  case-insensitive pattern (greek, cyrillic, fullwidth latin)
     j  hiragana-katakana-insensitive pattern

     s  treat string as single line
     m  treat string as multiple lines
     x  ignore whitespace (i.e. [ \n\r\t\f], but not comments!)
        unless backslashed or inside a character class

     o  once parsed (not compiled!) and the result is cached internally.

re('^ƒRƒ“ƒsƒ…�[ƒ^�[?$') matches 'ƒRƒ“ƒsƒ…�[ƒ^�[' or 'ƒRƒ“ƒsƒ…�[ƒ^'.

re('^‚ç‚­‚¾$','j') matches '‚ç‚­‚¾', 'ƒ‰ƒNƒ_', '‚çƒN‚¾', etc.

o modifier

     while(<DATA>){
       print replace($_, '(perl)', '<strong>$1</strong>', 'igo');
     }
        is more efficient than

     while(<DATA>){
       print replace($_, '(perl)', '<strong>$1</strong>', 'ig');
     }

     because in the latter case the pattern is parsed every time
     whenever the function is called.
match(STRING, PATTERN)
match(STRING, PATTERN, MODIFIER)

emulation of m// operator for the Shift_JIS encoding.

PATTERN is specified as a string.

MODIFIER is specified as a string.

     i  case-insensitive pattern (only for ascii alphabets)
     I  case-insensitive pattern (greek, cyrillic, fullwidth latin)
     j  hiragana-katakana-insensitive pattern

     s  treat string as single line
     m  treat string as multiple lines
     x  ignore whitespace (i.e. [ \n\r\t\f], but not comments!)
        unless backslashed or inside a character class
     g  match globally
     z  tell the function the pattern matches zero-length substring
           (sorry, due to the poor auto-detection)

     o  once parsed (not compiled!) and the result is cached internally.
replace(STRING or SCALAR REF, PATTERN, REPLACEMENT)
replace(STRING or SCALAR REF, PATTERN, REPLACEMENT, MODIFIER)

emulation of s/// operator for the Shift_JIS encoding.

If a reference of scalar variable is specified as the first argument, returns the number of substitutions made. If a string is specified as the first argument, returns the substituted string and the specified string is unaffected.

    my $d = '\p{IsDigit}';
    my $str = '‹à‚P‚T‚R‚O‚O‚O‚O‰~';
    1 while replace(\$str, "($d)($d$d$d)(?!$d)", '$1�C$2');
    print $str; # ‹à‚P�C‚T‚R‚O�C‚O‚O‚O‰~

MODIFIER is specified as a string.

     i  case-insensitive pattern (only for ascii alphabets)
     I  case-insensitive pattern (greek, cyrillic, fullwidth latin)
     j  hiragana-katakana-insensitive pattern

     s  treat string as single line  treat string as single line
     m  treat string as multiple lines
     x  ignore whitespace (i.e. [ \n\r\t\f], but not comments!)
        unless backslashed or inside a character class
     g  match globally
     z  tell the function the pattern matches zero-length substring
           (sorry, due to the poor auto-detection)

     o  once parsed (not compiled!) and the result is cached internally.
jsplit(PATTERN or ARRAY REF of [PATTERN, MODIFIER], STRING)
jsplit(PATTERN or ARRAY REF of [PATTERN, MODIFIER], STRING, LIMIT)

This function emulates CORE::split.

If not in list context, these functions do only return the number of fields found, but do not split into the @_ array.

PATTERN is specified as a string.

But ' ' as PATTERN has no special meaning; when you want to split the string on whitespace, you can use splitspace() function.

    jsplit('�^', '‚ ‚¢‚¤�^‚¦‚¨ƒ�^');

If you want to pass pattern with modifiers, specify an arrayref of [PATTERN, MODIFIER] as the first argument.

    jsplit([ '‚ ', 'jo' ], '01234‚ ‚¢‚¤‚¦‚¨ƒAƒCƒEƒGƒI');

Or you can say (see "Embedded Modifiers"):

    jsplit('(?jo)‚ ', '01234‚ ‚¢‚¤‚¦‚¨ƒAƒCƒEƒGƒI');

MODIFIER is specified as a string.

     i  do case-insensitive pattern matching (only for ascii alphabets)
     I  do case-insensitive pattern matching
        (greek, cyrillic, fullwidth latin)
     j  do hiragana-katakana-insensitive pattern matching

     s  treat string as single line
     m  treat string as multiple lines
     x  ignore whitespace (i.e. [ \n\r\t\f], but not comments!)
        unless backslashed or inside a character class

     o  once parsed (not compiled!) and the result is cached internally.
splitspace(STRING)
splitspace(STRING, LIMIT)

This function emulates CORE::split ' ', STRING and returns the array given by split on whitespace including IDEOGRAPHIC SPACE. Leading whitespace characters do not produce any field.

splitchar(STRING)
splitchar(STRING, LIMIT)

This function emulates CORE::split //, STRING and returns the array given by split of the specified string into characters.

Regexps

   regexp          meaning

   ^               match the start of the string
                   match the start of any line with 'm' modifier

   $               match the end of the string, or before newline at the end
                   match the end of any line with 'm' modifier

   .               match any character except \n
                   match any character with 's' modifier

   \A              only at beginning of string
   \Z              at the end of the string, or before newline at the end
   \z              only at the end of the string (eq. '(?!\n)\Z')

   \C              match a single C char (octet), i.e. [\0-\xFF] in perl.
   \j              match any character, i.e. [\0-\x{FCFC}] in this module.
   \J              match any character except \n, i.e. [^\n] in this module.

     * \j and \J are extensions by this module. e.g.

        match($_, '(\j{5})\z') returns last five chars including \n at the end
        match($_, '(\J{5})\Z') returns last five chars excluding \n at the end

   \a              alarm      (BEL)
   \b              backspace  (BS) * within character classes *
   \t              tab        (HT, TAB)
   \n              newline    (LF, NL)
   \f              form feed  (FF)
   \r              return     (CR)
   \e              escape     (ESC)

   \0              null       (NUL)

   \ooo            octal single-byte character
   \xhh            hexadecimal single-byte character
   \x{hhhh}        hexadecimal double-byte character
   \c[             control character

      e.g. \012 \123 \x5c \x5C \x{824F} \x{9Fae} \cA \cZ \c^ \c?

   regexp           equivalent character class

   \d               [\d]              [0-9]
 ! \D               [\D]              [^0-9]
   \w               [\w]              [0-9A-Z_a-z]
 ! \W               [\W]              [^0-9A-Z_a-z]
   \s               [\s]              [\t\n\r\f ]
 ! \S               [\S]              [^\t\n\r\f ]

   \p{IsDigit}      [[:digit:]]       [0-9‚O-‚X]
 ! \P{IsDigit}      [[:^digit:]]      [^0-9‚O-‚X]
   \p{IsUpper}      [[:upper:]]       [A-Z‚`-‚y]
 ! \P{IsUpper}      [[:^upper:]]      [^A-Z‚`-‚y]
   \p{IsLower}      [[:lower:]]       [a-z‚�-‚š]
 ! \P{IsLower}      [[:^lower:]]      [^a-z‚�-‚š]
   \p{IsAlpha}      [[:alpha:]]       [A-Za-z‚`-‚y‚�-‚š]
 ! \P{IsAlpha}      [[:^alpha:]]      [^A-Za-z‚`-‚y‚�-‚š]
   \p{IsAlnum}      [[:alnum:]]       [0-9A-Za-z‚O-‚X‚`-‚y‚�-‚š]
 ! \P{IsAlnum}      [[:^alnum:]]      [^0-9A-Za-z‚O-‚X‚`-‚y‚�-‚š]

   \p{IsWord}       [[:word:]]
          [0-9A-Z_a-z‚O-‚X‚`-‚y‚�-‚šƒŸ-ƒ¶ƒ¿-ƒÖ„@-„`„p-„‘¦-ß‚Ÿ-‚ñƒ@-ƒ–�J�K�R-�[ˆŸ-˜r˜Ÿ-ê¤]
 ! \P{IsWord}       [[:^word:]]
          [^0-9A-Z_a-z‚O-‚X‚`-‚y‚�-‚šƒŸ-ƒ¶ƒ¿-ƒÖ„@-„`„p-„‘¦-ß‚Ÿ-‚ñƒ@-ƒ–�J�K�R-�[ˆŸ-˜r˜Ÿ-ê¤]

   \p{IsPunct}      [[:punct:]]
                [!-/:-@[-`{-~¡-¥�A-�I�L-�Q�\-�¬�¸-�¿�È-�Î�Ú-�è�ð-�÷�ü„Ÿ-„¾]
 ! \P{IsPunct}      [[:^punct:]]
                [^!-/:-@[-`{-~¡-¥�A-�I�L-�Q�\-�¬�¸-�¿�È-�Î�Ú-�è�ð-�÷�ü„Ÿ-„¾]
   \p{IsSpace}      [[:space:]]       [\t\n\r\f \x{8140}]
 ! \P{IsSpace}      [[:^space:]]      [^\t\n\r\f \x{8140}]
 ! \p{IsGraph}      [[:graph:]]       [^\0- \x7F\x{8140}]
   \P{IsGraph}      [[:^graph:]]      [\0- \x7F\x{8140}]
 ! \p{IsPrint}      [[:print:]]       [^\0- \x0B\x0E-\x1F\x7F]
   \P{IsPrint}      [[:^print:]]      [\x00-\x08\x0B\x0E-\x1F\x7F]
   \p{IsCntrl}      [[:cntrl:]]       [\x00-\x1F]
 ! \P{IsCntrl}      [[:^cntrl:]]      [^\x00-\x1F]

   \p{IsAscii}      [[:ascii:]]       [\x00-\x7F]
 ! \P{IsAscii}      [[:^ascii:]]      [^\x00-\x7F]
   \p{IsHankaku}    [[:hankaku:]]     [\xA1-\xDF]
 ! \P{IsHankaku}    [[:^hankaku:]]    [^\xA1-\xDF]
 ! \p{IsZenkaku}    [[:zenkaku:]]     [\x{8140}-\x{FCFC}]
   \P{IsZenkaku}    [[:^zenkaku:]]    [^\x{8140}-\x{FCFC}]

   \p{IsX0201}      [[:x0201:]]       [\x00-\x7F\xA1-\xDF]
 ! \P{IsX0201}      [[:^x0201:]]      [^\x00-\x7F\xA1-\xDF]
   \p{IsX0208}      [[:x0208:]]
          [\x{8140}-�¬�¸-�¿�È-�Î�Ú-�è�ð-�÷�ü‚O-‚X‚`-‚y‚�-‚š‚Ÿ-‚ñƒ@-ƒ–ƒŸ-ƒ¶ƒ¿-ƒÖ„@-„`„p-„‘„Ÿ-„¾ˆŸ-˜r˜Ÿ-ê¤]

 ! \P{IsX0208}      [[:^x0208:]]
          [^\x{8140}-�¬�¸-�¿�È-�Î�Ú-�è�ð-�÷�ü‚O-‚X‚`-‚y‚�-‚š‚Ÿ-‚ñƒ@-ƒ–ƒŸ-ƒ¶ƒ¿-ƒÖ„@-„`„p-„‘„Ÿ-„¾ˆŸ-˜r˜Ÿ-ê¤]

   \p{InLatin}      [[:latin:]]       [A-Za-z]
 ! \P{InLatin}      [[:^latin:]]      [^A-Za-z]
   \p{InFullLatin}  [[:fulllatin:]]   [‚`-‚y‚�-‚š]
 ! \P{InFullLatin}  [[:^fulllatin:]]  [^‚`-‚y‚�-‚š]
   \p{InGreek}      [[:greek:]]       [ƒŸ-ƒ¶ƒ¿-ƒÖ]
 ! \P{InGreek}      [[:^greek:]]      [^ƒŸ-ƒ¶ƒ¿-ƒÖ]
   \p{InCyrillic}   [[:cyrillic:]]    [„@-„`„p-„‘]
 ! \P{InCyrillic}   [[:^cyrillic:]]   [^„@-„`„p-„‘]
   \p{InHalfKana}   [[:halfkana:]]    [¦-ß]
 ! \P{InHalfKana}   [[:^halfkana:]]   [^¦-ß]
   \p{InHiragana}   [[:hiragana:]]    [‚Ÿ-‚ñ�J�K�T�U]
 ! \P{InHiragana}   [[:^hiragana:]]   [^‚Ÿ-‚ñ�J�K�T�U]
   \p{InKatakana}   [[:katakana:]]    [ƒ@-ƒ–�[�R�S]
 ! \P{InKatakana}   [[:^katakana:]]   [^ƒ@-ƒ–�[�R�S]
   \p{InFullKana}   [[:fullkana:]]    [‚Ÿ-‚ñƒ@-ƒ–�J�K�[�T�U�R�S]
 ! \P{InFullKana}   [[:^fullkana:]]   [^‚Ÿ-‚ñƒ@-ƒ–�J�K�[�T�U�R�S]
   \p{InKana}       [[:kana:]]        [¦-ß‚Ÿ-‚ñƒ@-ƒ–�J�K�[�T�U�R�S]
 ! \P{InKana}       [[:^kana:]]       [^¦-ß‚Ÿ-‚ñƒ@-ƒ–�J�K�[�T�U�R�S]
   \p{InKanji1}     [[:kanji1:]]      [ˆŸ-˜r]
 ! \P{InKanji1}     [[:^kanji1:]]     [^ˆŸ-˜r]
   \p{InKanji2}     [[:kanji2:]]      [˜Ÿ-ê¤]
 ! \P{InKanji2}     [[:^kanji2:]]     [^˜Ÿ-ê¤]
   \p{InKanji}      [[:kanji:]]       [�V-�ZˆŸ-˜r˜Ÿ-ê¤]
 ! \P{InKanji}      [[:^kanji:]]      [^�V-�ZˆŸ-˜r˜Ÿ-ê¤]
   \p{InBoxDrawing} [[:boxdrawing:]]  [„Ÿ-„¾]
 ! \P{InBoxDrawing} [[:^boxdrawing:]] [^„Ÿ-„¾]

   * On \p{Prop} or \P{Prop} expressions, 'Is' or 'In' can be omitted
     like \p{Digit} or \P{Kanji}.
    (the omission of 'In' is an extension by this module)

   * Character classes marked with <!> contain undefined codepoints
     in JIS (JIS X 0201 or JIS X 0208).

Character class

Ranges in character class are supported.

The order of Shift_JIS characters is: 0x00 .. 0x7F, 0xA1 .. 0xDF, 0x8140 .. 0x9FFC, 0xE040 .. 0xFCFC.

So [\0-\x{fcfc}] matches any one Shift_JIS character.

In character classes, any character or byte sequence that does not match any one Shift_JIS character, e.g. re('[\xA0-\xFF]'), is croaked.

Character classes that match non-Shift_JIS substring are not supported (use \C or alternation).

Character Equivalences

Since the version 0.13, the POSIX character equivalent classes [=cc=] are supported. e.g. [[=‚ =]] is identical to [‚Ÿƒ@§‚ ƒA±]; [[=P=]] to [pP‚�‚o]; [[=4=]] to [4‚S]. They are used in a character class, like [[=cc=]], [[=p=][=e=][=r=][=l=]].

As cc in [=cc=], any character literal or meta chatacter (\xhh, \x{hhhh}) that belongs to the character equivalents can be used. e.g. [=‚ =], [=ƒA=], [=\x{82A0}=], [=\xB1=], etc. have identical meanings.

[[=‚©=]] matches '‚©', 'ƒJ', '¶', '‚ª', 'ƒK', '¶Þ', 'ƒ•' ('¶Þ' is a two-character string, but one collation element, HALFWIDTH FORM FOR KATAKANA LETTER GA.

[[===]] matches EQUALS SIGN or FULLWIDTH EQUALS SIGN; [[=[=]] matches LEFT SQUARE BRACKET or FULLWIDTH LEFT SQUARE BRACKET; [[=]=]] matches RIGHT SQUARE BRACKET or FULLWIDTH RIGHT SQUARE BRACKET; [[=\=]] matches YEN SIGN or FULLWIDTH YEN SIGN.

Code Embedded in a Regexp (Perl 5.005 or later)

Parsing (?{ ... }) or (??{ ... }) assertions is carried out without any special care of double-byte characters.

(?{ ... }) assertions are disallowed in match() or replace() functions by perl due to security concerns. Use them via re() function inside your scope.

  use ShiftJIS::Regexp qw(:all);

  use re 'eval';

  $::res = 0;
  $_ = 'ƒ|' x 8;

  my $regex = re(q/
       \j*?
       (?{ $cnt = 0 })
       (
         ƒ| (?{ local $cnt = $cnt + 1; })
       )*  
       ƒ|ƒ|ƒ|
       (?{ $::res = $cnt })
     /, 'x');

  /$regex/;
  print $::res; # 5

Embedded Modifiers

Since version 0.15, embedded modifiers are extended.

An embedded modifier, (?iIjsmxo), that appears at the beginning of the 'regexp' or that follows one of regexps ^, \A, or \G at the beginning of the 'regexp' is allowed to contain I, j, o modifiers.

  e.g. (?sm)pattern  ^(?i)pattern  \G(?j)pattern  \A(?ijo)pattern

And match('ƒG', '(?i)ƒg') returns false (Good result) even on Perl below 5.005, since it works like match('ƒG', 'ƒg', 'i').

CAVEATS

A legal Shift_JIS character in this module must match the following regexp:

   [\x00-\x7F\xA1-\xDF]|[\x81-\x9F\xE0-\xFC][\x40-\x7E\x80-\xFC]

Any string from external resource should be checked by issjis() function of ShiftJIS::String, excepting you know it is surely encoded in Shift_JIS.

Use of an illegal Shift_JIS string may lead to odd results.

Some Shift_JIS double-byte characters have one of [\x40-\x7E] as the trail byte.

   @ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~

The Perl lexer doesn't take any care to these characters, so they sometimes make trouble. e.g. the quoted literal "•\" causes fatal error, since its trail byte 0x5C backslashes the closing quote.

Such a problem doesn't arise when the string is gotten from any external resource. But writing the script containing Shift_JIS double-byte characters needs the greatest care.

The use of single-quoted heredoc, << '', or \xhh meta characters is recommended in order to define a Shift_JIS string literal.

The safe ASCII-graphic characters, [\x21-\x3F], are:

   !"#$%&'()*+,-./0123456789:;<=>?

They are preferred as the delimiter of quote-like operators.

BUGS

The \U, \L, \Q, \E, and interpolation are not considered. If necessary, use them in "" (or qq//) operators in the argument list.

The regexps of the word boundary, \b and \B, don't work correctly.

Never pass any regexp containing '(?i)' on perl below 5.005. Pass 'i' modifier as the second argument. (On Perl 5.005 or later, '(?i)' is allowed because '(?-i:RE)' prevents it from wrong matching)

e.g.

  match('ƒG', '(?i)ƒg') returns true on Perl below 5.005 (Wrong).
  match('ƒG', '(?i)ƒg') returns false on Perl 5.005 or later (Good).
  match('ƒG', 'ƒg', 'i') returns false, ok.
  # The trail byte of 'ƒG' is 'G' and that of 'ƒg' is 'g';

(see also "Embedded Modifiers")

The i, I and j modifiers are invalid to \p{}, \P{}, and POSIX [: :]. (e.g. \p{IsLower}, [:lower:], etc). So use re('\p{IsAlpha}') instead of re('\p{IsLower}', 'iI').

The look-behind assertion like (?<=[A-Z]) is not prevented from matching trail byte of the previous double byte character: e.g. match("ƒAƒCƒE", '(?<=[A-Z])(\p{InKana})') returns ('ƒC').

Use of not greedy regexp, which can match empty string, such as .?? and \d*?, as the PATTERN in jsplit(), may cause failure to the emulation of CORE::split.

AUTHOR

Tomoyuki SADAHIRO

  bqw10602@nifty.com
  http://homepage1.nifty.com/nomenclator/perl/
  This program is free software; you can redistribute it and/or 
  modify it under the same terms as Perl itself.

SEE ALSO

ShiftJIS::String
ShiftJIS::Collate

1 POD Error

The following errors were encountered while parsing the POD:

Around line 898:

Non-ASCII character seen before =encoding in 'match('‚ ‚¨‚P‚Q','. Assuming CP1252