NAME

Char::UTF2 - Source code filter to escape UTF-2 script

SYNOPSIS

use Char::UTF2;
use Char::UTF2 ver.sion;        --- require minimum version
use Char::UTF2 ver.sion.0;      --- expects version (match or die)
use Char::UTF2 qw(ord reverse); --- demand enhanced feature of ord and reverse
use Char::UTF2 ver.sion qw(ord reverse);
use Char::UTF2 ver.sion.0 qw(ord reverse);

# "no Char::UTF2;" not supported

or

$ perl Char/UTF2.pm UTF-2_script.pl > Escaped_script.pl.e

then

$ perl Escaped_script.pl.e

UTF-2_script.pl  --- script written in UTF-2
Escaped_script.pl.e --- escaped script

functions:
  Char::UTF2::ord(...);
  Char::UTF2::reverse(...);
  Char::UTF2::length(...);
  Char::UTF2::substr(...);
  Char::UTF2::index(...);
  Char::UTF2::rindex(...);
  CORE::chop(...);
  CORE::ord(...);
  CORE::reverse(...);
  CORE::index(...);
  CORE::rindex(...);

emulate Perl5.6 on perl5.00503
  use warnings;
  use warnings::register;

emulate Perl5.16
  use feature qw(fc);

dummy functions:
  utf8::upgrade(...);
  utf8::downgrade(...);
  utf8::encode(...);
  utf8::decode(...);
  utf8::is_utf8(...);
  utf8::valid(...);
  bytes::chr(...);
  bytes::index(...);
  bytes::length(...);
  bytes::ord(...);
  bytes::rindex(...);
  bytes::substr(...);

ABSTRACT

Let's start with a bit of history: jperl 4.019+1.3 introduced UTF-2 support. You could apply chop() and regexps even to complex CJK characters.

JPerl in CPAN Perl Ports (Binary Distributions)

said before,

As of Perl 5.8.0 it is suggested that instead of JPerl (which is
based on a quite old release of Perl) you should just use Perl 5.8.0,
since it can do all that JPerl did, and more.

But was it really so?

In this country, UTF-2 is widely used on mainframe I/O, the personal computer, and the cellular phone. This software treats UTF-2 directly, but doesn't treat Latin-1. Therefore, this software doesn't use UTF8 flag.

Shall we escape from the encode problem?

Yet Another Future Of

JPerl is very useful software. -- Oops, note, this "JPerl" means "Japanized Perl" or "Japanese Perl". Therefore, it is unrelated to JPerl of the following.

JPerl is an implementation of Perl written in Java.
http://www.javainc.com/projects/jperl/

jPerl - Perl on the JVM
http://www.dzone.com/links/175948.html

Jamie's PERL scripts for bioinformatics
http://code.google.com/p/jperl/

jperl (Jonathan Perl)
https://github.com/jperl

Now, the last version of JPerl is 5.005_04 and is not maintained now.

Japanization modifier WATANABE Hirofumi said,

"Because WATANABE am tired I give over maintaing JPerl."

at Slide #15: "The future of JPerl" of

ftp://ftp.oreilly.co.jp/pcjp98/watanabe/jperlconf.ppt

in The Perl Confernce Japan 1998.

When I heard it, I thought that someone excluding me would maintain JPerl. And I slept every night hanging a sock. Night and day, I kept having hope. After 10 years, I noticed that white beard exists in the sock :-)

This software is a source code filter to escape Perl script encoded by UTF-2 given from STDIN or command line parameter. The character code is never converted by escaping the script. Neither the value of the character nor the length of the character string change even if it escapes.

I learned the following things from the successful software.

Upper Compatibility like Perl4 to Perl5
Maximum Portability like jcode.pl
Handles Raw UTF-2, Doesn't use UTF8 flag like JPerl
Remains One Interpreter like Encode module
Code Set Independent like Ruby
There's more than one way to do it like Perl itself

Let's make yet another future by JPerl's future.

JRE: JPerl Runtime Environment

+---------------------------------------+
|        JPerl Application Script       | Your Script
+---------------------------------------+
|  Source Code Filter, Runtime Routine  | ex. Char/UTF2.pm, Char/Eutf2.pm
+---------------------------------------+
|          PVM 5.00503 or later         | ex. perl 5.00503
+---------------------------------------+

A Perl Virtual Machine (PVM) enables a set of computer software programs and data structures to use a virtual machine model for the execution of other computer programs and scripts. The model used by a PVM accepts a form of computer intermediate language commonly referred to as Perl byteorientedcode. This language conceptually represents the instruction set of a byte-oriented, capability architecture.

Basic Idea Of Source Code Filter

I discovered this mail again recently.

[Tokyo.pm] jus Benkyoukai

http://mail.pm.org/pipermail/tokyo-pm/1999-September/001854.html

save as: SJIS.pm

package SJIS;
use Filter::Util::Call;
sub multibyte_filter {
    my $status;
    if (($status = filter_read()) > 0 ) {
        s/([\x81-\x9f\xe0-\xef])([\x40-\x7e\x80-\xfc])/
            sprintf("\\x%02x\\x%02x",ord($1),ord($2))
        /eg;
    }
    $status;
}
sub import {
    filter_add(\&multibyte_filter);
}
1;

I am glad that I could confirm my idea is not so wrong.

Software Composition

Char/UTF2.pm               --- source code filter to escape UTF-2
Char/Eutf2.pm              --- run-time routines for Char/UTF2.pm
perl5.bat             --- find and run perl5    without %PATH% settings
perl55.bat            --- find and run perl5.5  without %PATH% settings
perl56.bat            --- find and run perl5.6  without %PATH% settings
perl58.bat            --- find and run perl5.8  without %PATH% settings
perl510.bat           --- find and run perl5.10 without %PATH% settings
perl512.bat           --- find and run perl5.12 without %PATH% settings
perl514.bat           --- find and run perl5.14 without %PATH% settings
perl516.bat           --- find and run perl5.16 without %PATH% settings
perl64.bat            --- find and run perl64   without %PATH% settings
perl64512.bat         --- find and run perl5.12 (x64) without %PATH% settings
perl64514.bat         --- find and run perl5.14 (x64) without %PATH% settings
perl64516.bat         --- find and run perl5.16 (x64) without %PATH% settings
aperl58.bat           --- find and run ActivePerl 5.8  without %PATH% settings
aperl510.bat          --- find and run ActivePerl 5.10 without %PATH% settings
aperl512.bat          --- find and run ActivePerl 5.12 without %PATH% settings
aperl514.bat          --- find and run ActivePerl 5.14 without %PATH% settings
aperl516.bat          --- find and run ActivePerl 5.16 without %PATH% settings
aperl64512.bat        --- find and run ActivePerl 5.12 (x64) without %PATH% settings
aperl64514.bat        --- find and run ActivePerl 5.14 (x64) without %PATH% settings
aperl64516.bat        --- find and run ActivePerl 5.16 (x64) without %PATH% settings
sperl58.bat           --- find and run Strawberry Perl 5.8  without %PATH% settings
sperl510.bat          --- find and run Strawberry Perl 5.10 without %PATH% settings
sperl512.bat          --- find and run Strawberry Perl 5.12 without %PATH% settings
sperl514.bat          --- find and run Strawberry Perl 5.14 without %PATH% settings
sperl516.bat          --- find and run Strawberry Perl 5.16 without %PATH% settings
sperl64512.bat        --- find and run Strawberry Perl 5.12 (x64) without %PATH% settings
sperl64514.bat        --- find and run Strawberry Perl 5.14 (x64) without %PATH% settings
sperl64516.bat        --- find and run Strawberry Perl 5.16 (x64) without %PATH% settings
strict.pm_            --- dummy strict.pm
warnings.pm_          --- poor warnings.pm
warnings/register.pm_ --- poor warnings/register.pm
feature.pm_           --- dummy feature.pm

Upper Compatibility By Escaping

This software adds the function by 'Escaping' it always, and nothing of the past is broken. Therefore, 'Possible job' never becomes 'Impossible job'. This approach is effective in the field where the retreat is never permitted. Modern Perl/perl can not always solve the problem. Often, it means an incompatible upgrade part to traditional Perl should be rewound.

Escaping Your Script (You do)

You need write 'use Char::UTF2;' in your script.

---------------------------------
Before      You do
---------------------------------
(nothing)   use Char::UTF2;
---------------------------------

Multiple-Octet Anchoring Of Regular Expression (Char/UTF2.pm provides)

Char/UTF2.pm applies multiple-octet anchoring at beginning of regular expression.

--------------------------------------------------------------------------------
Before                  After
--------------------------------------------------------------------------------
m/regexp/               m/${Char::Eutf2::anchor}(?:regexp).../
--------------------------------------------------------------------------------

Multiple-Octet Character Regular Expression (Char/UTF2.pm provides)

Char/UTF2.pm clusters multiple-octet character with quantifier, makes cluster from multiple-octet custom character classes. And makes multiple-octet version metasymbol from classic Perl character class shortcuts and POSIX-style character classes.

--------------------------------------------------------------------------------
Before                  After
--------------------------------------------------------------------------------
m/...MULTIOCT+.../      m/...(?:MULTIOCT)+.../
m/...[AN-EM].../        m/...(?:A[N-Z]|[B-D][A-Z]|E[A-M]).../
m/...\D.../             m/...${Char::Eutf2::eD}.../
m/...[[:^digit:]].../   m/...${Char::Eutf2::not_digit}.../
--------------------------------------------------------------------------------

Calling 'Char::Eutf2::ignorecase()' (Char/UTF2.pm provides)

Char/UTF2.pm applies calling 'Char::Eutf2::ignorecase()' instead of /i modifier.

--------------------------------------------------------------------------------
Before                  After
--------------------------------------------------------------------------------
m/...$var.../i          m/...@{[Char::Eutf2::ignorecase($var)]}.../
--------------------------------------------------------------------------------

Character-Oriented Regular Expression

Regular expression works as character-oriented that has no /b modifier.

 --------------------------------------------------------------------------------
 Before                  After
 --------------------------------------------------------------------------------
 /regexp/                /ditto$Char::Eutf2::matched/
 m/regexp/               m/ditto$Char::Eutf2::matched/
 ?regexp?                m?ditto$Char::Eutf2::matched?
 m?regexp?               m?ditto$Char::Eutf2::matched?

 $_ =~                   ($_ =~ m/ditto$Char::Eutf2::matched/) ?
 s/regexp/replacement/   eval{ Char::Eutf2::s_matched(); local $^W=0; my $__r=qq/replacement/; $_="${1}$__r$'"; 1 } :
                         undef

 $_ !~                   ($_ !~ m/ditto$Char::Eutf2::matched/) ?
 s/regexp/replacement/   1 :
                         eval{ Char::Eutf2::s_matched(); local $^W=0; my $__r=qq/replacement/; $_="${1}$__r$'"; undef }

 split(/regexp/)         Char::Eutf2::split(qr/regexp/)
 split(m/regexp/)        Char::Eutf2::split(qr/regexp/)
 split(qr/regexp/)       Char::Eutf2::split(qr/regexp/)
 qr/regexp/              qr/ditto$Char::Eutf2::matched/
 --------------------------------------------------------------------------------

Byte-Oriented Regular Expression

Regular expression works as byte-oriented that has /b modifier.

 --------------------------------------------------------------------------------
 Before                  After
 --------------------------------------------------------------------------------
 /regexp/b               /(?:regexp)$Char::Eutf2::matched/
 m/regexp/b              m/(?:regexp)$Char::Eutf2::matched/
 ?regexp?b               m?regexp$Char::Eutf2::matched?
 m?regexp?b              m?regexp$Char::Eutf2::matched?

 $_ =~                   ($_ =~ m/(\G[\x00-\xFF]*?)(?:regexp)$Char::Eutf2::matched/) ?
 s/regexp/replacement/b  eval{ Char::Eutf2::s_matched(); local $^W=0; my $__r=qq/replacement/; $_="${1}$__r$'"; 1 } :
                         undef

 $_ !~                   ($_ !~ m/(\G[\x00-\xFF]*?)(?:regexp)$Char::Eutf2::matched/) ?
 s/regexp/replacement/b  1 :
                         eval{ Char::Eutf2::s_matched(); local $^W=0; my $__r=qq/replacement/; $_="${1}$__r$'"; undef }

 split(/regexp/b)        split(qr/regexp/)
 split(m/regexp/b)       split(qr/regexp/)
 split(qr/regexp/b)      split(qr/regexp/)
 qr/regexp/b             qr/(?:regexp)$Char::Eutf2::matched/
 --------------------------------------------------------------------------------

Escaping Character Classes (Char/Eutf2.pm provides)

The character classes are redefined as follows to backward compatibility.

---------------------------------------------------------------
Before        After
---------------------------------------------------------------
 .            ${Char::Eutf2::dot}
              ${Char::Eutf2::dot_s}    (/s modifier)
\d            [0-9]
\s            [\x09\x0A\x0C\x0D\x20]
\w            [0-9A-Z_a-z]
\D            ${Char::Eutf2::eD}
\S            ${Char::Eutf2::eS}
\W            ${Char::Eutf2::eW}
\h            [\x09\x20]
\v            [\x0A\x0B\x0C\x0D]
\H            ${Char::Eutf2::eH}
\V            ${Char::Eutf2::eV}
\C            [\x00-\xFF]
\X            X (so, just 'X')
\R            ${Char::Eutf2::eR}
\N            ${Char::Eutf2::eN}
---------------------------------------------------------------

Also POSIX-style character classes.

---------------------------------------------------------------
Before        After
---------------------------------------------------------------
[:alnum:]     [\x30-\x39\x41-\x5A\x61-\x7A]
[:alpha:]     [\x41-\x5A\x61-\x7A]
[:ascii:]     [\x00-\x7F]
[:blank:]     [\x09\x20]
[:cntrl:]     [\x00-\x1F\x7F]
[:digit:]     [\x30-\x39]
[:graph:]     [\x21-\x7F]
[:lower:]     [\x61-\x7A]
              [\x41-\x5A\x61-\x7A]     (/i modifier)
[:print:]     [\x20-\x7F]
[:punct:]     [\x21-\x2F\x3A-\x3F\x40\x5B-\x5F\x60\x7B-\x7E]
[:space:]     [\x09\x0A\x0B\x0C\x0D\x20]
[:upper:]     [\x41-\x5A]
              [\x41-\x5A\x61-\x7A]     (/i modifier)
[:word:]      [\x30-\x39\x41-\x5A\x5F\x61-\x7A]
[:xdigit:]    [\x30-\x39\x41-\x46\x61-\x66]
[:^alnum:]    ${Char::Eutf2::not_alnum}
[:^alpha:]    ${Char::Eutf2::not_alpha}
[:^ascii:]    ${Char::Eutf2::not_ascii}
[:^blank:]    ${Char::Eutf2::not_blank}
[:^cntrl:]    ${Char::Eutf2::not_cntrl}
[:^digit:]    ${Char::Eutf2::not_digit}
[:^graph:]    ${Char::Eutf2::not_graph}
[:^lower:]    ${Char::Eutf2::not_lower}
              ${Char::Eutf2::not_lower_i}    (/i modifier)
[:^print:]    ${Char::Eutf2::not_print}
[:^punct:]    ${Char::Eutf2::not_punct}
[:^space:]    ${Char::Eutf2::not_space}
[:^upper:]    ${Char::Eutf2::not_upper}
              ${Char::Eutf2::not_upper_i}    (/i modifier)
[:^word:]     ${Char::Eutf2::not_word}
[:^xdigit:]   ${Char::Eutf2::not_xdigit}
---------------------------------------------------------------

Also \b and \B are redefined as follows to backward compatibility.

---------------------------------------------------------------
Before      After
---------------------------------------------------------------
\b          ${Char::Eutf2::eb}
\B          ${Char::Eutf2::eB}
---------------------------------------------------------------

Definitions in Char/Eutf2.pm.

---------------------------------------------------------------------------------------------------------------------------------------------------------
After                    Definition
---------------------------------------------------------------------------------------------------------------------------------------------------------
${Char::Eutf2::anchor}         qr{\G(?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x80-\xFF])*?}
${Char::Eutf2::dot}            qr{(?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x80-\xFF\x0A])}
${Char::Eutf2::dot_s}          qr{(?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x80-\xFF])}
${Char::Eutf2::eD}             qr{(?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x80-\xFF0-9])}
${Char::Eutf2::eS}             qr{(?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x80-\xFF\x09\x0A\x0C\x0D\x20])}
${Char::Eutf2::eW}             qr{(?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x80-\xFF0-9A-Z_a-z])}
${Char::Eutf2::eH}             qr{(?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x80-\xFF\x09\x20])}
${Char::Eutf2::eV}             qr{(?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x80-\xFF\x0A\x0B\x0C\x0D])}
${Char::Eutf2::eR}             qr{(?:\x0D\x0A|[\x0A\x0D])}
${Char::Eutf2::eN}             qr{(?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x80-\xFF\x0A])}
${Char::Eutf2::not_alnum}      qr{(?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x80-\xFF\x30-\x39\x41-\x5A\x61-\x7A])}
${Char::Eutf2::not_alpha}      qr{(?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x80-\xFF\x41-\x5A\x61-\x7A])}
${Char::Eutf2::not_ascii}      qr{(?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x80-\xFF\x00-\x7F])}
${Char::Eutf2::not_blank}      qr{(?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x80-\xFF\x09\x20])}
${Char::Eutf2::not_cntrl}      qr{(?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x80-\xFF\x00-\x1F\x7F])}
${Char::Eutf2::not_digit}      qr{(?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x80-\xFF\x30-\x39])}
${Char::Eutf2::not_graph}      qr{(?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x80-\xFF\x21-\x7F])}
${Char::Eutf2::not_lower}      qr{(?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x80-\xFF\x61-\x7A])}
${Char::Eutf2::not_lower_i}    qr{(?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x80-\xFF])}
${Char::Eutf2::not_print}      qr{(?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x80-\xFF\x20-\x7F])}
${Char::Eutf2::not_punct}      qr{(?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x80-\xFF\x21-\x2F\x3A-\x3F\x40\x5B-\x5F\x60\x7B-\x7E])}
${Char::Eutf2::not_space}      qr{(?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x80-\xFF\x09\x0A\x0B\x0C\x0D\x20])}
${Char::Eutf2::not_upper}      qr{(?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x80-\xFF\x41-\x5A])}
${Char::Eutf2::not_upper_i}    qr{(?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x80-\xFF])}
${Char::Eutf2::not_word}       qr{(?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x80-\xFF\x30-\x39\x41-\x5A\x5F\x61-\x7A])}
${Char::Eutf2::not_xdigit}     qr{(?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x80-\xFF\x30-\x39\x41-\x46\x61-\x66])}
${Char::Eutf2::eb}             qr{(?:\A(?=[0-9A-Z_a-z])|(?<=[\x00-\x2F\x40\x5B-\x5E\x60\x7B-\xFF])(?=[0-9A-Z_a-z])|(?<=[0-9A-Z_a-z])(?=[\x00-\x2F\x40\x5B-\x5E\x60\x7B-\xFF]|\z))}
${Char::Eutf2::eB}             qr{(?:(?<=[0-9A-Z_a-z])(?=[0-9A-Z_a-z])|(?<=[\x00-\x2F\x40\x5B-\x5E\x60\x7B-\xFF])(?=[\x00-\x2F\x40\x5B-\x5E\x60\x7B-\xFF]))}
---------------------------------------------------------------------------------------------------------------------------------------------------------

Un-Escaping \ Of \N, \p, \P and \X (Char/UTF2.pm provides)

Char/UTF2.pm removes '\' at head of alphanumeric regexp metasymbols \N, \p, \P and \X. By this method, you can avoid the trap of the abstraction.

See also, Deprecate literal unescaped "{" in regexes. http://perl5.git.perl.org/perl.git/commit/2a53d3314d380af5ab5283758219417c6dfa36e9

------------------------------------
Before           After
------------------------------------
\N{CHARNAME}     N\{CHARNAME}
\p{L}            p\{L}
\p{^L}           p\{^L}
\p{\^L}          p\{\^L}
\pL              pL
\P{L}            P\{L}
\P{^L}           P\{^L}
\P{\^L}          P\{\^L}
\PL              PL
\X               X
------------------------------------

Escaping Built-in Functions (Char/UTF2.pm and Char/Eutf2.pm provide)

Insert 'Char::Eutf2::' at head of function name. Char/Eutf2.pm provides your script Char::Eutf2::* functions.

-------------------------------------------
Before      After            Works as
-------------------------------------------
length      length           Byte
substr      substr           Byte
pos         pos              Byte
split       Char::Eutf2::split     Character
tr///       Char::Eutf2::tr        Character
tr///b      tr///            Byte
tr///B      tr///            Byte
y///        Char::Eutf2::tr        Character
y///b       tr///            Byte
y///B       tr///            Byte
chop        Char::Eutf2::chop      Character
index       Char::Eutf2::index     Character
rindex      Char::Eutf2::rindex    Character
lc          Char::Eutf2::lc        Character
lcfirst     Char::Eutf2::lcfirst   Character
uc          Char::Eutf2::uc        Character
ucfirst     Char::Eutf2::ucfirst   Character
fc          Char::Eutf2::fc        Character
chr         Char::Eutf2::chr       Character
glob        Char::Eutf2::glob      Character
-------------------------------------------

------------------------------------------------------------------------------------------------------------------------
Before                   After
------------------------------------------------------------------------------------------------------------------------
use Perl::Module;        BEGIN { require 'Perl/Module.pm'; Perl::Module->import() if Perl::Module->can('import'); }
use Perl::Module @list;  BEGIN { require 'Perl/Module.pm'; Perl::Module->import(@list) if Perl::Module->can('import'); }
use Perl::Module ();     BEGIN { require 'Perl/Module.pm'; }
no Perl::Module;         BEGIN { require 'Perl/Module.pm'; Perl::Module->unimport() if Perl::Module->can('unimport'); }
no Perl::Module @list;   BEGIN { require 'Perl/Module.pm'; Perl::Module->unimport(@list) if Perl::Module->can('unimport'); }
no Perl::Module ();      BEGIN { require 'Perl/Module.pm'; }
------------------------------------------------------------------------------------------------------------------------

Escaping Function Name (You do)

You need write 'Char::UTF2::' at head of function name when you want character- oriented function. See 'Character-Oriented Functions'.

--------------------------------------------------------
Function   Character-Oriented   Description
--------------------------------------------------------
ord        Char::UTF2::ord
reverse    Char::UTF2::reverse
length     Char::UTF2::length
substr     Char::UTF2::substr
index      Char::UTF2::index          See 'About Indexes'
rindex     Char::UTF2::rindex         See 'About Rindexes'
--------------------------------------------------------

About Indexes
-------------------------------------------------------------------------
Function       Works as    Returns as   Description
-------------------------------------------------------------------------
index          Character   Byte         JPerl semantics (most useful)
(same as Char::Eutf2::index)
Char::UTF2::index    Character   Character    Character-oriented semantics
CORE::index    Byte        Byte         Byte-oriented semantics
(nothing)      Byte        Character    (most useless)
-------------------------------------------------------------------------

About Rindexes
-------------------------------------------------------------------------
Function       Works as    Returns as   Description
-------------------------------------------------------------------------
rindex         Character   Byte         JPerl semantics (most useful)
(same as Char::Eutf2::rindex)
Char::UTF2::rindex   Character   Character    Character-oriented semantics
CORE::rindex   Byte        Byte         Byte-oriented semantics
(nothing)      Byte        Character    (most useless)
-------------------------------------------------------------------------

Character-Oriented Functions

Order Of Character

$ord = Char::UTF2::ord($string);

This function returns the numeric value (ASCII or UTF-2 character) of the
first character of $string, not Unicode. If $string is omitted, it uses $_.
The return value is always unsigned.

If you import ord "use Char::UTF2 qw(ord);", ord of your script will be rewritten in
Char::UTF2::ord. Char::UTF2::ord is not compatible with ord of JPerl.

Reverse List Or String

@reverse = Char::UTF2::reverse(@list);
$reverse = Char::UTF2::reverse(@list);

In list context, this function returns a list value consisting of the elements of
@list in the opposite order.

In scalar context, the function concatenates all the elements of @list and then
returns the reverse of that resulting string, character by character.

If you import reverse "use Char::UTF2 qw(reverse);", reverse of your script will be
rewritten in Char::UTF2::reverse. Char::UTF2::reverse is not compatible with reverse of
JPerl.

Length By UTF-2 Character

$length = Char::UTF2::length($string);
$length = Char::UTF2::length();

This function returns the length in characters (programmer-visible characters) of
the scalar value $string. If $string is omitted, it returns the Char::UTF2::length of
$_.

Do not try to use Char::UTF2::length to find the size of an array or hash. Use scalar
@array for the size of an array, and scalar keys %hash for the number of key/value
pairs in a hash. (The scalar is typically omitted when redundant.)

To find the length of a string in bytes rather than characters, say simply:

$bytes = length($string);

Substr By UTF-2 Character

$substr = Char::UTF2::substr($string,$offset,$length,$replacement);
$substr = Char::UTF2::substr($string,$offset,$length);
$substr = Char::UTF2::substr($string,$offset);

This function extracts a substring out of the string given by $string and returns
it. The substring is extracted starting at $offset characters from the front of
the string.
If $offset is negative, the substring starts that far from the end of the string
instead. If $length is omitted, everything to the end of the string is returned.
If $length is negative, the length is calculated to leave that many characters off
the end of the string. Otherwise, $length indicates the length of the substring to
extract, which is sort of what you'd expect.

For bytes, use the substr from built-in Perl functions.

An alternative to using Char::UTF2::substr as an lvalue is to specify the $replacement
string as the fourth argument. This lets you replace parts of the $string and return
what was there before in one operation, just as you can with splice. The next
example also replaces the last character of $var with "Curly" and puts that replaced
character into $oldstr: 

$oldstr = Char::UTF2::substr($var, -1, 1, "Curly");

To prepend the string "Larry" to the current value of $var, use:

Char::UTF2::substr($var, 0, 0, "Larry");

To instead replace the first character of $var with "Moe", use:

Char::UTF2::substr($var, 0, 1, "Moe");

And, finally, to replace the last character of $var with "Curly", use:

Char::UTF2::substr($var, -1, 1, "Curly");

Index By UTF-2 Character

$index = Char::UTF2::index($string,$substring,$offset);
$index = Char::UTF2::index($string,$substring);

This function searches for one string within another. It returns the character
position of the first occurrence of $substring in $string. The $offset, if
specified, says how many characters from the start to skip before beginning to
look. Positions are based at 0. If the substring is not found, the function
returns one less than the base, ordinarily -1. To work your way through a string,
you might say:

$pos = -1;
while (($pos = Char::UTF2::index($string, $lookfor, $pos)) > -1) {
    print "Found at $pos\n";
    $pos++;
}

Rindex By UTF-2 Character

$rindex = Char::UTF2::rindex($string,$substring,$offset);
$rindex = Char::UTF2::rindex($string,$substring);

This function works just like Char::UTF2::index except that it returns the character
position of the last occurrence of $substring in $string (a reverse Char::UTF2::index).
The function returns -1 if $substring is not found. $offset, if specified, is
the rightmost character position that may be returned. To work your way through
a string backward, say:

$pos = Char::UTF2::length($string);
while (($pos = Char::UTF2::rindex($string, $lookfor, $pos)) >= 0) {
    print "Found at $pos\n";
    $pos--;
}

Byte-Oriented Functions

Chop Byte String

$byte = CORE::chop($string);
$byte = CORE::chop(@list);
$byte = CORE::chop;

This function chops off the last byte of a string variable and returns the
byte chopped. The CORE::chop operator is used primarily to remove the newline
from the end of an input record, and is more efficient than using a
substitution (s/\n$//). If that's all you're doing, then it would be safer to
use chomp, since CORE::chop always shortens the string no matter what's there,
and chomp is more selective.

You cannot CORE::chop a literal, only a variable.

If you CORE::chop a @list of variables, each string in the list is chopped:

@lines = `cat myfile`;
CORE::chop @lines;

You can CORE::chop anything that is an lvalue, including an assignment:

CORE::chop($cwd = `pwd`);
CORE::chop($answer = <STDIN>);

This is different from:

$answer = CORE::chop($temp = <STDIN>); # WRONG

which puts a newline into $answer because CORE::chop returns the byte chopped,
not the remaining string (which is in $tmp). One way to get the result
intended here is with substr:

$answer = substr <STDIN>, 0, -1;

But this is more commonly written as:

CORE::chop($answer = <STDIN>);

In the most general case, CORE::chop can be expressed in terms of substr:

$last_byte = CORE::chop($var);
$last_byte = substr($var, -1, 1, ""); # same thing

Once you understand this equivalence, you can use it to do bigger chops. To
CORE::chop more than one byte, use substr as an lvalue, assigning a null
string. The following removes the last five bytes of $caravan:

substr($caravan, -5) = "";

The negative subscript causes substr to count from the end of the string
instead of the beginning. If you wanted to save the bytes so removed, you
could use the four-argument form of substr, creating something of a quintuple
CORE::chop:

$tail = substr($caravan, -5, 5, "");

If no argument is given, the function chops the $_ variable.

Order Of Byte

$ord = CORE::ord($expr);

This function returns the numeric value of the first byte of $expr, regardless
of "use Char::UTF2 qw(ord);" exists or not. If $expr is omitted, it uses $_.
The return value is always unsigned.

If you want a signed value, use unpack('c',$expr). If you want all the bytes of
the string converted to a list of numbers, use unpack('C*',$expr) instead.

Reverse List Or Byte String

@reverse = CORE::reverse(@list);
$reverse = CORE::reverse(@list);

In list context, this function returns a list value consisting of the elements
of @list in the opposite order.

In scalar context, the function concatenates all the elements of @list and then
returns the reverse of that resulting string, byte by byte, regardless of
"use Char::UTF2 qw(reverse);" exists or not.

Index By Byte String

$index = CORE::index($string,$substring,$offset);
$index = CORE::index($string,$substring);

This function searches for one byte string within another. It returns the position
of the first occurrence of $substring in $string. The $offset, if specified, says
how many bytes from the start to skip before beginning to look. Positions are based
at 0. If the substring is not found, the function returns one less than the base,
ordinarily -1. To work your way through a string, you might say:

$pos = -1;
while (($pos = CORE::index($string, $lookfor, $pos)) > -1) {
    print "Found at $pos\n";
    $pos++;
}

Rindex By Byte String

$rindex = CORE::rindex($string,$substring,$offset);
$rindex = CORE::rindex($string,$substring);

This function works just like CORE::index except that it returns the position of
the last occurrence of $substring in $string (a reverse CORE::index). The function
returns -1 if not $substring is found. $offset, if specified, is the rightmost
position that may be returned. To work your way through a string backward, say:

$pos = CORE::length($string);
while (($pos = CORE::rindex($string, $lookfor, $pos)) >= 0) {
    print "Found at $pos\n";
    $pos--;
}

Un-Escaping bytes::* Functions (Char/UTF2.pm provides)

Char/UTF2.pm removes 'bytes::' at head of function name.

---------------------------------------
Before           After     Works as
---------------------------------------
bytes::chr       chr       Byte
bytes::index     index     Byte
bytes::length    length    Byte
bytes::ord       ord       Byte
bytes::rindex    rindex    Byte
bytes::substr    substr    Byte
---------------------------------------

Escaping Built-in Standard Module (Char/Eutf2.pm provides)

Char/Eutf2.pm does "BEGIN { unshift @INC, '/Perl/site/lib/Char::UTF2' }" at head. Store the standard module modified for Char::UTF2 software in this directory to override built-in standard modules.

Escaping Standard Module Content (You do)

You need copy built-in standard module to /Perl/site/lib/Char::UTF2 and change 'use utf8;' to 'use Char::UTF2;' in its. You need help yourself for now.

Back to and see 'Escaping Your Script'. Enjoy hacking!!

Ignore Pragmas And Modules

-----------------------------------------------------------
Before                    After
-----------------------------------------------------------
use strict;               use strict; no strict qw(refs);
use 5.12.0;               use 5.12.0; no strict qw(refs);
require utf8;             # require utf8;
require bytes;            # require bytes;
require charnames;        # require charnames;
require I18N::Japanese;   # require I18N::Japanese;
require I18N::Collate;    # require I18N::Collate;
require I18N::JExt;       # require I18N::JExt;
require File::DosGlob;    # require File::DosGlob;
require Wild;             # require Wild;
require Wildcard;         # require Wildcard;
require Japanese;         # require Japanese;
use utf8;                 # use utf8;
use bytes;                # use bytes;
use charnames;            # use charnames;
use I18N::Japanese;       # use I18N::Japanese;
use I18N::Collate;        # use I18N::Collate;
use I18N::JExt;           # use I18N::JExt;
use File::DosGlob;        # use File::DosGlob;
use Wild;                 # use Wild;
use Wildcard;             # use Wildcard;
use Japanese;             # use Japanese;
no utf8;                  # no utf8;
no bytes;                 # no bytes;
no charnames;             # no charnames;
no I18N::Japanese;        # no I18N::Japanese;
no I18N::Collate;         # no I18N::Collate;
no I18N::JExt;            # no I18N::JExt;
no File::DosGlob;         # no File::DosGlob;
no Wild;                  # no Wild;
no Wildcard;              # no Wildcard;
no Japanese;              # no Japanese;
-----------------------------------------------------------

Comment out pragma to ignore utf8 environment, and Char/Eutf2.pm provides these
functions.

Dummy utf8::upgrade

$num_octets = utf8::upgrade($string);

Returns the number of octets necessary to represent the string.

Dummy utf8::downgrade

$success = utf8::downgrade($string[, FAIL_OK]);

Returns true always.

Dummy utf8::encode

utf8::encode($string);

Returns nothing.

Dummy utf8::decode

$success = utf8::decode($string);

Returns true always.

Dummy utf8::is_utf8

$flag = utf8::is_utf8(STRING);

Returns false always.

Dummy utf8::valid

$flag = utf8::valid(STRING);

Returns true always.

Dummy bytes::chr
```
This function is same as chr.
```
Dummy bytes::index
```
This function is same as index.
```
Dummy bytes::length
```
This function is same as length.
```
Dummy bytes::ord
```
This function is same as ord.
```
Dummy bytes::rindex
```
This function is same as rindex.
```
Dummy bytes::substr
```
This function is same as substr.
```

Environment Variable

This software uses the flock function for exclusive control. The execution of the
program is blocked until it becomes possible to read or write the file.
You can have it not block in the flock function by defining environment variable
SJIS_NONBLOCK.

Example:

  SET SJIS_NONBLOCK=1

(The value '1' doesn't have the meaning)

Perl5.6 Emulation On perl5.005

Using warnings pragma on perl5.00503 by rename files.

warnings.pm_ --> warnings.pm
warnings/register.pm_ --> warnings/register.pm

Perl5.16 Emulation On perl5.005

Using feature pragma on perl5.00503 by rename files.

feature.pm_ --> feature.pm

BUGS AND LIMITATIONS

I have tested and verified this software using the best of my ability. However, a software containing much regular expression is bound to contain some bugs. Thus, if you happen to find a bug that's in Char::UTF2 software and not your own program, you can try to reduce it to a minimal test case and then report it to the following author's address. If you have an idea that could make this a more useful tool, please let everyone share it.

format

Function "format" can't handle multiple-octet code same as original Perl.
Char::UTF2::substr As Lvalue

Char::UTF2::substr differs from CORE::substr, and cannot be used as a lvalue. To change part of a string, you can use the optional fourth argument which is the replacement string.

Char::UTF2::substr($string, 13, 4, "JPerl");

Special Variables $` And $& Need /( Capture All )/

Because $` and $& use $1.

-------------------------------------------------------------------------------------------
Before          After                Works as
-------------------------------------------------------------------------------------------
$`              Char::Eutf2::PREMATCH()    CORE::substr($&,0,CORE::length($&)-CORE::length($1))
$PREMATCH       Char::Eutf2::PREMATCH()    CORE::substr($&,0,CORE::length($&)-CORE::length($1))
${^PREMATCH}    Char::Eutf2::PREMATCH()    CORE::substr($&,0,CORE::length($&)-CORE::length($1))
$&              Char::Eutf2::MATCH()       $1
$MATCH          Char::Eutf2::MATCH()       $1
${^MATCH}       Char::Eutf2::MATCH()       $1
$'              Char::Eutf2::POSTMATCH()   $'
$POSTMATCH      Char::Eutf2::POSTMATCH()   $'
${^POSTMATCH}   Char::Eutf2::POSTMATCH()   $'
-------------------------------------------------------------------------------------------

Limitation Of Regular Expression

This software has limitation from \G in multibyte anchoring. On perl5.006, perl5.008, perl5.010, perl5.012, perl5.014 and perl5.016 it doesn't match in the place in which it should match at over 32,767 octets. Moreover, at that time, neither the error nor warning are displayed.

see also, Bug #89792 \G can't treat over 32,767 octets http://bugs.activestate.com/show_bug.cgi?id=89792
Empty Variable In Regular Expression

Unlike literal null string, an interpolated variable evaluated to the empty string can't use the most recent pattern from a previous successful regular expression.
Limitation Of ?? and m??

Multibyte character needs ( ) which is before {n,m} {n,} {n} * and + in ?? or m??. As a result, you need to rewrite a script about $1,$2,$3,... You cannot use (?: ) ? {n,m}? {n,}? and {n}? in ?? and m??, because delimiter of m?? is '?'.
Modifier /a /d /l And /u Of Regular Expression

The concept of this software is not to use two or more encoding methods at the same time. Therefore, modifier /a /d /l and /u are not supported. \d means [0-9] always.

AUTHOR

INABA Hitoshi <ina@cpan.org>

This project was originated by INABA Hitoshi.

LICENSE AND COPYRIGHT

This software is free software; you can redistribute it and/or modify it under the same terms as Perl itself. See perlartistic.

This software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

My Goal

P.401 See chapter 15: Unicode of ISBN 0-596-00027-8 Programming Perl Third Edition.

Before the introduction of Unicode support in perl, The eq operator just compared the byte-strings represented by two scalars. Beginning with perl 5.8, eq compares two byte-strings with simultaneous consideration of the UTF8 flag.

 Information processing model beginning with perl 5.8

   +----------------------+---------------------+
   |     Text strings     |                     |
   +----------+-----------|    Binary strings   |
   |   UTF8   |  Latin-1  |                     |
   +----------+-----------+---------------------+
   | UTF8     |            Not UTF8             |
   | Flagged  |            Flagged              |
   +--------------------------------------------+
   http://perl-users.jp/articles/advent-calendar/2010/casual/4

   You should memorize this figure.

   (Why is only Latin-1 special?)

This change consequentially made a big gap between a past script and new script. Both scripts cannot re-use the code mutually any longer. Because a new method puts a strain in the programmer, it will still take time to replace all the in existence scripts.

The biggest problem of new method is that the UTF8 flag can't synchronize to real encode of string. Thus you must debug about UTF8 flag, before your script. How to solve it by returning to a past method, let's drag out page 402 of the old dusty Programming Perl, 3rd ed. again.

Information processing model with this software

  +--------------------------------------------+
  |       Text strings as Binary strings       |
  |       Binary strings as Text strings       |
  +--------------------------------------------+
  |              Not UTF8 Flagged              |
  +--------------------------------------------+

Ideally, I'd like to achieve these five Goals:

Goal #1:

Old byte-oriented programs should not spontaneously break on the old byte-oriented data they used to work on.

It has already been achieved by UTF-2 designed for combining with old byte-oriented ASCII.

Goal #2:

Old byte-oriented programs should magically start working on the new character-oriented data when appropriate.

Still now, 1 octet is counted with 1 by embedded functions length, substr, index, rindex and pos that handle length and position of string. In this part, there is no change. The length of 1 character of 2 octet code is 2.

On the other hand, the regular expression in the script is added the multibyte anchoring processing with this software, instead of you.

figure of Goal #1 and Goal #2.

                         GOAL#1  GOAL#2
                  (a)     (b)     (c)     (d)     (e)
+--------------+-------+-------+-------+-------+-------+
| data         |  Old  |  Old  |  New  |  Old  |  New  |
+--------------+-------+-------+-------+-------+-------+
| script       |  Old  |      Old      |      New      |
+--------------+-------+---------------+---------------+
| interpreter  |  Old  |              New              |
+--------------+-------+-------------------------------+
Old --- Old byte-oriented
New --- New character-oriented

There is a combination from (a) to (e) in data, script and interpreter of old and new. Let's add the Encode module and this software did not exist at time of be written this document and JPerl did exist.

                  (a)     (b)     (c)     (d)     (e)
                                JPerl           Encode,Char::UTF2
+--------------+-------+-------+-------+-------+-------+
| data         |  Old  |  Old  |  New  |  Old  |  New  |
+--------------+-------+-------+-------+-------+-------+
| script       |  Old  |      Old      |      New      |
+--------------+-------+---------------+---------------+
| interpreter  |  Old  |              New              |
+--------------+-------+-------------------------------+
Old --- Old byte-oriented
New --- New character-oriented

The reason why JPerl is very excellent is that it is at the position of (c). That is, it is not necessary to do a special description to the script to process new character-oriented string.

Contrasting is Encode module and describing "use Char::UTF2;" on this software, in this case, a new description is necessary.

Goal #3:

Programs should run just as fast in the new character-oriented mode as in the old byte-oriented mode.

It is impossible. Because the following time is necessary.

(1) Time of escape script for old byte-oriented perl.

(2) Time of processing regular expression by escaped script while multibyte anchoring.
Goal #4:

Perl should remain one language, rather than forking into a byte-oriented Perl and a character-oriented Perl.

JPerl remains one Perl language by forking to two interpreters. However, the Perl core team did not desire fork of the interpreter. As a result, Perl language forked contrary to goal #4.

A character-oriented perl is not necessary to make it specially, because a byte-oriented perl can already treat the binary data. This software is only an application program of byte-oriented Perl, a filter program.

And you will get support from the Perl community, when you solve the problem by the Perl script.
Goal #5:

JPerl users will be able to maintain JPerl by Perl.

May the JPerl be with you, always.

Back when Programming Perl, 3rd ed. was written, UTF8 flag was not born and Perl is designed to make the easy jobs easy. This software provide programming environment like at that time.

Words Of Learning Perl

  Some computer scientists (the reductionists, in particular) would
 like to deny it, but people have funny-shaped minds. Mental geography
 is not linear, and cannot be mapped onto a flat surface without
 severe distortion. But for the last score years or so, computer
 reductionists have been first bowing down at the Temple of Orthogonality,
 then rising up to preach their ideas of ascetic rectitude to any who
 would listen.

  Their fervent but misguided desire was simply to squash your mind to
 fit their mindset, to smush your patterns of thought into some sort of
 Hyperdimensional Flatland. It's a joyless existence, being smushed.

 --- Learning Perl on Win32 Systems

 If you think this is a big headache, you're right. No one likes
 this situation, but Perl does the best it can with the input and
 encodings it has to deal with. If only we could reset history and
 not make so many mistakes nest time.

 --- Learning Perl 6th Edition

ACKNOWLEDGEMENTS

This software was made referring to software and the document that the following hackers or persons had made. I am thankful to all persons.

Rick Yamashita, Shift_JIS
ttp://furukawablog.spaces.live.com/Blog/cns!1pmWgsL289nm7Shn7cS0jHzA!2225.entry (dead link)
ttp://shino.tumblr.com/post/116166805/1981-us-jis
(add 'h' at head)
http://www.wdic.org/w/WDIC/%E3%82%B7%E3%83%95%E3%83%88JIS

Larry Wall, Perl
http://www.perl.org/

Kazumasa Utashiro, jcode.pl
ftp://ftp.iij.ad.jp/pub/IIJ/dist/utashiro/perl/
http://log.utashiro.com/pub/2006/07/jkondo_a580.html

Jeffrey E. F. Friedl, Mastering Regular Expressions
http://regex.info/

SADAHIRO Tomoyuki, The right way of using Shift_JIS
http://homepage1.nifty.com/nomenclator/perl/shiftjis.htm

Yukihiro "Matz" Matsumoto, YAPC::Asia2006 Ruby on Perl(s)
http://www.rubyist.net/~matz/slides/yapc2006/

jscripter, For jperl users
http://homepage1.nifty.com/kazuf/jperl.html

Bruce., Unicode in Perl
http://www.rakunet.org/tsnet/TSabc/18/546.html

Hiroaki Izumi, Perl5.8/Perl5.10 is not useful on the Windows.
http://www.aritia.jp/hizumi/perl/perlwin.html

TSUKAMOTO Makio, Perl memo/file path of Windows
http://digit.que.ne.jp/work/wiki.cgi?Perl%E3%83%A1%E3%83%A2%2FWindows%E3%81%A7%E3%81%AE%E3%83%95%E3%82%A1%E3%82%A4%E3%83%AB%E3%83%91%E3%82%B9

chaichanPaPa, Matching Shift_JIS file name
http://d.hatena.ne.jp/chaichanPaPa/20080802/1217660826

SUZUKI Norio, Jperl
http://homepage2.nifty.com/kipp/perl/jperl/

WATANABE Hirofumi, Jperl
http://www.cpan.org/src/5.0/jperl/
http://search.cpan.org/~watanabe/
ftp://ftp.oreilly.co.jp/pcjp98/watanabe/jperlconf.ppt

Chuck Houpt, Michiko Nozu, MacJPerl
http://habilis.net/macjperl/index.j.html

Kenichi Ishigaki, Pod-PerldocJp, Welcome to modern Perl world
http://search.cpan.org/dist/Pod-PerldocJp/
http://gihyo.jp/dev/serial/01/modern-perl/0031
http://gihyo.jp/dev/serial/01/modern-perl/0032
http://gihyo.jp/dev/serial/01/modern-perl/0033

Dan Kogai, Encode module
http://search.cpan.org/dist/Encode/
http://www.archive.org/details/YAPCAsia2006TokyoPerl58andUnicodeMythsFactsandChanges (video)
http://yapc.g.hatena.ne.jp/jkondo/ (audio)

Juerd, Perl Unicode Advice
http://juerd.nl/site.plp/perluniadvice

daily dayflower, 2008-06-25 perluniadvice
http://d.hatena.ne.jp/dayflower/20080625/1214374293

Jesse Vincent, Compatibility is a virtue
http://www.nntp.perl.org/group/perl.perl5.porters/2010/05/msg159825.html

Tokyo-pm archive
http://mail.pm.org/pipermail/tokyo-pm/
http://mail.pm.org/pipermail/tokyo-pm/1999-September/001844.html
http://mail.pm.org/pipermail/tokyo-pm/1999-September/001854.html

ruby-list
http://blade.nagaokaut.ac.jp/ruby/ruby-list/index.shtml
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-list/2440
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-list/2446
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-list/2569
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-list/9427
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-list/9431
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-list/10500
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-list/10501
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-list/10502
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-list/12385
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-list/12392
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-list/12393
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-list/19156

To install Char::UTF2, copy and paste the appropriate command in to your terminal.

cpanm

cpanm Char::UTF2

CPAN shell

perl -MCPAN -e shell
install Char::UTF2

For more information on module installation, please visit the detailed CPAN module installation guide.

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	Go to GitHub issues (only if GitHub is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

	Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)