Char::UTF2 - Source code filter to escape UTF-2


  use Char::UTF2;
  use Char::UTF2 version;         --- require version
  use Char::UTF2 qw(ord reverse); --- demand enhanced feature of ord and reverse
  use Char::UTF2 version qw(ord reverse);

  # "no Char::UTF2;" not supported


  $ perl Char/ >


  $ perl  --- script written in UTF-2 --- escaped script


  emulate Perl5.6 on perl5.00503
    use warnings;
    use warnings::register;

  dummy functions:


Let's start with a bit of history: jperl 4.019+1.3 introduced UTF-2 support. You could apply chop() and regexps even to complex CJK characters.

JPerl in CPAN Perl Ports (Binary Distributions)

said before,

  As of Perl 5.8.0 it is suggested that instead of JPerl (which is
  based on a quite old release of Perl) you should just use Perl 5.8.0,
  since it can do all that JPerl did, and more.

But was it really so?

In this country, UTF-2 is widely used on mainframe I/O, the personal computer, and the cellular phone. This software treats UTF-2 directly. Therefor there is not UTF8 flag.

A difficult solution makes the problem more difficult. Shall we escape from the encode problem?

Yet Another Future Of

JPerl is very useful software. -- Oops, note, this "JPerl" means Japanized or Japanese Perl, so is unrelated to Java and JVM. Therefore, I named this software better, fitter Char::UTF2.

Now, the last version of JPerl is 5.005_04 and is not maintained now.

Japanization modifier WATANABE Hirofumi said,

  "Because WATANABE am tired I give over maintaing JPerl."

at Slide #15: "The future of JPerl" of

in The Perl Confernce Japan 1998.

When I heard it, I thought that someone excluding me would maintain JPerl. And I slept every night hanging a sock. Night and day, I kept having hope. After 10 years, I noticed that white beard exists in the sock :-)

This software is a source code filter to escape Perl script encoded by UTF-2 given from STDIN or command line parameter. The character code is never converted by escaping the script. Neither the value of the character nor the length of the character string change even if it escapes.

What's this software good for ...

  • Possible to handle raw UTF-2 values

  • Backward compatibility of data, script and how to

  • Only Char/ and Char/, other modules are unnecessary

  • No UTF8 flag, perlunitut and perluniadvice

  • No C programming (for maintain JPerl)

  • Independent from binary file (CPU, OS, perl version, 32bit/64bit)

Let's make yet another future by JPerl's future.

JRE: JPerl Runtime Environment

  |        JPerl Application Script       | Your Script
  |  Source Code Filter, Runtime Routine  | ex. Char/, Char/
  |          PVM 5.00503 or later         | ex. perl 5.00503

A Perl Virtual Machine (PVM) enables a set of computer software programs and data structures to use a virtual machine model for the execution of other computer programs and scripts. The model used by a PVM accepts a form of computer intermediate language commonly referred to as Perl byteorientedcode. This language conceptually represents the instruction set of a byte-oriented, capability architecture.

Basic Idea Of Source Code Filter

I discovered this mail again recently.

[] jus Benkyoukai

save as:

  package SJIS;
  use Filter::Util::Call;
  sub multibyte_filter {
      my $status;
      if (($status = filter_read()) > 0 ) {
  sub import {

I am glad that I could confirm my idea is not so wrong.

Software Composition

   Char/               --- source code filter to escape UTF-2
   Char/              --- run-time routines for Char/
   perl55.bat            --- find and run perl5.5  without %PATH% settings
   perl56.bat            --- find and run perl5.6  without %PATH% settings
   perl58.bat            --- find and run perl5.8  without %PATH% settings
   perl510.bat           --- find and run perl5.10 without %PATH% settings
   perl512.bat           --- find and run perl5.12 without %PATH% settings
   perl514.bat           --- find and run perl5.14 without %PATH% settings
   perl64.bat            --- find and run perl64   without %PATH% settings
   strict.pm_            --- dummy
   warnings.pm_          --- poor
   warnings/register.pm_ --- poor warnings/

   Rename and install strict.pm_ of this distribution to if your system
   doesn't have

Upper Compatibility By Escaping

This software adds the function by 'Escaping' it always, and nothing of the past is broken. Therefore, 'Possible job' never becomes 'Impossible job'. This approach is effective in the field where the retreat is never permitted. Modern Perl/perl can not always solve the problem.

Escaping Your Script (You do)

You need write 'use Char::UTF2;' in your script.

  Before      After
  (nothing)   use Char::UTF2;

Escaping Multiple Octet Code (Char/ provides)

Insert chr(0x5c) before @ [ \ ] ^ ` { | and } in multiple octet of

  • string in single quote ('', q{}, <<'END' and qw{})

  • string in double quote ("", qq{}, <<END, <<"END", ``, qx{} and <<`END`)

  • regexp in single quote (m'', s''', split(''), split(m'') and qr'')

  • regexp in double quote (//, m//, ??, s///, split(//), split(m//) and qr//)

  • character in tr/// (tr/// and y///)

  ex. Japanese Katakana "SO" like [ `/ ] code is "\x83\x5C" in SJIS
                  see     hex dump
  source script   "`/"    [83 5c]
  Here, use SJIS;
                          hex dump
  escaped script  "`\/"   [83 [5c] 5c]
                    ^--- escape by SJIS software
  by the by       see     hex dump
  your eye's      "`/\"   [83 5c] [5c]
  perl eye's      "`\/"   [83] \[5c]
                          hex dump
  in the perl     "`/"    [83] [5c]

Escaping Character Classes (Char/ provides)

The character classes are redefined as follows to backward compatibility.

  Before      After
   .          (?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x0A])
              (?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[\x00-\xFF]) (/s modifier)
  \d          [0-9]
  \s          [\x09\x0A\x0C\x0D\x20]
  \w          [0-9A-Z_a-z]
  \D          (?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^0-9])
  \S          (?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x09\x0A\x0C\x0D\x20])
  \W          (?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^0-9A-Z_a-z])
  \h          [\x09\x20]
  \v          [\x0C\x0A\x0D]
  \H          (?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x09\x20])
  \V          (?:(?:[\xC2-\xDF]|[\xE0-\xE0][\xA0-\xBF]|[\xE1-\xEC][\x80-\xBF]|[\xED-\xED][\x80-\x9F]|[\xEE-\xEF][\x80-\xBF]|[\xF0-\xF0][\x90-\xBF][\x80-\xBF]|[\xF1-\xF3][\x80-\xBF][\x80-\xBF]|[\xF4-\xF4][\x80-\x8F][\x80-\xBF])[\x00-\xFF]|[^\x0C\x0A\x0D])

Also \b and \B are redefined as follows to backward compatibility.

  Before      After
  \b          (?:\A(?=[0-9A-Z_a-z])|(?<=[\x00-\x2F\x40\x5B-\x5E\x60\x7B-\xFF])(?=[0-9A-Z_a-z])|(?<=[0-9A-Z_a-z])(?=[\x00-\x2F\x40\x5B-\x5E\x60\x7B-\xFF]|\z))
  \B          (?:(?<=[0-9A-Z_a-z])(?=[0-9A-Z_a-z])|(?<=[\x00-\x2F\x40\x5B-\x5E\x60\x7B-\xFF])(?=[\x00-\x2F\x40\x5B-\x5E\x60\x7B-\xFF]))

Escaping Built-in Functions (Char/ and Char/ provide)

Insert 'Char::Eutf2::' at head of function name. Char/ provides your script Char::Eutf2::* functions.

  Before      After
  length      length
  substr      substr
  pos         pos
  split       Char::Eutf2::split
  tr///       Char::Eutf2::tr
  tr///b      tr///
  tr///B      tr///
  y///        Char::Eutf2::tr
  y///b       tr///
  y///B       tr///
  chop        Char::Eutf2::chop
  chr         Char::Eutf2::chr
  glob        Char::Eutf2::glob

  Before                   After
  use Perl::Module;        BEGIN { require 'Perl/'; Perl::Module->import() if Perl::Module->can('import'); }
  use Perl::Module @list;  BEGIN { require 'Perl/'; Perl::Module->import(@list) if Perl::Module->can('import'); }
  use Perl::Module ();     BEGIN { require 'Perl/'; }
  no Perl::Module;         BEGIN { require 'Perl/'; Perl::Module->unimport() if Perl::Module->can('unimport'); }
  no Perl::Module @list;   BEGIN { require 'Perl/'; Perl::Module->unimport(@list) if Perl::Module->can('unimport'); }
  no Perl::Module ();      BEGIN { require 'Perl/'; }

Un-Escaping bytes::* Functions (Char/ provides)

Char/ remove 'bytes::' at head of function name.

  Before           After
  bytes::chr       chr
  bytes::index     index
  bytes::length    length
  bytes::ord       ord
  bytes::rindex    rindex
  bytes::substr    substr

Escaping Built-in Standard Module (Char/ provides)

Char/ does "BEGIN { unshift @INC, '/Perl/site/lib/Char::UTF2' }" at head. Store the standard module modified for Char::UTF2 software in this directory to override built-in standard modules.

Escaping Standard Module Content (You do)

You need copy built-in standard module to /Perl/site/lib/Char::UTF2 and change 'use utf8;' to 'use Char::UTF2;' in its. You need help yourself for now.

Back to and see 'Escaping Your Script'. Enjoy hacking!!

Escaping Function Name (You do)

You need write 'Char::UTF2::' at head of function name when you want character oriented function. See 'CHARACTER ORIENTED FUNCTIONS'.

  Before      After
  ord         Char::UTF2::ord
  reverse     Char::UTF2::reverse
  length      Char::UTF2::length
  substr      Char::UTF2::substr
  index       Char::UTF2::index
  rindex      Char::UTF2::rindex

Character Oriented Functions

  • Order Of Character

      $ord = Char::UTF2::ord($string);
      This function returns the numeric value (ASCII or UTF-2) of the first character
      of $string. The return value is always unsigned.
  • Reverse List Or String

      @reverse = Char::UTF2::reverse(@list);
      $reverse = Char::UTF2::reverse(@list);
      In list context, this function returns a list value consisting of the elements of
      @list in the opposite order. The function can be used to create descending
      for (Char::UTF2::reverse(1 .. 10)) { ... }
      Because of the way hashes flatten into lists when passed as a @list, reverse can
      also be used to invert a hash, presuming the values are unique:
      %barfoo = Char::UTF2::reverse(%foobar);
      In scalar context, the function concatenates all the elements of LIST and then
      returns the reverse of that resulting string, character by character.
  • Length By UTF-2 Character

      $length = Char::UTF2::length($string);
      $length = Char::UTF2::length();
      This function returns the length in characters of the scalar value $string. If
      $string is omitted, it returns the Char::UTF2::length of $_.
      Do not try to use length to find the size of an array or hash. Use scalar @array
      for the size of an array, and scalar keys %hash for the number of key/value pairs
      in a hash. (The scalar is typically omitted when redundant.)
      To find the length of a string in bytes rather than characters, say simply:
      $bytes = length($string);
  • Substr By UTF-2 Character

      $substr = Char::UTF2::substr($string,$offset,$length,$replacement);
      $substr = Char::UTF2::substr($string,$offset,$length);
      $substr = Char::UTF2::substr($string,$offset);
      This function extracts a substring out of the string given by $string and returns
      it. The substring is extracted starting at $offset characters from the front of
      the string.
      If $offset is negative, the substring starts that far from the end of the string
      instead. If $length is omitted, everything to the end of the string is returned.
      If $length is negative, the length is calculated to leave that many characters off
      the end of the string. Otherwise, $length indicates the length of the substring to
      extract, which is sort of what you'd expect.
      An alternative to using Char::UTF2::substr as an lvalue is to specify the $replacement
      string as the fourth argument. This allows you to replace parts of the $string and
      return what was there before in one operation, just as you can with splice. The next
      example also replaces the last character of $var with "Curly" and puts that replaced
      character into $oldstr: 
      $oldstr = Char::UTF2::substr($var, -1, 1, "Curly");
      If you assign something shorter than the length of your substring, the string will
      shrink, and if you assign something longer than the length, the string will grow to
      accommodate it. To keep the string the same length, you may need to pad or chop your
      value using sprintf or the x operator. If you attempt to assign to an unallocated
      area past the end of the string, Char::UTF2::substr raises an exception.
      To prepend the string "Larry" to the current value of $_, use:
      Char::UTF2::substr($var, 0, 0, "Larry");
      To instead replace the first character of $_ with "Moe", use:
      Char::UTF2::substr($var, 0, 1, "Moe");
      And finally, to replace the last character of $var with "Curly", use:
      Char::UTF2::substr($var, -1, 1, "Curly");
  • Index By UTF-2 Character

      $index = Char::UTF2::index($string,$substring,$offset);
      $index = Char::UTF2::index($string,$substring);
      This function searches for one string within another. It returns the position of
      the first occurrence of $substring in $string. The $offset, if specified, says how
      many characters from the start to skip before beginning to look. Positions are
      based at 0. If the substring is not found, the function returns one less than the
      base, ordinarily -1. To work your way through a string, you might say:
      $pos = -1;
      while (($pos = Char::UTF2::index($string, $lookfor, $pos)) > -1) {
          print "Found at $pos\n";
  • Rindex By UTF-2 Character

      $rindex = Char::UTF2::rindex($string,$substring,$position);
      $rindex = Char::UTF2::rindex($string,$substring);
      This function works just like Char::UTF2::index except that it returns the position of
      the last occurrence of $substring in $string (a reverse index). The function
      returns -1 if not $substring is found. $position, if specified, is the rightmost
      position that may be returned. To work your way through a string backward, say:
      $pos = Char::UTF2::length($string);
      while (($pos = Char::UTF2::rindex($string, $lookfor, $pos)) >= 0) {
          print "Found at $pos\n";

Perl5.6 Emulation On perl5.005

  Using warnings pragma on perl5.00503 by rename files.

  warnings.pm_ -->
  warnings/register.pm_ --> warnings/

  To be compatible with Perl5.6 on perl5.005, script is converted as follows.

  Before          After                  in BEGIN { } of Char/
  binmode(...);   Char::Eutf2::binmode(...);   *CORE::GLOBAL::binmode = ...
  open(...);      Char::Eutf2::open(...);      *CORE::GLOBAL::open    = ...
  • binmode (Perl5.6 emulation on perl5.005)

      binmode(FILEHANDLE, $disciplines);
      binmode($filehandle, $disciplines);
      * two arguments
      If you are using perl5.005 other than MacPerl, Char::UTF2 software emulate perl5.6's
      binmode function. Only the point is here. See also perlfunc/binmode for details.
      This function arranges for the FILEHANDLE to have the semantics specified by the
      $disciplines argument. If $disciplines is omitted, ':raw' semantics are applied
      to the filehandle. If FILEHANDLE is an expression, the value is taken as the
      name of the filehandle or a reference to a filehandle, as appropriate.
      The binmode function should be called after the open but before any I/O is done
      on the filehandle. The only way to reset the mode on a filehandle is to reopen
      the file, since the various disciplines may have treasured up various bits and
      pieces of data in various buffers.
      The ":raw" discipline tells Perl to keep its cotton-pickin' hands off the data.
      For more on how disciplines work, see the open function.
  • open (Perl5.6 emulation on perl5.005)

      open(FILEHANDLE, $mode, $expr);
      open(FILEHANDLE, $expr);
      open(my $filehandle, $mode, $expr);
      open(my $filehandle, $expr);
      open(my $filehandle);
      * autovivification filehandle
      * three arguments
      If you are using perl5.005, Char::UTF2 software emulate perl5.6's open function.
      Only the point is here. See also perlfunc/open for details.
      As that example shows, the FILEHANDLE argument is often just a simple identifier
      (normally uppercase), but it may also be an expression whose value provides a
      reference to the actual filehandle. (The reference may be either a symbolic
      reference to the filehandle name or a hard reference to any object that can be
      interpreted as a filehandle.) This is called an indirect filehandle, and any
      function that takes a FILEHANDLE as its first argument can handle indirect
      filehandles as well as direct ones. But open is special in that if you supply
      it with an undefined variable for the indirect filehandle, Perl will automatically
      define that variable for you, that is, autovivifying it to contain a proper
      filehandle reference.
          my $fh;                   # (uninitialized)
          open($fh, ">logfile")     # $fh is autovivified
              or die "Can't create logfile: $!";
              ...                   # do stuff with $fh
      }                             # $fh closed here
      The my $fh declaration can be readably incorporated into the open:
      open my $fh, ">logfile" or die ...
      The > symbol you've been seeing in front of the filename is an example of a mode.
      Historically, the two-argument form of open came first. The recent addition of
      the three-argument form lets you separate the mode from the filename, which has
      the advantage of avoiding any possible confusion between the two. In the following
      example, we know that the user is not trying to open a filename that happens to
      start with ">". We can be sure that they're specifying a $mode of ">", which opens
      the file named in $expr for writing, creating the file if it doesn't exist and
      truncating the file down to nothing if it already exists:
      open(LOG, ">", "logfile") or die "Can't create logfile: $!";
      With the one- or two-argument form of open, you have to be careful when you use
      a string variable as a filename, since the variable may contain arbitrarily
      weird characters (particularly when the filename has been supplied by arbitrarily
      weird characters on the Internet). If you're not careful, parts of the filename
      might get interpreted as a $mode string, ignorable whitespace, a dup specification,
      or a minus.
      Here's one historically interesting way to insulate yourself:
      $path =~ s#^([ ])#./$1#;
      open (FH, "< $path\0") or die "can't open $path: $!";
      But that's still broken in several ways. Instead, just use the three-argument
      form of open to open any arbitrary filename cleanly and without any (extra)
      security risks:
      open(FH, "<", $path) or die "can't open $path: $!";
      As of the 5.6 release of Perl, you can specify binary mode in the open function
      without a separate call to binmode. As part of the $mode
      argument (but only in the three-argument form), you may specify various input
      and output disciplines.
      To do the equivalent of a binmode, use the three argument form of open and stuff
      a discipline of :raw in after the other $mode characters:
      open(FH, "<:raw", $path) or die "can't open $path: $!";
      Table 1. I/O Disciplines
      Discipline      Meaning
      :raw            Binary mode; do no processing
      :crlf           Text mode; Intuit newlines
                      (DOS-like system only)
      :encoding(...)  Legacy encoding
      You'll be able to stack disciplines that make sense to stack, so, for instance,
      you could say:
      open(FH, "<:crlf:encoding(Char::UTF2)", $path) or die "can't open $path: $!";

Ignore Pragmas And Modules

  Before                    After
  use strict;               use strict; no strict qw(refs);
  require utf8;             # require utf8;
  require bytes;            # require bytes;
  require I18N::Japanese;   # require I18N::Japanese;
  require I18N::Collate;    # require I18N::Collate;
  require I18N::JExt;       # require I18N::JExt;
  require File::DosGlob;    # require File::DosGlob;
  require Wild;             # require Wild;
  require Wildcard;         # require Wildcard;
  require Japanese;         # require Japanese;
  use utf8;                 # use utf8;
  use bytes;                # use bytes;
  use I18N::Japanese;       # use I18N::Japanese;
  use I18N::Collate;        # use I18N::Collate;
  use I18N::JExt;           # use I18N::JExt;
  use File::DosGlob;        # use File::DosGlob;
  use Wild;                 # use Wild;
  use Wildcard;             # use Wildcard;
  use Japanese;             # use Japanese;
  no utf8;                  # no utf8;
  no bytes;                 # no bytes;
  no I18N::Japanese;        # no I18N::Japanese;
  no I18N::Collate;         # no I18N::Collate;
  no I18N::JExt;            # no I18N::JExt;
  no File::DosGlob;         # no File::DosGlob;
  no Wild;                  # no Wild;
  no Wildcard;              # no Wildcard;
  no Japanese;              # no Japanese;

  Comment out pragma to ignore utf8 environment, and Char/ provides these
  • Dummy utf8::upgrade

      $num_octets = utf8::upgrade($string);
      Returns the number of octets necessary to represent the string.
  • Dummy utf8::downgrade

      $success = utf8::downgrade($string[, FAIL_OK]);
      Returns true always.
  • Dummy utf8::encode

      Returns nothing.
  • Dummy utf8::decode

      $success = utf8::decode($string);
      Returns true always.
  • Dummy utf8::is_utf8

      $flag = utf8::is_utf8(STRING);
      Returns false always.
  • Dummy utf8::valid

      $flag = utf8::valid(STRING);
      Returns true always.
  • Dummy bytes::chr

      This function is same as chr.
  • Dummy bytes::index

      This function is same as index.
  • Dummy bytes::length

      This function is same as length.
  • Dummy bytes::ord

      This function is same as ord.
  • Dummy bytes::rindex

      This function is same as rindex.
  • Dummy bytes::substr

      This function is same as substr.

Environment Variable

 This software uses the flock function for exclusive control. The execution of the
 program is blocked until it becomes possible to read or write the file.
 You can have it not block in the flock function by defining environment variable
 (The value '1' doesn't have the meaning)


Please patches and report problems to author are welcome.

  • format

    Function "format" can't handle multiple octet code same as original Perl.

  • Char::UTF2::substr As Lvalue

    Char::UTF2::substr differs from CORE::substr, and cannot be used as a lvalue. To change part of a string, you can use the optional fourth argument which is the replacement string.

    Char::UTF2::substr($string, 13, 4, "JPerl");


INABA Hitoshi <>

This project was originated by INABA Hitoshi.


This software is free software; you can redistribute it and/or modify it under the same terms as Perl itself. See perlartistic.

This software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

My Goal

P.401 See chapter 15: Unicode of ISBN 0-596-00027-8 Programming Perl Third Edition.

Before the introduction of Unicode support in perl, The eq operator just compared the byte-strings represented by two scalars. Beginning with perl 5.8, eq compares two byte-strings with simultaneous consideration of the UTF8 flag.

  Information processing model beginning with perl 5.8
    |     Text strings     |                     |
    +----------+-----------|    Binary strings   |
    |   UTF8   |  Latin-1  |                     |
    | UTF8     |            Not UTF8             |
    | Flagged  |            Flagged              |
    You should memorize this figure.
    (Why is only Latin-1 special?)

This change consequentially made a big gap between a past script and new script. Both scripts cannot re-use the code mutually any longer. Because a new method puts a strain in the programmer, it will still take time to replace all the in existence scripts.

The biggest problem of new method is that the UTF8 flag can't synchronize to real encode of string. Thus you must debug about UTF8 flag, before your script. How to solve it by returning to a past method, I will quote page 402 of Programming Perl, 3rd ed. again.

  Information processing model beginning with this software
    |           Octet Strings           | aka Binary strings
    |         Character Strings         | aka Text strings
    |      ASCII Compatible Encoding    | ex. UTF-2
                (No UTF8 Flag)
    You need not memorize this figure.

Ideally, I'd like to achieve these five Goals:

  • Goal #1:

    Old byte-oriented programs should not spontaneously break on the old byte-oriented data they used to work on.

    It has already been achieved by UTF-2 designed for combining with old byte-oriented ASCII.

  • Goal #2:

    Old byte-oriented programs should magically start working on the new character-oriented data when appropriate.

    Still now, 1 octet is counted with 1 by embedded functions length, substr, index, rindex and pos that handle length and position of string. In this part, there is no change. The length of 1 character of 2 octet code is 2.

    On the other hand, the regular expression in the script is added the multibyte anchoring processing with this software, instead of you.

    figure of Goal #1 and Goal #2.

                                   GOAL#1  GOAL#2
                            (a)     (b)     (c)     (d)     (e)
          | data         |  Old  |  Old  |  New  |  Old  |  New  |
          | script       |  Old  |      Old      |      New      |
          | interpreter  |  Old  |              New              |
          Old --- Old byte-oriented
          New --- New character-oriented

    There is a combination from (a) to (e) in data, script and interpreter of old and new. Let's add the Encode module and this software did not exist at time of be written this document and JPerl did exist.

                            (a)     (b)     (c)     (d)     (e)
                                          JPerl           Encode,Char::UTF2
          | data         |  Old  |  Old  |  New  |  Old  |  New  |
          | script       |  Old  |      Old      |      New      |
          | interpreter  |  Old  |              New              |
          Old --- Old byte-oriented
          New --- New character-oriented

    The reason why JPerl is very excellent is that it is at the position of (c). That is, it is not necessary to do a special description to the script to process new character-oriented string.

    Contrasting is Encode module and describing "use Char::UTF2;" on this software, in this case, a new description is necessary.

  • Goal #3:

    Programs should run just as fast in the new character-oriented mode as in the old byte-oriented mode.

    It is impossible. Because the following time is necessary.

    (1) Time of escape script for old byte-oriented perl.

  • Goal #4:

    Perl should remain one language, rather than forking into a byte-oriented Perl and a character-oriented Perl.

    JPerl forked the perl interpreter so as not to fork the Perl language. But the Perl core team might not hope for the perl interpreter's fork. As a result, the Perl language forked, and the community was reduced through necessity.

    A character-oriented perl is not necessary to make it specially, because a byte-oriented perl can already treat the binary data. This software is only an application program of Perl, a filter program. If perl can be executed, this software will be able to be executed.

    And you will get support from the Perl community, when you solve the problem by the Perl script.

  • Goal #5:

    JPerl users will be able to maintain JPerl by Perl.

    May the JPerl be with you, always.

Back when Programming Perl, 3rd ed. was written, UTF8 flag was not born and Perl is designed to make the easy jobs easy. This software provide programming environment like at that time.


 Programming Perl, Second Edition
 By Larry Wall, Tom Christiansen, Randal L. Schwartz
 January 1900 (really so?)
 Pages: 670
 ISBN 10: 1-56592-149-6 | ISBN 13: 9781565921498

 Programming Perl, Third Edition
 By Larry Wall, Tom Christiansen, Jon Orwant
 Third Edition  July 2000
 Pages: 1104
 ISBN 10: 0-596-00027-8 | ISBN 13:9780596000271

 Perl Cookbook, Second Edition
 By Tom Christiansen, Nathan Torkington
 Second Edition  August 2003
 Pages: 964
 ISBN 10: 0-596-00313-7 | ISBN 13: 9780596003135

 Perl in a Nutshell, Second Edition
 By Stephen Spainhour, Ellen Siever, Nathan Patwardhan
 Second Edition  June 2002
 Pages: 760
 Series: In a Nutshell
 ISBN 10: 0-596-00241-6 | ISBN 13: 9780596002411

 Learning Perl on Win32 Systems
 By Randal L. Schwartz, Erik Olson, Tom Christiansen
 August 1997
 Pages: 306
 ISBN 10: 1-56592-324-3 | ISBN 13: 9781565923249

 Learning Perl, Fifth Edition
 By Randal L. Schwartz, Tom Phoenix, brian d foy
 June 2008
 Pages: 352
 Print ISBN:978-0-596-52010-6 | ISBN 10: 0-596-52010-7
 Ebook ISBN:978-0-596-10316-3 | ISBN 10: 0-596-10316-6

 Futato, Irving, Jepson, Patwardhan, Siever
 ISBN 10: 1-56592-370-7

 Understanding Japanese Information Processing
 By Ken Lunde
 January 1900
 Pages: 470
 ISBN 10: 1-56592-043-0 | ISBN 13: 9781565920439

 CJKV Information Processing
 Chinese, Japanese, Korean & Vietnamese Computing
 By Ken Lunde
 First Edition  January 1999
 Pages: 1128
 ISBN 10: 1-56592-224-7 | ISBN 13:9781565922242
 ISBN 4-87311-108-0

 Mastering Regular Expressions, Second Edition
 By Jeffrey E. F. Friedl
 Second Edition  July 2002
 Pages: 484
 ISBN 10: 0-596-00289-0 | ISBN 13: 9780596002893

 Mastering Regular Expressions, Third Edition
 By Jeffrey E. F. Friedl
 Third Edition  August 2006
 Pages: 542
 ISBN 10: 0-596-52812-4 | ISBN 13:9780596528126
 ISBN 978-4-87311-359-3

 Regular Expressions Cookbook
 By Jan Goyvaerts, Steven Levithan
 May 2009
 Pages: 512
 ISBN 10:0-596-52068-9 | ISBN 13: 978-0-596-52068-7

 Larry Wall, Randal L.Schwartz, Yoshiyuki Kondo
 December 1997
 ISBN 4-89052-384-7

 Kouji Shibano
 Pages: 1456
 ISBN 4-542-20129-5

 1993 Aug
 Pages: 172
 T1008901080816 ZASSHI 08901-8

 MacPerl Power and Ease
 By Vicki Brown, Chris Nandor
 April 1998
 Pages: 350
 ISBN 10: 1881957322 | ISBN 13: 978-1881957324

 Other Tools



This software was made referring to software and the document that the following hackers or persons had made. I am thankful to all persons.

 Rick Yamashita, Shift_JIS
 ttp://!1pmWgsL289nm7Shn7cS0jHzA!2225.entry (dead link)
 (add 'h' at head)

 Larry Wall, Perl

 Kazumasa Utashiro,

 Jeffrey E. F. Friedl, Mastering Regular Expressions

 SADAHIRO Tomoyuki, The right way of using Shift_JIS

 Yukihiro "Matz" Matsumoto, YAPC::Asia2006 Ruby on Perl(s)

 jscripter, For jperl users

 Bruce., Unicode in Perl

 Hiroaki Izumi, Perl5.8/Perl5.10 is not useful on the Windows.

 TSUKAMOTO Makio, Perl memo/file path of Windows

 chaichanPaPa, Matching Shift_JIS file name

 SUZUKI Norio, Jperl

 WATANABE Hirofumi, Jperl

 Chuck Houpt, Michiko Nozu, MacJPerl

 Kenichi Ishigaki, Pod-PerldocJp, Welcome to modern Perl world

 Dan Kogai, Encode module

 Juerd, Perl Unicode Advice

 daily dayflower, 2008-06-25 perluniadvice

 Jesse Vincent, Compatibility is a virtue

 Tokyo-pm archive