The London Perl and Raku Workshop takes place on 26th Oct 2024. If your company depends on Perl, please consider sponsoring and/or attending.

NAME

Latin3 - Source code filter to escape Latin-3 script

Install and Usage

There are two steps there:

  • You'll have to download Latin3.pm and Elatin3.pm and put it in your perl lib directory.

  • You'll need to write "use Latin3;" at head of the script.

SYNOPSIS

use Latin3;
use Latin3 ver.sion;             --- require minimum version
use Latin3 ver.sion.0;           --- expects version (match or die)

# "no Latin3;" not supported

or

$ perl Latin3.pm Latin-3_script.pl > Escaped_script.pl.e

then

$ perl Escaped_script.pl.e

Latin-3_script.pl  --- script written in Latin-3
Escaped_script.pl.e --- escaped script

subroutines:
  Latin3::eval(...);
dummy functions:
  utf8::upgrade(...);
  utf8::downgrade(...);
  utf8::encode(...);
  utf8::decode(...);
  utf8::is_utf8(...);
  utf8::valid(...);
  bytes::chr(...);
  bytes::index(...);
  bytes::length(...);
  bytes::ord(...);
  bytes::rindex(...);
  bytes::substr(...);

ABSTRACT

Latin3 software is "middleware" between perl interpreter and your Perl script written in Latin-3.

Perl is optimized for problems which are about 90% working with text and about 10% everything else. Even if this "text" doesn't contain Latin-3, Perl3 or later can treat Latin-3 as binary data.

By "use Latin3;", it automatically interpret your script as Latin-3. The various functions of perl including a regular expression can treat Latin-3 now. The function length treats length per byte. This software does not use UTF8 flag.

Yet Another Future Of

JPerl is very useful software. -- Oops, note, this "JPerl" means "Japanized Perl" or "Japanese Perl". Therefore, it is unrelated to JPerl of the following.

JPerl is an implementation of Perl written in Java.
http://www.javainc.com/projects/jperl/

jPerl - Perl on the JVM
http://www.dzone.com/links/175948.html

Jamie's PERL scripts for bioinformatics
http://code.google.com/p/jperl/

jperl (Jonathan Perl)
https://github.com/jperl

Now, the last version of JPerl is 5.005_04 and is not maintained now.

Japanization modifier WATANABE Hirofumi said,

"Because WATANABE am tired I give over maintaing JPerl."

at Slide #15: "The future of JPerl" of

ftp://ftp.oreilly.co.jp/pcjp98/watanabe/jperlconf.ppt

in The Perl Confernce Japan 1998.

When I heard it, I thought that someone excluding me would maintain JPerl. And I slept every night hanging a sock. Night and day, I kept having hope. After 10 years, I noticed that white beard exists in the sock :-)

This software is a source code filter to escape Perl script encoded by Latin-3 given from STDIN or command line parameter. The character code is never converted by escaping the script. Neither the value of the character nor the length of the character string change even if it escapes.

I learned the following things from the successful software.

  • Upper Compatibility like Perl4 to Perl5

  • Maximum Portability like jcode.pl

  • Remains One Language Handling Raw Latin-3, Doesn't Use UTF8 flag like JPerl

  • Remains One Interpreter like Encode module

  • Code Set Independent like Ruby

  • Monolithic Script like cpanminus

  • There's more than one way to do it like Perl itself

I am excited about this software and Perl's future --- I hope you are too.

JRE: JPerl Runtime Environment

+---------------------------------------+
|        JPerl Application Script       | Your Script
+---------------------------------------+
|  Source Code Filter, Runtime Routine  | ex. Latin3.pm, Elatin3.pm
+---------------------------------------+
|          PVM 5.00503 or later         | ex. perl 5.00503
+---------------------------------------+

A Perl Virtual Machine (PVM) enables a set of computer software programs and data structures to use a virtual machine model for the execution of other computer programs and scripts. The model used by a PVM accepts a form of computer intermediate language commonly referred to as Perl byteorientedcode. This language conceptually represents the instruction set of a byte-oriented, capability architecture.

Basic Idea of Source Code Filter

I discovered this mail again recently.

[Tokyo.pm] jus Benkyoukai

http://mail.pm.org/pipermail/tokyo-pm/1999-September/001854.html

save as: SJIS.pm

package SJIS;
use Filter::Util::Call;
sub multibyte_filter {
    my $status;
    if (($status = filter_read()) > 0 ) {
        s/([\x81-\x9f\xe0-\xef])([\x40-\x7e\x80-\xfc])/
            sprintf("\\x%02x\\x%02x",ord($1),ord($2))
        /eg;
    }
    $status;
}
sub import {
    filter_add(\&multibyte_filter);
}
1;

I am glad that I could confirm my idea is not so wrong.

Command-line Wildcard Expansion on DOS-like Systems

The default command shells on DOS-like systems (COMMAND.COM or cmd.exe or Win95Cmd.exe) do not expand wildcard arguments supplied to programs. Instead, import of Elatin3.pm works well.

in Elatin3.pm
#
# @ARGV wildcard globbing
#
sub import {

    if ($^O =~ /\A (?: MSWin32 | NetWare | symbian | dos ) \z/oxms) {
        my @argv = ();
        for (@ARGV) {

            # has space
            if (/\A (?:$q_char)*? [ ] /oxms) {
                if (my @glob = Elatin3::glob(qq{"$_"})) {
                    push @argv, @glob;
                }
                else {
                    push @argv, $_;
                }
            }

            # has wildcard metachar
            elsif (/\A (?:$q_char)*? [*?] /oxms) {
                if (my @glob = Elatin3::glob($_)) {
                    push @argv, @glob;
                }
                else {
                    push @argv, $_;
                }
            }

            # no wildcard globbing
            else {
                push @argv, $_;
            }
        }
        @ARGV = @argv;
    }
}

Software Composition

Latin3.pm               --- source code filter to escape Latin-3
Elatin3.pm              --- run-time routines for Latin3.pm

Upper Compatibility by Escaping

This software adds the function by 'Escaping' it always, and nothing of the past is broken. Therefore, 'Possible job' never becomes 'Impossible job'. This approach is effective in the field where the retreat is never permitted. It means incompatible upgrade of Perl should be rewound.

Escaping Your Script (You do)

You need write 'use Latin3;' in your script.

---------------------
You do
---------------------
use Latin3;
---------------------

Calling 'Elatin3::ignorecase()' (Latin3 software provides)

Latin3 software applies calling 'Elatin3::ignorecase()' instead of /i modifier.

--------------------------------------------------------------------------------
Before                  After
--------------------------------------------------------------------------------
m/...$var.../i          m/...@{[Elatin3::ignorecase($var)]}.../
--------------------------------------------------------------------------------

Escaping Character Classes (Elatin3.pm provides)

The character classes are redefined as follows to backward compatibility.

---------------------------------------------------------------
Before        After
---------------------------------------------------------------
 .            ${Elatin3::dot}
              ${Elatin3::dot_s}    (/s modifier)
\d            [0-9]              (universally)
\s            \s
\w            [0-9A-Z_a-z]       (universally)
\D            ${Elatin3::eD}
\S            ${Elatin3::eS}
\W            ${Elatin3::eW}
\h            [\x09\x20]
\v            [\x0A\x0B\x0C\x0D]
\H            ${Elatin3::eH}
\V            ${Elatin3::eV}
\C            [\x00-\xFF]
\X            X                  (so, just 'X')
\R            ${Elatin3::eR}
\N            ${Elatin3::eN}
---------------------------------------------------------------

Also POSIX-style character classes.

---------------------------------------------------------------
Before        After
---------------------------------------------------------------
[:alnum:]     [\x30-\x39\x41-\x5A\x61-\x7A]
[:alpha:]     [\x41-\x5A\x61-\x7A]
[:ascii:]     [\x00-\x7F]
[:blank:]     [\x09\x20]
[:cntrl:]     [\x00-\x1F\x7F]
[:digit:]     [\x30-\x39]
[:graph:]     [\x21-\x7F]
[:lower:]     [\x61-\x7A]
              [\x41-\x5A\x61-\x7A]     (/i modifier)
[:print:]     [\x20-\x7F]
[:punct:]     [\x21-\x2F\x3A-\x3F\x40\x5B-\x5F\x60\x7B-\x7E]
[:space:]     [\s\x0B]
[:upper:]     [\x41-\x5A]
              [\x41-\x5A\x61-\x7A]     (/i modifier)
[:word:]      [\x30-\x39\x41-\x5A\x5F\x61-\x7A]
[:xdigit:]    [\x30-\x39\x41-\x46\x61-\x66]
[:^alnum:]    ${Elatin3::not_alnum}
[:^alpha:]    ${Elatin3::not_alpha}
[:^ascii:]    ${Elatin3::not_ascii}
[:^blank:]    ${Elatin3::not_blank}
[:^cntrl:]    ${Elatin3::not_cntrl}
[:^digit:]    ${Elatin3::not_digit}
[:^graph:]    ${Elatin3::not_graph}
[:^lower:]    ${Elatin3::not_lower}
              ${Elatin3::not_lower_i}    (/i modifier)
[:^print:]    ${Elatin3::not_print}
[:^punct:]    ${Elatin3::not_punct}
[:^space:]    ${Elatin3::not_space}
[:^upper:]    ${Elatin3::not_upper}
              ${Elatin3::not_upper_i}    (/i modifier)
[:^word:]     ${Elatin3::not_word}
[:^xdigit:]   ${Elatin3::not_xdigit}
---------------------------------------------------------------

\b and \B are redefined as follows to backward compatibility.

---------------------------------------------------------------
Before      After
---------------------------------------------------------------
\b          ${Elatin3::eb}
\B          ${Elatin3::eB}
---------------------------------------------------------------

Definitions in Elatin3.pm.

---------------------------------------------------------------------------------------------------------------------------------------------------------
After                    Definition
---------------------------------------------------------------------------------------------------------------------------------------------------------
${Elatin3::dot}            qr{(?>[^\x0A])};
${Elatin3::dot_s}          qr{(?>[\x00-\xFF])};
${Elatin3::eD}             qr{(?>[^0-9])};
${Elatin3::eS}             qr{(?>[^\s])};
${Elatin3::eW}             qr{(?>[^0-9A-Z_a-z])};
${Elatin3::eH}             qr{(?>[^\x09\x20])};
${Elatin3::eV}             qr{(?>[^\x0A\x0B\x0C\x0D])};
${Elatin3::eR}             qr{(?>\x0D\x0A|[\x0A\x0D])};
${Elatin3::eN}             qr{(?>[^\x0A])};
${Elatin3::not_alnum}      qr{(?>[^\x30-\x39\x41-\x5A\x61-\x7A])};
${Elatin3::not_alpha}      qr{(?>[^\x41-\x5A\x61-\x7A])};
${Elatin3::not_ascii}      qr{(?>[^\x00-\x7F])};
${Elatin3::not_blank}      qr{(?>[^\x09\x20])};
${Elatin3::not_cntrl}      qr{(?>[^\x00-\x1F\x7F])};
${Elatin3::not_digit}      qr{(?>[^\x30-\x39])};
${Elatin3::not_graph}      qr{(?>[^\x21-\x7F])};
${Elatin3::not_lower}      qr{(?>[^\x61-\x7A])};
${Elatin3::not_lower_i}    qr{(?>[^\x41-\x5A\x61-\x7A])}; # Perl 5.16 compatible
# ${Elatin3::not_lower_i}    qr{(?>[\x00-\xFF])};                   # older Perl compatible
${Elatin3::not_print}      qr{(?>[^\x20-\x7F])};
${Elatin3::not_punct}      qr{(?>[^\x21-\x2F\x3A-\x3F\x40\x5B-\x5F\x60\x7B-\x7E])};
${Elatin3::not_space}      qr{(?>[^\s\x0B])};
${Elatin3::not_upper}      qr{(?>[^\x41-\x5A])};
${Elatin3::not_upper_i}    qr{(?>[^\x41-\x5A\x61-\x7A])}; # Perl 5.16 compatible
# ${Elatin3::not_upper_i}    qr{(?>[\x00-\xFF])};                   # older Perl compatible
${Elatin3::not_word}       qr{(?>[^\x30-\x39\x41-\x5A\x5F\x61-\x7A])};
${Elatin3::not_xdigit}     qr{(?>[^\x30-\x39\x41-\x46\x61-\x66])};

# This solution is not perfect. I beg better solution from you who are reading this.
${Elatin3::eb}             qr{(?:\A(?=[0-9A-Z_a-z])|(?<=[\x00-\x2F\x40\x5B-\x5E\x60\x7B-\xFF])(?=[0-9A-Z_a-z])|(?<=[0-9A-Z_a-z])(?=[\x00-\x2F\x40\x5B-\x5E\x60\x7B-\xFF]|\z))};
${Elatin3::eB}             qr{(?:(?<=[0-9A-Z_a-z])(?=[0-9A-Z_a-z])|(?<=[\x00-\x2F\x40\x5B-\x5E\x60\x7B-\xFF])(?=[\x00-\x2F\x40\x5B-\x5E\x60\x7B-\xFF]))};
---------------------------------------------------------------------------------------------------------------------------------------------------------

Un-Escaping \ of \b{}, \B{}, \N{}, \p{}, \P{}, and \X (Latin3 software provides)

Latin3 software removes '\' at head of alphanumeric regexp metasymbols \b{}, \B{}, \N{}, \p{}, \P{} and \X. By this method, you can avoid the trap of the abstraction.

See also, Deprecate literal unescaped "{" in regexes. http://perl5.git.perl.org/perl.git/commit/2a53d3314d380af5ab5283758219417c6dfa36e9

------------------------------------
Before           After
------------------------------------
\b{...}          b\{...}
\B{...}          B\{...}
\N{CHARNAME}     N\{CHARNAME}
\p{L}            p\{L}
\p{^L}           p\{^L}
\p{\^L}          p\{\^L}
\pL              pL
\P{L}            P\{L}
\P{^L}           P\{^L}
\P{\^L}          P\{\^L}
\PL              PL
\X               X
------------------------------------

Escaping Built-in Functions (Latin3 software provides)

Insert 'Elatin3::' at head of function name. Elatin3.pm provides your script Elatin3::* subroutines.

-------------------------------------------
Before      After            Works as
-------------------------------------------
lc          Elatin3::lc        Character
lcfirst     Elatin3::lcfirst   Character
uc          Elatin3::uc        Character
ucfirst     Elatin3::ucfirst   Character
fc          Elatin3::fc        Character
chr         Elatin3::chr       Character
glob        Elatin3::glob      Character
-------------------------------------------

------------------------------------------------------------------------------------------------------------------------
Before                   After
------------------------------------------------------------------------------------------------------------------------
use Perl::Module;        BEGIN { require 'Perl/Module.pm'; Perl::Module->import() if Perl::Module->can('import'); }
use Perl::Module @list;  BEGIN { require 'Perl/Module.pm'; Perl::Module->import(@list) if Perl::Module->can('import'); }
use Perl::Module ();     BEGIN { require 'Perl/Module.pm'; }
no Perl::Module;         BEGIN { require 'Perl/Module.pm'; Perl::Module->unimport() if Perl::Module->can('unimport'); }
no Perl::Module @list;   BEGIN { require 'Perl/Module.pm'; Perl::Module->unimport(@list) if Perl::Module->can('unimport'); }
no Perl::Module ();      BEGIN { require 'Perl/Module.pm'; }
------------------------------------------------------------------------------------------------------------------------

Yada Yada Operator (Latin3 software provides)

The yada yada operator (noted ...) is a placeholder for code. Perl parses it
without error, but when you try to execute a yada yada, it throws an exception
with the text Unimplemented:

sub unimplemented { ... }
eval { unimplemented() };
if ( $@ eq 'Unimplemented' ) {
    print "I found the yada yada!\n";
}

You can only use the yada yada to stand in for a complete statement. These
examples of the yada yada work:

{ ... }
sub foo { ... }
...;
eval { ... };
sub foo {
    my( $self ) = shift;
    ...;
}
do { my $n; ...; print 'Hurrah!' };

The yada yada cannot stand in for an expression that is part of a larger statement
since the ... is also the three-dot version of the range operator
(see "Range Operators"). These examples of the yada yada are still syntax errors:

print ...;
open my($fh), '>', '/dev/passwd' or ...;
if ( $condition && ... ) { print "Hello\n" };

There are some cases where Perl can't immediately tell the difference between an
expression and a statement. For instance, the syntax for a block and an anonymous
hash reference constructor look the same unless there's something in the braces that
give Perl a hint. The yada yada is a syntax error if Perl doesn't guess that the
{ ... } is a block. In that case, it doesn't think the ... is the yada yada because
it's expecting an expression instead of a statement:

my @transformed = map { ... } @input;  # syntax error

You can use a ; inside your block to denote that the { ... } is a block and not a
hash reference constructor. Now the yada yada works:

my @transformed = map {; ... } @input; # ; disambiguates
my @transformed = map { ...; } @input; # ; disambiguates

Un-Escaping bytes::* Subroutines (Latin3 software provides)

Latin3 software removes 'bytes::' at head of subroutine name.

---------------------------------------
Before           After     Works as
---------------------------------------
bytes::chr       chr       Byte
bytes::index     index     Byte
bytes::length    length    Byte
bytes::ord       ord       Byte
bytes::rindex    rindex    Byte
bytes::substr    substr    Byte
---------------------------------------

Ignore Pragmas and Modules

-----------------------------------------------------------
Before                    After
-----------------------------------------------------------
use strict;               use strict; no strict qw(refs);
use 5.12.0;               use 5.12.0; no strict qw(refs);
require utf8;             # require utf8;
require bytes;            # require bytes;
require charnames;        # require charnames;
require I18N::Japanese;   # require I18N::Japanese;
require I18N::Collate;    # require I18N::Collate;
require I18N::JExt;       # require I18N::JExt;
require File::DosGlob;    # require File::DosGlob;
require Wild;             # require Wild;
require Wildcard;         # require Wildcard;
require Japanese;         # require Japanese;
use utf8;                 # use utf8;
use bytes;                # use bytes;
use charnames;            # use charnames;
use I18N::Japanese;       # use I18N::Japanese;
use I18N::Collate;        # use I18N::Collate;
use I18N::JExt;           # use I18N::JExt;
use File::DosGlob;        # use File::DosGlob;
use Wild;                 # use Wild;
use Wildcard;             # use Wildcard;
use Japanese;             # use Japanese;
no utf8;                  # no utf8;
no bytes;                 # no bytes;
no charnames;             # no charnames;
no I18N::Japanese;        # no I18N::Japanese;
no I18N::Collate;         # no I18N::Collate;
no I18N::JExt;            # no I18N::JExt;
no File::DosGlob;         # no File::DosGlob;
no Wild;                  # no Wild;
no Wildcard;              # no Wildcard;
no Japanese;              # no Japanese;
-----------------------------------------------------------

Comment out pragma to ignore utf8 environment, and Elatin3.pm provides these
functions.
  • Dummy utf8::upgrade

    $num_octets = utf8::upgrade($string);
    
    Returns the number of octets necessary to represent the string.
  • Dummy utf8::downgrade

    $success = utf8::downgrade($string[, FAIL_OK]);
    
    Returns true always.
  • Dummy utf8::encode

    utf8::encode($string);
    
    Returns nothing.
  • Dummy utf8::decode

    $success = utf8::decode($string);
    
    Returns true always.
  • Dummy utf8::is_utf8

    $flag = utf8::is_utf8(STRING);
    
    Returns false always.
  • Dummy utf8::valid

    $flag = utf8::valid(STRING);
    
    Returns true always.
  • Dummy bytes::chr

    This subroutine is same as chr.
  • Dummy bytes::index

    This subroutine is same as index.
  • Dummy bytes::length

    This subroutine is same as length.
  • Dummy bytes::ord

    This subroutine is same as ord.
  • Dummy bytes::rindex

    This subroutine is same as rindex.
  • Dummy bytes::substr

    This subroutine is same as substr.

Environment Variable

This software uses the flock function for exclusive control. The execution of the
program is blocked until it becomes possible to read or write the file.
You can have it not block in the flock function by defining environment variable
CHAR_NONBLOCK.

Example:

  SET CHAR_NONBLOCK=1

(The value '1' doesn't have the meaning)

BUGS, LIMITATIONS, and COMPATIBILITY

I have tested and verified this software using the best of my ability. However, a software containing much regular expression is bound to contain some bugs. Thus, if you happen to find a bug that's in Latin3 software and not your own program, you can try to reduce it to a minimal test case and then report it to the following author's address. If you have an idea that could make this a more useful tool, please let everyone share it.

  • (dummy item to avoid Test::Pod error)

  • cloister of regular expression

    The cloister (?s) and (?i) of a regular expression will not be implemented for the time being. Cloister (?s) can be substituted with the .(dot) and \N on /s modifier. Cloister (?i) can be substituted with \F...\E.

  • Modifier /a /d /l and /u of Regular Expression

    The concept of this software is not to use two or more encoding methods as literal string and literal of regexp in one Perl script. Therefore, modifier /a, /d, /l, and /u are not supported. \d means [0-9] universally.

  • Named Character

    A named character, such \N{GREEK SMALL LETTER EPSILON}, \N{greek:epsilon}, or \N{epsilon} is not supported.

  • Unicode Properties (aka Character Properties) of Regular Expression

    Unicode properties (aka character properties) of regexp are not available. Also (?[]) in regexp of Perl 5.18 is not available. There is no plans to currently support these.

  • Delimiter of String and Regexp

    qq//, q//, qw//, qx//, qr//, m//, s///, tr///, and y/// can't use a wide character as the delimiter.

  • \b{...} Boundaries in Regular Expressions

    Following \b{...} available starting in v5.22 are not supported.

    \b{gcb} or \b{g}   Unicode "Grapheme Cluster Boundary"
    \b{sb}             Unicode "Sentence Boundary"
    \b{wb}             Unicode "Word Boundary"
    \B{gcb} or \B{g}   Unicode "Grapheme Cluster Boundary" doesn't match
    \B{sb}             Unicode "Sentence Boundary" doesn't match
    \B{wb}             Unicode "Word Boundary" doesn't match

AUTHOR

INABA Hitoshi <ina@cpan.org>

This project was originated by INABA Hitoshi.

LICENSE AND COPYRIGHT

This software is free software; you can redistribute it and/or modify it under the same terms as Perl itself. See perlartistic.

This software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

My Goal

P.401 See chapter 15: Unicode of ISBN 0-596-00027-8 Programming Perl Third Edition.

Before the introduction of Unicode support in perl, The eq operator just compared the byte-strings represented by two scalars. Beginning with perl 5.8, eq compares two byte-strings with simultaneous consideration of the UTF8 flag.

/* You are not expected to understand this */

 Information processing model beginning with perl 5.8

   +----------------------+---------------------+
   |     Text strings     |                     |
   +----------+-----------|    Binary strings   |
   |  UTF-8   |  Latin-1  |                     |
   +----------+-----------+---------------------+
   | UTF8     |            Not UTF8             |
   | Flagged  |            Flagged              |
   +--------------------------------------------+
   http://perl-users.jp/articles/advent-calendar/2010/casual/4

 Confusion of Perl string model is made from double meanings of
 "Binary string."
 Meanings of "Binary string"
 1. Non-Text string
 2. Digital octet string

 Let's draw again using those term.

   +----------------------+---------------------+
   |     Text strings     |                     |
   +----------+-----------|   Non-Text strings  |
   |  UTF-8   |  Latin-1  |                     |
   +----------+-----------+---------------------+
   | UTF8     |            Not UTF8             |
   | Flagged  |            Flagged              |
   +--------------------------------------------+
   |            Digital octet string            |
   +--------------------------------------------+

There are people who don't agree to change in the character string processing model of Perl 5.8. It is impossible to get to agree it to majority of Perl user who hardly ever use Perl. How to solve it by returning to a original method, let's drag out page 402 of the old dusty Programming Perl, 3rd ed. again.

Information processing model beginning with perl3 or this software
of UNIX/C-ism.

  +--------------------------------------------+
  |    Text string as Digital octet string     |
  |    Digital octet string as Text string     |
  +--------------------------------------------+
  |       Not UTF8 Flagged, No Mojibake        |
  +--------------------------------------------+

In UNIX Everything is a File
- In UNIX everything is a stream of bytes
- In UNIX the filesystem is used as a universal name space

Native Encoding Scripting
- native encoding of file contents
- native encoding of file name on filesystem
- native encoding of command line
- native encoding of environment variable
- native encoding of API
- native encoding of network packet
- native encoding of database

Ideally, I'd like to achieve these five Goals:

  • Goal #1:

    Old byte-oriented programs should not spontaneously break on the old byte-oriented data they used to work on.

    This goal has been achieved by that this software is additional code for perl like utf8 pragma. Perl should work same as past Perl if added nothing.

  • Goal #2:

    Old byte-oriented programs should magically start working on the new character-oriented data when appropriate.

    Still now, 1 octet is counted with 1 by built-in functions length, substr, index, rindex, and pos that handle length and position of string. In this part, there is no change. The length of 1 character of 2 octet code is 2.

    On the other hand, the regular expression in the script is added the multibyte anchoring processing with this software, instead of you.

    figure of Goal #1 and Goal #2.

                             GOAL#1  GOAL#2
                      (a)     (b)     (c)     (d)     (e)
    +--------------+-------+-------+-------+-------+-------+
    | data         |  Old  |  Old  |  New  |  Old  |  New  |
    +--------------+-------+-------+-------+-------+-------+
    | script       |  Old  |      Old      |      New      |
    +--------------+-------+---------------+---------------+
    | interpreter  |  Old  |              New              |
    +--------------+-------+-------------------------------+
    Old --- Old byte-oriented
    New --- New character-oriented

    There is a combination from (a) to (e) in data, script, and interpreter of old and new. Let's add the Encode module and this software did not exist at time of be written this document and JPerl did exist.

                      (a)     (b)     (c)     (d)     (e)
                                    JPerl,japerl    Encode,Latin3
    +--------------+-------+-------+-------+-------+-------+
    | data         |  Old  |  Old  |  New  |  Old  |  New  |
    +--------------+-------+-------+-------+-------+-------+
    | script       |  Old  |      Old      |      New      |
    +--------------+-------+---------------+---------------+
    | interpreter  |  Old  |              New              |
    +--------------+-------+-------------------------------+
    Old --- Old byte-oriented
    New --- New character-oriented

    The reason why JPerl is very excellent is that it is at the position of (c). That is, it is not necessary to do a special description to the script to process new character-oriented string. (May the japerl take over JPerl!)

  • Goal #3:

    Programs should run just as fast in the new character-oriented mode as in the old byte-oriented mode.

    It is impossible. Because the following time is necessary.

    (1) Time of escape script for old byte-oriented perl.

    Someday, I want to ask Larry Wall about this goal in the elevator.

  • Goal #4:

    Perl should remain one language, rather than forking into a byte-oriented Perl and a character-oriented Perl.

    JPerl remains one Perl language by forking to two interpreters. However, the Perl core team did not desire fork of the interpreter. As a result, Perl language forked contrary to goal #4.

    A character-oriented perl is not necessary to make it specially, because a byte-oriented perl can already treat the binary data. This software is only an application program of byte-oriented Perl, a filter program.

    And you will get support from the Perl community, when you solve the problem by the Perl script.

    Latin3 software remains one language and one interpreter.

  • Goal #5:

    JPerl users will be able to maintain JPerl by Perl.

    May the JPerl be with you, always.

Back when Programming Perl, 3rd ed. was written, UTF8 flag was not born and Perl is designed to make the easy jobs easy. This software provides programming environment like at that time.

Perl's motto

  Some computer scientists (the reductionists, in particular) would
 like to deny it, but people have funny-shaped minds. Mental geography
 is not linear, and cannot be mapped onto a flat surface without
 severe distortion. But for the last score years or so, computer
 reductionists have been first bowing down at the Temple of Orthogonality,
 then rising up to preach their ideas of ascetic rectitude to any who
 would listen.

  Their fervent but misguided desire was simply to squash your mind to
 fit their mindset, to smush your patterns of thought into some sort of
 Hyperdimensional Flatland. It's a joyless existence, being smushed.
 --- Learning Perl on Win32 Systems

 If you think this is a big headache, you're right. No one likes
 this situation, but Perl does the best it can with the input and
 encodings it has to deal with. If only we could reset history and
 not make so many mistakes next time.
 --- Learning Perl 6th Edition

  The most important thing for most people to know about handling
 Unicode data in Perl, however, is that if you don't ever use any Uni-
 code data -- if none of your files are marked as UTF-8 and you don't
 use UTF-8 locales -- then you can happily pretend that you're back in
 Perl 5.005_03 land; the Unicode features will in no way interfere with
 your code unless you're explicitly using them. Sometimes the twin
 goals of embracing Unicode but not disturbing old-style byte-oriented
 scripts has led to compromise and confusion, but it's the Perl way to
 silently do the right thing, which is what Perl ends up doing.
 --- Advanced Perl Programming, 2nd Edition

SEE ALSO

PERL PUROGURAMINGU
Larry Wall, Randal L.Schwartz, Yoshiyuki Kondo
December 1997
ISBN 4-89052-384-7
http://www.context.co.jp/~cond/books/old-books.html

Programming Perl, Second Edition
By Larry Wall, Tom Christiansen, Randal L. Schwartz
October 1996
Pages: 670
ISBN 10: 1-56592-149-6 | ISBN 13: 9781565921498
http://shop.oreilly.com/product/9781565921498.do

Programming Perl, Third Edition
By Larry Wall, Tom Christiansen, Jon Orwant
Third Edition  July 2000
Pages: 1104
ISBN 10: 0-596-00027-8 | ISBN 13: 9780596000271
http://shop.oreilly.com/product/9780596000271.do

The Perl Language Reference Manual (for Perl version 5.12.1)
by Larry Wall and others
Paperback (6"x9"), 724 pages
Retail Price: $39.95 (pound 29.95 in UK)
ISBN-13: 978-1-906966-02-7
http://www.network-theory.co.uk/perl/language/

Perl Pocket Reference, 5th Edition
By Johan Vromans
Publisher: O'Reilly Media
Released: July 2011
Pages: 102
http://shop.oreilly.com/product/0636920018476.do

Programming Perl, 4th Edition
By: Tom Christiansen, brian d foy, Larry Wall, Jon Orwant
Publisher: O'Reilly Media
Formats: Print, Ebook, Safari Books Online
Released: March 2012
Pages: 1130
Print ISBN: 978-0-596-00492-7 | ISBN 10: 0-596-00492-3
Ebook ISBN: 978-1-4493-9890-3 | ISBN 10: 1-4493-9890-1
http://shop.oreilly.com/product/9780596004927.do

Perl Cookbook
By Tom Christiansen, Nathan Torkington
August 1998
Pages: 800
ISBN 10: 1-56592-243-3 | ISBN 13: 978-1-56592-243-3
http://shop.oreilly.com/product/9781565922433.do

Perl Cookbook, Second Edition
By Tom Christiansen, Nathan Torkington
Second Edition  August 2003
Pages: 964
ISBN 10: 0-596-00313-7 | ISBN 13: 9780596003135
http://shop.oreilly.com/product/9780596003135.do

Perl in a Nutshell, Second Edition
By Stephen Spainhour, Ellen Siever, Nathan Patwardhan
Second Edition  June 2002
Pages: 760
Series: In a Nutshell
ISBN 10: 0-596-00241-6 | ISBN 13: 9780596002411
http://shop.oreilly.com/product/9780596002411.do

Learning Perl on Win32 Systems
By Randal L. Schwartz, Erik Olson, Tom Christiansen
August 1997
Pages: 306
ISBN 10: 1-56592-324-3 | ISBN 13: 9781565923249
http://shop.oreilly.com/product/9781565923249.do

Learning Perl, Fifth Edition
By Randal L. Schwartz, Tom Phoenix, brian d foy
June 2008
Pages: 352
Print ISBN:978-0-596-52010-6 | ISBN 10: 0-596-52010-7
Ebook ISBN:978-0-596-10316-3 | ISBN 10: 0-596-10316-6
http://shop.oreilly.com/product/9780596520113.do

Learning Perl, 6th Edition
By Randal L. Schwartz, brian d foy, Tom Phoenix
June 2011
Pages: 390
ISBN-10: 1449303587 | ISBN-13: 978-1449303587
http://shop.oreilly.com/product/0636920018452.do

Advanced Perl Programming, 2nd Edition
By Simon Cozens
June 2005
Pages: 300
ISBN-10: 0-596-00456-7 | ISBN-13: 978-0-596-00456-9
http://shop.oreilly.com/product/9780596004569.do

Perl RESOURCE KIT UNIX EDITION
Futato, Irving, Jepson, Patwardhan, Siever
ISBN 10: 1-56592-370-7
http://shop.oreilly.com/product/9781565923706.do

Perl Resource Kit -- Win32 Edition
Erik Olson, Brian Jepson, David Futato, Dick Hardt
ISBN 10:1-56592-409-6
http://shop.oreilly.com/product/9781565924093.do

MODAN Perl NYUMON
By Daisuke Maki
2009/2/10
Pages: 344
ISBN 10: 4798119172 | ISBN 13: 978-4798119175
http://www.seshop.com/product/detail/10250/

Understanding Japanese Information Processing
By Ken Lunde
January 1900
Pages: 470
ISBN 10: 1-56592-043-0 | ISBN 13: 9781565920439
http://shop.oreilly.com/product/9781565920439.do

CJKV Information Processing
Chinese, Japanese, Korean & Vietnamese Computing
By Ken Lunde
First Edition  January 1999
Pages: 1128
ISBN 10: 1-56592-224-7 | ISBN 13: 9781565922242
http://shop.oreilly.com/product/9781565922242.do

Mastering Regular Expressions, Second Edition
By Jeffrey E. F. Friedl
Second Edition  July 2002
Pages: 484
ISBN 10: 0-596-00289-0 | ISBN 13: 9780596002893
http://shop.oreilly.com/product/9780596002893.do

Mastering Regular Expressions, Third Edition
By Jeffrey E. F. Friedl
Third Edition  August 2006
Pages: 542
ISBN 10: 0-596-52812-4 | ISBN 13:9780596528126
http://shop.oreilly.com/product/9780596528126.do

Regular Expressions Cookbook
By Jan Goyvaerts, Steven Levithan
May 2009
Pages: 512
ISBN 10:0-596-52068-9 | ISBN 13: 978-0-596-52068-7
http://shop.oreilly.com/product/9780596520694.do

Regular Expressions Cookbook, 2nd Edition
By Jan Goyvaerts, Steven Levithan
Final Release Date: August 2012
Pages: 612
ISBN: 978-1-4493-1943-4 | ISBN 10:1-4493-1943-2

JIS KANJI JITEN
By Kouji Shibano
Pages: 1456
ISBN 4-542-20129-5
http://www.webstore.jsa.or.jp/lib/lib.asp?fn=/manual/mnl01_12.htm

UNIX MAGAZINE
1993 Aug
Pages: 172
T1008901080816 ZASSHI 08901-8
http://ascii.asciimw.jp/books/books/detail/978-4-7561-5008-0.shtml

LINUX NIHONGO KANKYO
By YAMAGATA Hiroo, Stephen J. Turnbull, Craig Oda, Robert J. Bickel
June, 2000
Pages: 376
ISBN 4-87311-016-5
http://www.oreilly.co.jp/books/4873110165/

MacPerl Power and Ease
By Vicki Brown, Chris Nandor
April 1998
Pages: 350
ISBN 10: 1881957322 | ISBN 13: 978-1881957324
http://www.amazon.com/Macperl-Power-Ease-Vicki-Brown/dp/1881957322

Windows NT Shell Scripting
By Timothy Hill
April 27, 1998
Pages: 400
ISBN 10: 1578700477 | ISBN 13: 9781578700479
http://search.barnesandnoble.com/Windows-NT-Shell-Scripting/Timothy-Hill/e/9781578700479/

Windows(R) Command-Line Administrators Pocket Consultant, 2nd Edition
By William R. Stanek
February 2009
Pages: 594
ISBN 10: 0-7356-2262-0 | ISBN 13: 978-0-7356-2262-3
http://shop.oreilly.com/product/9780735622623.do

Kaoru Maeda, Perl's history Perl 1,2,3,4
http://www.slideshare.net/KaoruMaeda/perl-perl-1234

nurse, What is "string"
http://d.hatena.ne.jp/nurse/20141107#1415355181

NISHIO Hirokazu, What's meant "string as a sequence of characters"?
http://d.hatena.ne.jp/nishiohirokazu/20141107/1415286729

nurse, History of Japanese EUC 22:00
http://d.hatena.ne.jp/nurse/20090308/1236517235

Mike Whitaker, Perl And Unicode
http://www.slideshare.net/Penfold/perl-and-unicode

Ricardo Signes, Perl 5.14 for Pragmatists
http://www.slideshare.net/rjbs/perl-514-8809465

Ricardo Signes, What's New in Perl? v5.10 - v5.16 #'
http://www.slideshare.net/rjbs/whats-new-in-perl-v510-v516

YAP(achimon)C::Asia Hachioji 2016 mid in Shinagawa
Kenichi Ishigaki (@charsbar) July 3, 2016 YAP(achimon)C::Asia Hachioji 2016mid
https://www.slideshare.net/charsbar/cpan-63708689

CPAN Directory INABA Hitoshi
http://search.cpan.org/~ina/

BackPAN
http://backpan.perl.org/authors/id/I/IN/INA/

Recent Perl packages by "INABA Hitoshi"
http://code.activestate.com/ppm/author:INABA-Hitoshi/

ACKNOWLEDGEMENTS

This software was made referring to software and the document that the following hackers or persons had made. I am thankful to all persons.

Rick Yamashita, Shift_JIS
ttp://furukawablog.spaces.live.com/Blog/cns!1pmWgsL289nm7Shn7cS0jHzA!2225.entry (dead link)
ttp://shino.tumblr.com/post/116166805/1981-us-jis
(add 'h' at head)
http://www.wdic.org/w/WDIC/%E3%82%B7%E3%83%95%E3%83%88JIS

Larry Wall, Perl
http://www.perl.org/

Kazumasa Utashiro, jcode.pl
http://search.cpan.org/~utashiro/
ftp://ftp.iij.ad.jp/pub/IIJ/dist/utashiro/perl/
http://log.utashiro.com/pub/2006/07/jkondo_a580.html

Jeffrey E. F. Friedl, Mastering Regular Expressions
http://regex.info/

SADAHIRO Tomoyuki, The right way of using Shift_JIS
http://homepage1.nifty.com/nomenclator/perl/shiftjis.htm
http://search.cpan.org/~sadahiro/

Yukihiro "Matz" Matsumoto, YAPC::Asia2006 Ruby on Perl(s)
http://www.rubyist.net/~matz/slides/yapc2006/

jscripter, For jperl users
http://homepage1.nifty.com/kazuf/jperl.html

Bruce., Unicode in Perl
http://www.rakunet.org/tsnet/TSabc/18/546.html

Hiroaki Izumi, Perl5.8/Perl5.10 is not useful on the Windows.
http://dl.dropbox.com/u/23756062/perlwin.html
https://sites.google.com/site/hiroa63iz/perlwin

TSUKAMOTO Makio, Perl memo/file path of Windows
http://digit.que.ne.jp/work/wiki.cgi?Perl%E3%83%A1%E3%83%A2%2FWindows%E3%81%A7%E3%81%AE%E3%83%95%E3%82%A1%E3%82%A4%E3%83%AB%E3%83%91%E3%82%B9

chaichanPaPa, Matching Shift_JIS file name
http://d.hatena.ne.jp/chaichanPaPa/20080802/1217660826

SUZUKI Norio, Jperl
http://homepage2.nifty.com/kipp/perl/jperl/

WATANABE Hirofumi, Jperl
http://www.cpan.org/src/5.0/jperl/
http://search.cpan.org/~watanabe/
ftp://ftp.oreilly.co.jp/pcjp98/watanabe/jperlconf.ppt

Chuck Houpt, Michiko Nozu, MacJPerl
http://habilis.net/macjperl/index.j.html

Kenichi Ishigaki, Pod-PerldocJp, Welcome to modern Perl world
http://search.cpan.org/dist/Pod-PerldocJp/
http://gihyo.jp/dev/serial/01/modern-perl/0031
http://gihyo.jp/dev/serial/01/modern-perl/0032
http://gihyo.jp/dev/serial/01/modern-perl/0033

Fuji, Goro (gfx), Perl Hackers Hub No.16
http://gihyo.jp/dev/serial/01/perl-hackers-hub/001602

Dan Kogai, Encode module
http://search.cpan.org/dist/Encode/
http://www.archive.org/details/YAPCAsia2006TokyoPerl58andUnicodeMythsFactsandChanges (video)
http://yapc.g.hatena.ne.jp/jkondo/ (audio)

Takahashi Masatuyo, JPerl Wiki
http://ja.jperl.wikia.com/wiki/JPerl_Wiki

Juerd, Perl Unicode Advice
http://juerd.nl/site.plp/perluniadvice

daily dayflower, 2008-06-25 perluniadvice
http://d.hatena.ne.jp/dayflower/20080625/1214374293

Unicode issues in Perl
http://www.i-programmer.info/programming/other-languages/1973-unicode-issues-in-perl.html

Jesse Vincent, Compatibility is a virtue
http://www.nntp.perl.org/group/perl.perl5.porters/2010/05/msg159825.html

Tokyo-pm archive
http://mail.pm.org/pipermail/tokyo-pm/
http://mail.pm.org/pipermail/tokyo-pm/1999-September/001844.html
http://mail.pm.org/pipermail/tokyo-pm/1999-September/001854.html

Error: Runtime exception on jperl 5.005_03
http://www.rakunet.org/tsnet/TSperl/12/374.html
http://www.rakunet.org/tsnet/TSperl/12/375.html
http://www.rakunet.org/tsnet/TSperl/12/376.html
http://www.rakunet.org/tsnet/TSperl/12/377.html
http://www.rakunet.org/tsnet/TSperl/12/378.html
http://www.rakunet.org/tsnet/TSperl/12/379.html
http://www.rakunet.org/tsnet/TSperl/12/380.html
http://www.rakunet.org/tsnet/TSperl/12/382.html

ruby-list
http://blade.nagaokaut.ac.jp/ruby/ruby-list/index.shtml
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-list/2440
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-list/2446
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-list/2569
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-list/9427
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-list/9431
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-list/10500
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-list/10501
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-list/10502
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-list/12385
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-list/12392
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-list/12393
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-list/19156

Object-oriented with Perl
http://www.freeml.com/perl-oo/486
http://www.freeml.com/perl-oo/487
http://www.freeml.com/perl-oo/490
http://www.freeml.com/perl-oo/491
http://www.freeml.com/perl-oo/492
http://www.freeml.com/perl-oo/494
http://www.freeml.com/perl-oo/514