yapc_bratislava08 - Need help with the perl compiler, emit C or JIT, blabla


  open L<B-C/perloptreeguts.pod> and 


I'll present very briefly about 5% of the internals, will debug some real-life problem and you have enough time to task deeper questions.

* History * Current Status * Compilation * The op tree * B::Bytecode * B::C * Using the compiler * Debugging a problem * Ideas, pro and contra B



The "perl compiler" modules B::Bytecode, B::C and B::CC (B::C compiles B to simple C, B::CC to optimized C source)

written 96-97 by Malcolm Beattie, Oxford.

in core since Alpha 5, perl5.002

Out of core since 5.9.5 (5.10)

Now at CPAN as B::C maintained by me, rurban, at (still as devel)


With alpha3 (1997) it could compile almost the whole perl test suite for all 3 compilers.

With 5.10 it cannot even compile a regexp (due to the 5.10 rewrite and SV changes) and its 21 internal tests (due to advanced features since 5.002: AUTOLOAD, our, ...).

  perl -MO=C,-otest.c
    => test.c
  gcc test.c


Current Status

5.002 up to 5.8.9 works "ok", most tests pass, but not all.

For 5.10 properly calling and saving a lexical context needs some help, most likely from eastern hackers. I have best experiences with russians.

For 5.10/5.11 calling and constructing a regexp for B::C needs help. From demerque in Frankfurt, or someone else who knows how to call PM_SETRE and CALLREGCOMP in XS in 5.10.

When these two problems are solved, I can release it as B-C-1.05 and replace B::C from 5.8.

When the testsuite with some advanced tests will pass, we can start using the compiler and bytecode features. Probably put the bytecode stuff back into core, because we need plc/pmc support and the ByteLoader part builtin.

5.6 and earlier will keep using the core B::C modules, as its internal structures changed too much.



perl has an internal compiler, i.e. a parser (perly.y) reads the source lines and compiles it to a so-called op tree, a tree of simple operators (ops), which are internal pp_() functions.

  See or perloptreeguts.pod

As with XS all internal perl pp_ functions take no arguments, all arguments are expected to be on the "perl stack", which is a special heap area, not the CPU stack. (pp for "push/pop")

The op tree represents the program code, but a program also needs the data, the SV's, AV's and HV's. The arguments for the ops are typically pointers to those SV's (SVREF) or lexicals (on PADs) or direct SV's.

perl is not too much functional, so there are seldom pointers to ops used as args to ops, mostly lexicals and SV's. In perlguts.pod you will see the all the used perl data, the internal variables, the SV's, AV's, HV's, but also CV's, GV's, ... But you will not learn about the op tree, the structure of the ops. The perl compiler is all about the op tree, the ops.

When executing a program, perl compiles ("parses and constructs") the optree and then simply runs linearly through the optree (a linear list now) from the beginning to the end.

In the "perl compiler", the B backend is just the XS representation of the optree as perl objects, you can use perl methods to read from the various OP structs.

The perl compiler consists of various B modules to convert from those B objects, representing the ops, to bytecode or C code.


The op tree

See perloptreeguts.pod (in B::C and the perl5 wiki) and perlhack.pod

Similar to the perl internal variables, the SV's, the OP's are built as hierarchical C structures, based upon the BASEOP, and then more specialized OPs, for the different number of args and types of operations.

  "$a + $b * $c"

is compiled to (in C syntax, but really in memory)

  newBINOP(OP_ADD, flags,
     newBINOP(OP_MULTIPLY, flags, 

Two BINOP's for ADD and MULTIPLY take two args (BINOP), and of those two args are the op for a SVREF (pointer to the SV for $a, $b and $c) and the OP_MULTIPLY.

This parse-tree is recursive and looks like nested LISP code.

The internal compiler (not B::C) runs in three passes over the perl code. The various passes contain also a "peephole" optimizer, which optimizes this recursive op tree and in the end it is ensured that we can linearly run through the tree by simply stepping through the op_next pointers and with lists through the op_sibling pointers.

The Walker:

        while ((PL_op = CALL_FPTR(PL_op->op_ppaddr)(aTHX))) {
        return 0;


Generate the optree from a binary .plc/.pmc file, platform-compatible.

CROSS-PLATFORM PORTABILITY For different endian-ness there are ByteLoader converters in effect. Header entry: byteorder. 64int - 64all - 32int is portable. Header entry: ivsize ITHREADS are unportable.

Needs much less opcodes (~100) than perl, all the pp_ functions (~400). Just for every op, all the op flags (the struct fields) and for every sv/av/hv type.

Assembler and disassembler roundtrips.


Similar to bytecode it generates the whole optree ("code") and data in memory with XS functions, and then jumps into ENTER via the main walker Perl_runops_standard.

But it generates C code, which is statically compiled and linked to libperl. Dynamic perl features are still dynamic, but guaranteed static decisions can be optimized. => B::CC

Using the compiler


  t/ (see the .plc, .asm, .disasm files and 
                the roundtrips)

  t/ 2

Debugging a problem


  t/ 02 

Debug a failure in the PREGCOMP call

Expand the preprocessor C macros to find the actual failing calls.

  gcc -E => .cee

Fix the line number from main() on for the gdb stepper. Our main() is perl_init_aaaa() here.

Step to the problem and inspect it. gdb b perl_init_aaaa p


The B modules can be used the read or change or transform the perl optree - a perl program in the internal representation.

We might want to convert perl5 to various other formats, such as native code (JIT), perl6 or PIR, but maybe also to java, LISP, scheme, and compile this then to fast native and optimized code.

Other possible advanced ways are:

1. PPI, the perl source level parser together with a source filter, which could be used for source level macro trickery.

2. MAD, compiles the optree externally to XML or YAML, and offline tools can convert these XML to other formats, such as perl6 code. Advantage: This looks awful, but is easily debuggable.

3. undump() and unexec

Some cool B modules are B::Concise, B::Deparse, B::Lint, B::Generate.

And as advanced modules Devel::TypeCheck, optimize, optimizer, types. These are the building blocks for a statically optimized compiler (as B::CC), in contrast to the current slow, dynamic interpreter.

-- rurban Bratislava, 2008-11-08 16:05 MET

__END__ Local Variables: fill-column:65 End: