The Perl Toolchain Summit needs more sponsors. If your company depends on Perl, please support this very important event.

NAME

XS::Framework::Manual::Typemap - XS::Framework C++ typemap API reference

C++ typemaps

Generic typemaps

Typemap is a concept, i.e. C++ struct with certain expectations, described below.

Typemaps are reponsible for transferring objects between perl scripts and C++ code. In the most general case typemap should be a full-specialization of C++ policy class, i.e. stateless class with group of static methods in the xs namespace. The specialization should be for the target C++ class, which objects should be available in perl scripts, e.g.

  namespace xs {

  struct Typemap<MyClass> {
    static [inline] MyClass in(SV* arg) { ... }
    static [inline] Sv out(const MyClass& var, const Sv& proto = {}) { ... }
    static [inline] void destroy (const MyClass&, SV*) { ... }
  }

  } // end of namespace xs

First af all for MyClass value or pointer semantics should be defined. Objects with value semantics are allocated on C++ stack and their destructor is automatically called when object is goes out of scope; it is suitable for short-living transient objects like integer, string etc., and for std::shared_ptr / panda::iptr. A typemap specialization should start as

  struct Typemap<MyClass> { ...

For objects with pointer semantics the only possible option is to return pointer to MyClass and then later, may be, clean it up in destroy method (see below). A typemap specialization should start as:

  struct Typemap<MyClass*> { ...

As the objects with value semantics is usually applicable for light-weight primitive objects structures with negligible construction and destruction costs (including std::shared_ptr), the classes with pointer semantics is the most general case.

  static [inline] MyClass in(SV* arg) { ...
  static [inline] MyClass* in(SV* arg) { ...

in method is responsible for returning C++ object from perl SV* arg. It's intention is to prepare C++ object to work with (external) C++ API, without being encumbered with Perl details. For object with value semantics it usually return the (transient) object copy, e.g. configuration object created from perl Hashes; but in the case of pointer semantics the in method should extract previously constructed object from perl SV*, i.e. return pointer MyClass*. Beware to do not construct object on heap and return pointer to it in the in method, as this leads to memory leak because the XS::framework will not delete the pointer.

  static [inline] Sv out(const MyClass& var, const Sv& proto = {}) { ...
  static [inline] Sv out(const MyClass* var, const Sv& proto = {}) { ...

The out method is responsible for returning Perl wrapper from C++ object to let it be useable from perl scripts. The wrapper should be returned in form of Sv (see XS::Framework::Sv). Usually it should return reference to blessed perl object (newRV), however it might return any other suitable perl SV*, e.g. number, string, or reference to Array/Hash. The proto argument serves as a hint without any restrictions on it's usage or interpretation. In future specializations (see TypemapObject below) it is used as information which final class an object MyClass should be blessed into, i.e. MyPackage::MyClass.

   static [inline] void destroy (const MyClass&, SV*) { }

The <destroy> method is responsible for cleaning up C++ object when Perl's SV* wrapper (returned in out method), is about to be destroyed in Perl: it is not always enough to have MyClass destructor, as destoy method is called the link between C++ and perl is going to be deleted. For transient objects like string, int, date, etc. which do not have some permanent connection with Perl's SV* it is OK to let the method be empty. For pointer-semantics objects it depends: if your XS-code is responsible for object instantiation, then in the sake of resource leaks prevension it should delete it, i.e.

   static [inline] void destroy (const MyClass* obj, SV*) { delete obj; }

If MyClass object was somehow customly constructed and expect it to be customly destructed, that can be done as

  static [inline] void destroy (const MyClass* obj, SV*) {
    obj->dec_refcounter();
    // or GlobalPool::release(obj)
  }

The perl scalar SV* might be used as place to store additional information, assosicated with the concrete C++ object instance, e.g. the used memory pool, then the memory pool can be extracted from SV* and object can be deleted with it.

  static [inline] void destroy (const MyClass* obj, SV* sv) {
    MyPool* pool = ...; // somehow get pool pointer from SV*
    pool->release(obj);
  }

There might be other uses cases. The destroy method is unconditionally inserted for XS-classes in DESTROY at pre-processing step by Parse::XS if there is input typemap. So, the method destroy have to be defined, but it can be empty. The destroy method might be invoked a few times in case of inheritance of XS-adapters, this issues is already solved in TypemapObject.

Let's summarize the for pointer semantics case: in - unwraps / extracs C++ object, out wraps/packs C++ object in perl scalar, destroy deletes Perl and C++ association. A reader might ask the question: where there my C++ object is created? The answer is: the object instantiation left out of typemap scope by design. C++ object instantiation should be done in user's XS-code (aka XS-adapter).

Hovewer, writing typemap specialization for MyClass from scratch is a bit cumbersome because devepoler still have to implement that wrap/unwrap code in using perl API. There are some ready-made and tested patterns, which will help you greatly with that. Go on reading the following sections.

TypemapBase

The TypemapBase is basic implementation of concept TypeMap. It ships with empty destroy method; it is expected that client typemap will inherit from TypemapBase and provide required methods:

  template<>
  struct Typemap<MyClass*>: public TypemapBase<MyClass*> {
    static inline MyClass* in (SV*) { ... }
    static inline Sv out (const MyClass* var, const Sv& proto = {}) { ... }
  };

TypemapObject

  struct Typemap<MyFinalClass>: TypemapObject<MyBaseClass, MyFinalClass, LifetimePolicy, StoragePolicy, CastingPolicy>;

TypemapObject is helper class in creating typemaps, when there is a blessed Perl object and corresponding C++ object, i.e. one object to one object relationship. In the best case all behaviour can be specified via policies without need of additional code writing.

It is expected that custom typemap will inherit TypemapObject, i.e. in common cases it should be enough to have something like:

  template <>
  struct Typemap<geos::geom::Triangle*>: TypemapObject<geos::geom::Triangle*, geos::geom::Triangle*, ObjectTypePtr, ObjectStorageMG, StaticCast>{
    static panda::string_view package() {return "Geo::Geos::Triangle"; }
  };

Template parameters overview

CastingPolicy defines how void* is casted to MyFinalClass. In most of cases the StaticCast should be used as it has zero-runtime costs; in rare cases, i.e. when MyFinalClass participates in virtual inheritance, the DynamicCast should be used. Under the hood the DynamicCast uses panda::dyn_cast which is faster implementation than standard dynamic_cast.

StoragePolicy defines how C++ object pointers can be stored in Perl SVs. It is available both for value-semantics object and pointer-semantic objects, however for value-semantics it might be a little bit tricky to do. Shipped implementations: ObjectStorageIV, ObjectStorageMG, ObjectStorageMGBackref

LifetimePolicy defines how C++ defines convertion rules (between void* and MyClass), defines deletion (see destroy method in generic typemap), upcast (from DerivedClass to BasicClass) and downcast (from BasicClass to DerivedClass) policies. Shipped implementations: ObjectTypePtr, ObjectTypeForeignPtr, ObjectTypeRefcntPtr, ObjectTypeSharedPtr.

MyBaseClass and MyFinalClass are needed to define the most specific and the most generic classes handled via typemap. If no class hierarchy have to be managed (i.e. in Perl only <MyFinalClass> should be visible), then MyBaseClass can be the same as MyFinalClass.

methods

    static [inline] MyClass in(SV* arg) { ... }
    static [inline] Sv out(const MyClass& var, const Sv& proto = {}) { ... }
    static [inline] Sv create (const MyClass& var, const Sv& proto = Sv()) { ... }
    static [inline] void destroy (const MyClass&, SV*) { ... }
    static [inline] void dispose (const MyClass&, SV*) { ... }
    static [inline] MyClass dup (const MyClass& obj) { return obj; }

in(SV* arg) does the same as in typemap, i.e. returns C++ from Perl SV*. It is implemented as: 1) delegate to StoragePolicy to get the void*, and then to LifetimePolicy to convert MyBasicType* and then upcast it to MyFinalType*.

out(const TYPE& var, const Sv& proto = Sv()) does the same as in typemap, i.e. returns Perls SV* from C++ object. In implementation it asks StoragePolicy to wrap/pack MyClass object in Perl SV* and ends up in create method below for classes without BackRefs support.

The create method is similar to out method described above: it is responsible for creating Perl wrapper from C++ object. That distinguish is irrelevant unless MyClass is supposed to support BackRefs; in the last case, i.e. if MyClass is supports BackRefs, if it is possible out returns previously created and cached SV* wrapper, otherwise it calls create method. It is actually a bit more complex, see out behaviour below.

destroy(const TYPE& var, const Sv& proto = Sv()) - does the same as in typemap, i.e. cleans up C++ object when Perls SV* is about to be deleted. It is implemented in safe manner, i.e. real object deletion (via dispose if StoragePolicy) is executed only once, despite the fact that it migth invoked many times from DESTROY method in XS-adapter due to inheritance. If the class is auto-disposable (defined in StoragePolicy), it is actually destroyed via dispose method. In other words, after the method invokation C++ object and Perls SV* might be destroyed.

dispose(const TYPE& var, SV* arg) - does the same as destroy method in typemap; i.e. unconditionally cleans up C++ object and Perl SV*. The actual job is delegated to LifetimePolicy::destroy

package() it is expected to be overriden in user-defined typemap. It should return the string for perl class; it is invoked only when Sv hint argument is not supplied in out or create methods. Unless overriden by user the package() method will throw runtime exception.

dup(const MyClass& obj is called when perl starts another thread, otherwise it is never invoked. It should return MyClass* object pointer, which will be attached to SV* duplicate in the new perl thread. The default behaviour is retain policy, i.e. it returns exatcly the same object as in main thread. This implies siginificant restriction on MyClass: it should be thread-safe and ref-counted; the restrictions cannot be checked at compile time, and if the conditions aren't met Perl interpreter will crash at runtime. Another possible policy is clone, i.e. underlying C++ object should return copy of itself, and that copy will be attached to SV*. The last possible policy is skip, i.e. when undef will be attached to cloned SV*; to implement it <CLONE_SKIP> should be defined in XS-adapter and it should return true.

Wrapping C++ objects into Perl SV* aka storage policy

  ObjectStorageIV
  ObjectStorageMG
  ObjectStorageMGBackref

XS::Framework ships three policies, defining how C++ object pointers can be stored in Perl SVs. They are applicable for both for value semantics and pointer semantics, hovewer for value-semantics it might be a little bit tricky to do, i.e. value have to be stored over pointer.

Storage policy concept

Storage policy concept has the following methods. They are needed only if custom (non-shipped with XS::Framework) policy is supplied.

    template <class TYPEMAP, class TYPE>
    struct ObjectStorage {
        static const bool auto_disposable = false;
        static inline void* get (SV* arg) { ... }
        static inline void set (SV* arg, void* ptr) { ... }
        static Sv out (const TYPE& var, const Sv& proto) { ... }
    };

The auto_disposable property defines, whether the destroy method of TypemapObject should be forwarded to LifetimePolicy::destroy. If underlying perl SV* is able to do self-cleaning and invoke LifetimePolicy::destroy, then this property should be set to true. Otherwise, there is a need of assitance to do proper cleaning custom (i.e. ObjectStorageIV), the property should be set to false.

The get and set methods are responsible for storing/retriving underlying object pointer (downcasted to void*) in/from Perl SV*.

out(const TYPE& var, const Sv& proto = Sv()) does the same as in typemap, i.e. returns Perls SV* from C++ object. If there is no BackRefs support, then it should forward call to TYPEMAP::create, otherwise, if possible, it should extract Backref and return it. Backrefs support is non-trivial topic, see ObjectStorageMGBackref source code to get the ideas.

ObjectStorageIV

ObjectStorageIV relies on the equality sizeof(MyClass*) == sizeof(IV). So, it stores pointer as integer in Perls SV*. This is the most frequently used approach in CPAN, hovewer, in general it is not the best in terms on performance.

This is non-autodisposable policy, i.e. you have to insert DESTROY method in XS-adapter to help it to be forwarded to LifetimePolicy::destoy, otherwise, if there was memory allocations for C++ object, there will be memory leak, as there is no other way to delete underlying C++ object. The actual object deletion is performed in LifetimePolicy::destroy.

As the XS-adapter DESTROY method conforms to general perl rules, it is executed in eval(), which leads to significant slow down on object deletion. Other storage policies do not suffer from that.

There is another restriction of this policy: inheritance. As there is no more free space in underlying perl SV*, the descedant Perl classes can only add methods, but no data, i.e. to store datat the SV* should be upgraded to HashRef or ArrayRef, but that's not possible. The only possible way to overcome that is use inheritance in C++. Due to space restriction there might be issues with multiple inheritance, as there is need to store additional pointer. Other storage policies do not suffer from constraint.

This storage policy has smallest memory footprint and no additional memory allocations, i.e. Perl object creation is the fastest.

ObjectStorageMG

ObjectStorageMG uses Perl Magic to store pointer. It have to allocate additional memory for Perl Magic, hence, object construction is a bit slower; also that Magic occupies some memory, in other words it has some non-zero costs to use that storage policy.

The SV* itself remains free and it can be used in Child classes defined in Perl, i.e.

  package MyPerlClass;

  use base qw/MyCPPClass/;

  sub new {
    my ($class, $data) = @_;
    # construct C++ object and XS-wrapper
    my $obj = SUPER::new($class, $data);
    # extend/upgrade UNDEF to Hash.
    XS::Framework::obj2hv($obj);
    $obj->{extended_field} = $data->{extended_field};
    return $obj;
  }

This is autodisposable storage policy, i.e. custom destruction actions are guaranteed to be invoked via LifetimePolicy::destroy. That's why it is NOT recommended to have DESTOY method in XS-adapter, as it has significant performance costs due to eval context execution.

ObjectStorageMGBackref

Basically it is the same as <ObjectStorageMG> with the abilitity to cache BackRef and return it on demand via out method by the cost of slightly increased memory footprint in perl descendant classes and slightly slower object construction.

Under certain conditions, i.e. when perl object of descendant perl class has been created in perl and frequently accessed from other XS-adapter, the storage out method returns cached object, which is much faster then allocating and initializing perl wrapper.

Storage policy feature matrix

[*1] It allows to use C++ inheritance, of course, and even perl single inheritance, but as it gives not hash, you cannot store additional properties in Perl inheritance, which makes it almost useless.

[*2] For ObjectStorageIV you have to write your own custom destroy method, which is executed in Perl's eval closure, which is quite slow; for ObjectStorageMG* you can (but we do not recommend) write destroy method, which will be the same speed slow too, instead the destroy method can be written in liftetime object policy, which is executed in very fast way as usual C++ object destructror.

[*3] It heavily depends on property access pattern in your program: when (a), the property XS-object or descendent XS-object class (i.e. with perl implementation) has been created from Perl, and (b) is is accessed frequently as property of some other XS-container, then for ObjectStorageMGBackref the accessor gets significant boost, because it always returns cashed object instead of allocating Perl-wrapper for it on each invocation.

LifetimePolicy for managing C++ objects

  ObjectTypePtr
  ObjectTypeForeignPtr
  ObjectTypeRefcntPtr
  ObjectTypeSharedPtr

LifetimePolicy defines how C++ defines convertion, deletion, upcasting and downcasting rules. As there is no need to write custom LifetimePolicy the concept is not described here. Let us know if we are wrong here.

LifetimePolicy is orthogonal to StoragePolicy, i.e. they can be used in any combination independently.

ObjectTypePtr

It assumes that XS-adapter creates object pointer MyClass* on heap and trasfers ownership to Perl. The policy will invoke delete on the object when it's SV* is about to be destroyed.

ObjectTypeForeignPtr

It assumes that XS-adapter returns object pointer MyClass* without transferring ownership to Perl. The deletion policy is empty. It might be useful when singleton object is returned from C++ to Perl.

ObjectTypeRefcntPtr

It assumes that XS-adapter returns <MyClass*> to Perl, i.e. object ownership is shared between C++ and Perl. Another assumption is that MyClass is ref-counted object, i.e. it should support the following operations via ADL (argument dependent lookup):

    void refcnt_inc(MyClass*);
    void refcnt_dec(MyClass*);
    std::uint32_t refcnt_get(MyClass*);

ObjectTypeSharedPtr

It assumes that XS-adapter returns std::shared_ptr<MyClass> to Perl, i.e. object ownership is shared between C++ and Perl.

C++ typemap lifetime policy feature matrix

out behaviour

The out method has optional const Sv& proto = {} parameter, which instructs how make the C++ pointer accessible in Perl.

it might be empty

In that case SV* pointing to undef is created, and it is blessed into package() returned by the TypemapObject.

This is automatically applied behaviour by ParseXS for the output parameters, i.e. ClassXXX below:

    ClassXXX* ClassYYY::create_xxx()

This is also applied when there is a code in xs-adapters like:

    ClassXXX* item = ... ; // get somehow pointer to ClassXXX
    RETVAL = xs::out<>(item);
it might contain Perl package name (in form of string or Stash)

In that case SV* pointing to undef is created, and it is blessed into the specified package name. This is needed to follow standart Perl inheritance model, i.e. respect when the current class might be derived in Perl or xs-adapter. In other words it works similar to:

    package My::XXX;

    sub new {
        my $class = shift;
        my $obj = {};
        return bless $obj => $class;
    }

    package My::YYY;
    use base qw/My::XXX;

    my $obj_a = My::XXX->new;
    my $obj_b = My::YYY->new;
it might be already blessed Perl object

In that case SV* the pointer to C++ object is attached to the existing SV*, the new SV* will not be created and the existing SV* will be returned as result of the operation.

The typical use case is C3-mixin and next::method awareness. It is similar to the perl code

    sub new {
        my $class = shift;
        my $obj = $class->next::method(@_);
        ...
        return $obj;
    }

Ususally in xs-adapter the proto is obtained via

    ..::new(...)
    PROTO = Stash::from_name(CLASS).call_next(cv, &ST(1), items-1);
    RETVAL = new MyClass();
    // ParseXS from XS::Framework will invoke xs::out automatically
it might be something else (ArrayRef or HashRef)

In that it behaves the same as with the empty proto, but instead of using undef as base value to bless the default typemap's package(), it uses the provided object as base to bless. This is similar to

    return bless {} => 'MyClass';  # or:  return bless [] => 'MyClass'

This might be useful when it is known that the class will be extended from Perl, i.e. to avoid XS::Framework::obj2hv invocation in Perl descedant class.

If the default behaviour is not desirable, i.e. there is need to return blessed hashref and tolerate possible descendant classes, then the new method should be like:

    Myclass* Myclass::new(...) {
        RETVAL = new Myclass(...)
        PROTO = Stash::from_name(CLASS).bless(Hash::create());
    }

Beware of the orded above: the C++ class have to be created first, and only then Perl SV* wrapper should be created next. If the order is reversed and C++ class throws an exception in constructor, the C++ object is not constructed, and there is nothing to attach to Perl SV*; but in the SV* desctruction it will be detected (invalid object reference), and the corresponding warning will be shown.

const TypemapObject

It is possible to have type map for some const class, e.g.

  struct Typemap<const MyClass*>: TypemapObject<const MyClass*, const MyClass*, ... >

It has the following sense: if C++ API returns const object, then corresponding Perl SV* wrapper will be marked as read only. Attempt to invoke non-const method on read-only SV* leads to runtime exception.

const-methods in xs-adapters should be marked as const in method attributes section.

Auto-deduced typemaps

Out of the box XS::Framework is shipped with the patrial specializations, when there is no need to write dependend/deduceable typemaps, when there is some basic typemap. For example, typemap<T*> is deduced from typemap<const T*>. This is written as:

    typemap<const T*> -> typemap<T*>

Here are all rules:

    typemap<const T*> -> typemap<T*>
    typemap<T*> -> Typemap<iptr<T>>
    typemap<T*> -> Typemap<iptr<T>&>
    typemap<T*> -> Typemap<T&>

List of predefined typemap:

  int8_t
  int16_t
  int32_t
  int64_t
  uint8_t
  uint16_t
  uint32_t
  uint64_t
  float
  double
  bool
  Simple
  Ref
  Glob
  Array
  Hash
  Stash
  std::string
  panda::string_view
  std::vector<T>
  std::map<Key, T>

GLOSSARY

XS-adapter, XS-wrapper

Perl class written in XS (file with .xs/xsi extension).

BackRef

The instance of Perl object (SV*), created from perl script, but stored outside of Perl (i.e. in C++/C/XS). The tricky part is to temporally "prolong" SV* lifetime in some C/C++ contrainer outside of Perl interpreter.