/ 
Scalar-ValueTags-LineageTracking-0.002
/ Scalar::ValueTags::LineageTracking

NAME

Scalar::ValueTags::LineageTracking - track data lineages using ValueTags

SYNOPIS

# setup: choose type of data to store in tags
my $dt = Scalar::ValueTags::LineageTracking->new( tag_type => 'string' );
my $dt = Scalar::ValueTags::LineageTracking->new( tag_type => 'ref' );

# default tag_type: string
my $dt = Scalar::ValueTags::LineageTracking->new();

### append tracking data sources when data is received from external sources
my $names = $dbh->selectcol_arrayref('select name from user where user_id = %s", undef, $id);
my $name = $names->[0];

# using 'ref' tag_type
$dt->add_data_source(\$name,
    { table => 'user', column => 'name', id => $id, file => __FILE__, line => __LINE__ }
);

# using 'string' tag_type
my $json = JSON->new->utf8->canonical->pretty(0);
$dt->add_data_source(\$name,
    $json->encode( { table => 'user', column => 'name', id => $id, file => __FILE__, line => __LINE__ } )
);

### clear previous data sources and set data source
$dt->set_data_source( \$var, { table => 'user', column => 'name' } ); # 'ref' tag_type
$dt->set_data_source( \$var, $json->encode( { table => 'user', column => 'name' } ) ); # 'string' tag_type

### data sources are propagated any time data is used
$dt->add_data_source(\$name, 'tag-one');
$dt->add_data_source(\$honorif, 'tag-two');
my $salutation = "Hello, $honorif $name";

# returns data sources set on both $honorif and $name:
my $data_sources = get_data_sources(\$salutation);

### retrieve and clear data sources, for reporting
my $sources = $dt->get_and_clear_data_sources(\$salutation);
send_lineage( { sources => $sources, var => 'salutation', file => __FILE__, line => __LINE__ } );

DESCRIPTION

This module uses Scalar::ValueTags (propagated value magic) to implement data flow tracking that can be used to populate data lineage systems such as OpenLineage.

LineageTracking allows arbitrary tracking data to be attached to data values when the values are received from an external system, propagates the tracking data any time that other data is derived from the original data, and captures the tracking data from the value when it is sent to an external sink.

The tracking data may be either an unblessed Perl reference or a serialized data string, depending on the tag_type parameter.

When the tracking items are propagated, they are de-duplicated using either the string or the refaddr of the data structure that was set in set_data_source.

The tracking data may be formatted in any way, such as OpenLineage.

Implementation

LineageTracking uses the SVTAGS_UNIQUE_REF_ARRAY behavior, which allows arbitrary references to data structures or serialized data strings to be added to the value magic, and de-duplicates the structures when propagating them to subsequent data values. If the data structure references are used, they are de-duplicated by refaddr of the structure.

FUNCTIONS

set_data_source

# using string
set_data_source( \$var, JSON->encode( { this => 1, that => 2 } );

# using Perl reference
set_data_source( \$var, { this => 1, that => 2 } );

Set the data source for the value of the given $var to be the given data structure.

get_data_sources

$data_sources = get_data_sources(\$var);
for my $data_source (@$data_sources) { ... }

Returns arrayref of all data sources inherited by value in given $var.

clear_data_sources

clear_data_sources(\$var);

Clears all data sources on value in given $var.

export_and_clear_data_sources

export_and_clear_data_sources( \$var, { sink => 'database', table => 'user', id => $id } );

Captures and sends the tracking data from $var, adding the given metadata to indicate where the data is being exported from.

LICENSE

This module is released under the same terms as Perl itself.

AUTHORS

  • Noel Maddy <zhtwnpanta@gmail.com>