Ben Deutsch
and 1 contributors


Text::Lossy - Lossy text compression


Version 0.40.2


    use Text::Lossy;

    my $lossy = Text::Lossy->new;
    my $short = $lossy->process($long);

    my $lossy = Text::Lossy->new->add('lower', 'punctuation');  # Chaining usage

    $lossy->process($long); # In place
    $lossy->process();      # Filters $_ in place


Text::Lossy is a collection of text filters for lossy compression. "Lossy compression" changes the data in a way which is irreversible, but results in a smaller file size after compression. One of the best known lossy compression uses is the JPEG image format.

Note that this module does not perform the actual compression itself, it merely changes the text so that it may be compressed better.

Alpha software

This code is currently alpha software. Anything can and will change, most likely in a backwards-incompatible manner. You have been warned.


Text::Lossy uses an object oriented interface. You create a new Text::Lossy object, set the filters you wish to use (described below), and call the "process" method on the object. You can call this method as often as you like. In addition, there is a method which produces a closure, an anonymous subroutine, that acts like the process method on the given object.

Adding new filters

New filters can be added with the "register_filters" class method. Each filter is a subroutine which takes a single string and returns this string filtered.



    my $lossy = Text::Lossy->new();

The constructor for a new lossy text compressor. The constructor is quite light-weight; the only purpose of a compressor object is to accept and remember a sequence of filters to apply to text.

The constructor takes no arguments.



    my $new_text = $lossy->process( $old_text );

This method takes a single text string, applies all the selected filters to it, and returns the filtered string. Filters are selected via "add"; see FILTERS.

The text is upgraded to character semantics via a call to utf8::upgrade, see utf8. This will not change the text you passed in, nor should it have too surprising an effect on the output.

If no text is passed in, nothing is returned (the empty list or undef, depending on context). If an explicit undef is passed in, an explicit undef is returned, even in list context.


    $lossy->add( 'lower', 'whitespace' );

This method takes a list of filter names and adds them to the filter list of the filter object, in the order given. This allows a programmatic selection of filters, for example via command line. Returns the object for method chaining.

If the filter is unknown, an exception is thrown. This may happen when you misspell the name, or forgot to use a module which registers the filter, or forgot to register it yourself.



Remove the filters from the filter object. The object will behave as if newly constructed. Returns the object for method chaining.


    my @names = $lossy->list();

List the filters added to this object, in order. The names (not the code) are returned in a list.


    my $code = $lossy->as_coderef();
    $new_text = $code->( $old_text );

Returns a code reference that closes over the object. This code reference acts like a bound "process" method on the constructed object. It can be used in places like Text::Filter that expect a code reference that filters text.

The code reference is bound to the object, not a particular object state. Adding filters to the object after calling as_coderef will also change the behaviour of the code reference.


The following filters are defined by this module. Other modules may define more filters. Each of these filters can be added to the set via the "add" method.


Corresponds exactly to the lc builtin in Perl, up to and including its Unicode handling.


Collapses any whitespace (\s in regular expressions) to a single space, U+0020. Whitespace at the beginning of the text is stripped completely. Whitespace at the end is also collapsed to a single space, to help separate lines. Text consisting only of whitespace results in an empty string.


A variant of the "whitespace" filter that leaves newlines on the end of the text alone. Other whitespace at the end will get collapsed into a single newline. If the text ends in whitespace that does not contain a new line, it is replaced by a space, as before.

This filter is most useful if you are creating a Unix-style text filter, and do not want to buffer the entire input before writing the (only) line to stdout. The newline at the end will allow downstream processes to work on new lines, too. Otherwise, this filter is not quite as efficient as the whitespace filter.

Any newlines in the middle of text are collapsed to a space, too. This is especially useful if you are reading in "paragraph mode", e.g. $/ = '', as you will get one long line per former paragraph.


Strips punctuation, that is anything matching \p{Punctuation}. It is replaced by nothing, removing it completely.


A variant of "punctuation" that replaces punctuation with a space character, U+0020, instead of removing it completely. This is usually less efficient for compression, but retains more readability, for example in the presence of URLs or email addresses.


Leaves the first and last letters of a word alone, but replaces the interior letters with the same set, sorted by the sort function. This is done on the observation (source uncertain at the time) that words can still be made out if the letters are present, but in a different order, as long as the outer ones remain the same.

This filter may not work as proposed with every language or writing system. Specifically, it uses end-of-word matches \b to determine which letters to leave alone.


These methods are not called on a filter object, but on the class Text::Lossy itself. They are typically concerned with the filters that can be added to filter objects.


      change_stuff => \&Other::Module::change_text,
      remove_ps    => sub { my ($text) = @_; $text =~ s{[Pp]}{}; return $text; },

Adds one or more named filters to the set of available filters. Filters are passed in an anonymous hash. Previously defined mappings may be overwritten by this function. Specifically, passing undef as the code reference removes the filter.


    my @filters = Text::Lossy->available_filters();

Lists the available filters at this point in time, specifically their names as used by "add" and "register_filters". The list is sorted alphabetically.


A filter is a subroutine which takes a single parameter (the text to be converted) and returns the filtered text. The text may also be changed in-place, as long as it is returned again.

These filters are then made available to the rest of the system via the "register_filters" function.

USAGE WITH Text::Filter

The Text::Filter module provides an infrastructure for filtering text, but no actual filters. It can be used with Text::Lossy by passing the result of "as_coderef" as the filter parameter.

It is recommended to set Text::Filter to leave line endings alone when using the "whitespace" filter, i.e. the input_postread and output_prewrite should be 0. This is the default for Text::Filter. It will allow "whitespace" to perform its assigned task on line endings.

One thing to note is that the Text::Lossy filters do not follow the Text::Filter's convention that lines "to be skipped" should result in an undef. This means you need to expect completely empty lines (q{}, not even a newline character) in your output. This should be no problem if you print to a file handle or append to a string, but may be surprising if you are filtering an array of lines.


Nothing exported or exportable; use the OO interface instead.


This code strives to be completely Unicode compatible. All filters aim to "do the right thing" on non-ASCII strings. Any failure to handle Unicode should be considered a bug; please report it.


Ben Deutsch, <ben at>


None known so far.

Please report any bugs or feature requests to bug-text-lossy at, or through the web interface at I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.


You can find documentation for this module with the perldoc command.

    perldoc Text::Lossy

You can also look for information at:



Copyright 2012 Ben Deutsch.

This program is free software; you can redistribute it and/or modify it under the terms of either: the GNU General Public License as published by the Free Software Foundation; or the Artistic License.

See for more information.