The London Perl and Raku Workshop takes place on 26th Oct 2024. If your company depends on Perl, please consider sponsoring and/or attending.


Search::Tools::Transliterate - transliterations of UTF-8 chars


 my $tr = Search::Tools::Transliterate->new();
 print $tr->convert( 'some string of utf8 chars' );


Search::Tools::Transliterate transliterates UTF-8 characters to single-byte equivalents. It is based on the transmap project by Markus Kuhn

NOTE: All the is_* encoding check methods that existed in this class prior to version 0.05 were moved to Search::Tools::UTF8 and refactored as functions, many using XS for speed improvements.



Create new instance. Takes the following optional parameters:


Customize the character mapping. Should be a hashref. See map() method.


Allow convert() to use full native 8bit characters for transliterating, rather than only 7bit ASCII. The default is true (1). Set to 0 to disable. NOTE: This must be set in new(). Changing via the accessor after new() will have no effect on map().


Called internally by new().


Access the transliteration character map. Example:

 use Search::Tools::Transliterate;
 my $tr = Search::Tools::Transliterate->new;
 $tr->map->{mychar} = 'my transliteration';
 print $tr->convert('mychar');  # prints 'my transliteration'

NOTE: The map() method is an accessor only. You can not pass in a new map.

convert( text )

Returns UTF-8 text converted with all single bytes, transliterated according to %Map. Will croak if text is not valid UTF-8, so if in doubt, check first with is_valid_utf8() in Search::Tools::UTF8.

convert1252( text )

Returns UTF-8 text converted to all single byte characters, transliterated with convert() and the Windows 1252 characters in the range 0x80 and 0x9f inclusive.

The 1252 codepoints are converted first to their UTF-8 counterparts per using Encoding::FixLatin::fix_latin() and then text is run through convert().

Note that text is checked with the looks_like_cp1252() function from Search::Tools::UTF8 before calling fix_latin().


You might consider the whole attempt as a bug. It's really an attempt to accomodate applications that don't support Unicode. Perhaps we shouldn't even try. But for things like curly quotes and other 'smart' punctuation, it's often helpful to render the UTF-8 character as something rather than just letting a character without a direct translation slip into the ether.

That said, if a character has no mapping (and there are plenty that do not) a single space will be used.


Peter Karman <>

Originally based on the HTML::HiLiter regular expression building code, by the same author, copyright 2004 by Cray Inc.

Thanks to Atomic Learning for sponsoring the development of some of these modules.


Please report any bugs or feature requests to bug-search-tools at, or through the web interface at I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.


You can find documentation for this module with the perldoc command.

    perldoc Search::Tools

You can also look for information at:


Copyright 2006-2010 by Peter Karman.

This package is free software; you can redistribute it and/or modify it under the same terms as Perl itself.


Search::Tools::UTF8, Unicode::Map, Encode, Test::utf8, Encoding::FixLatin