Lingua::EN::Segment - Split English-language domain names etc. into words


 my $segmenter = Lingua::EN::Segment->new;
 for my $domain (<>) {
     chomp $domain;
     my @words = $segmenter->segment($domain);
     print "$domain: ", join(', ', @words), "\n";


Sometimes you have a string that to a human eye is clearly made up of many words glommed together without spaces or hyphens. This module uses some mild cunning and a large list of known words from Google to try and work out how the string should be split into words.


 Out: $segmenter

Returns a Lingua::EN::Segment object.


 Out: $dist_dir

Returns the name of the directory where distribution-specific files are installed.


 In: $unsegmented_string
 Out: @words

Supplied with an unsegmented string - e.g. a domain name - returns a list of words that are most statistically likely to be the words that make up this string.


 Out: \%unigrams

Returns a hashref of word => likelihood to appear in Google's huge list of words that they got off the Internet. The higher the likelihood, the more likely that this is a genuine regularly-used word, rather than an obscure word or a typo.


 Out: \%bigrams

As "unigrams", but returns a lookup table of "word1 word2" => likelihood for combinations of words.


This code is based on chapter 14 of Peter Norvig's book Beautiful Data.