NAME

geo-latlon2place-makedb - generate database for use with Geo::LatLon2Place

SYNOPSIS

geo-latlon2place-makedb [OPTION]... inputfile.txt outputfile.cdb

DESCRIPTION

abc

OPTIONS AND ARGUMENTS

geo-latlon2place-makedb requires two arguments: a text file with geo data and the name of the database file to be written.

By default, the input file is considered to be in geonames gazetteer format, but this can be customized using --extract.

--cellsize km (default 20, or 10 for geonames-postalcodes)

The (minimum) size of a single grid cell in km - the data is binned into cells of at least this size. It should not be larger than the search radius.

--extract geonames|geonames-postalcodes|perl...

The extraction method: the default is geonames, which expects a geonames database (https://download.geonames.org/export/dump/, for example DE.txt, cities500.txt or allCountries.txt) and extracts placename, countrycode strings from it.

The method geonames-postalcodes does the same, but for a geonames postal code database https://download.geonames.org/export/zip, and extracts zip name, countrycopde strings.

Lastly, you can specify a perl fragment that implements your own filtering and extraction.

FILTERING AND EXTRACTION

Record selection and the returned data are not fixed - you can filter your records yourself and associate any (reasonably short, the maximum is a bit above 200 octets) binary blob with a coordinate pair.

To do this, you have to provide a perl fragment that extracts latitude, longitude, a weight and the associated data blob from an input line stored in $_. The file is opened using the :perlio layer, so if your input file is in UTF-8, so will be $_.

For example, the following would expect an input file with space separated latitude, longitude, weight and name, where name can contain spaces, which is useful when you want to provide your own input data:

   geo-latlon2place-makedb --extract 'chomp; split / /, 4' input output

A slighly more verbose example expecting only latitude, longitude and a name would be:

   geo-latlon2place-makedb --extract '
      chomp;
      my ($lat, $lon, $name) = split / /, 4;
      ($lat, $lon, 1, $name)
   ' input output

If you want to skip certain lines without adding anything to the database, you can return nothing:

   geo-latlon2place-makedb --extract '
      chomp;
      my ($lat, $lon, $name) = split / /;
      return unless $lat < 0; # only add southern hemisphere points
      ($lat, $lon, 1, $name)
   ' input output

In general, the fragment should return either an empty list, or a four-tuple with decimal latitude, longitude, a weight (integer 0..255) and the binary data to be associated with the coordinate. Other than the weight, these should be self-explaining. The weight is used during search and will be multiplied to the square of the distance, and is used to make larger cities win over small ones when the coordinate is somewhere between them.

The standard extractors (geonames and geonames-postalcodes) provide a UTF-8-encoded string as blob, but any binary data will do, for example, if you want to associate your coordinate pairs with some short-ish integer codes, you could do this:

   geo-latlon2place-makedb --extract '
      chomp;
      my ($lat, $lon, $id) = split / /, 4;
      ($lat, $lon, 1, pack "w", $id)
   ' input output

And later use unpack "w" on the data returned by lookup.

The geonames filter looks similar to the following fragment, which shows off some more filtering possibilities:

   my ($id, $name, undef, undef, $lat, $lon, $t1, $t2, $cc, undef, $a1, $s2, $a3, $a4, $pop, undef) = split /\t/;

   return if $t1 ne "P"; # only places

   # minimum population 200, or it is a populated place with no known population
   $pop => 200
      or ($pop eq "" and $t2 eq "PPL")
      or return;

   # skip certain places we are not interested in
   return if $t2 eq "PPLX"; # section of populated place
   return if $t2 eq "PPLW"; # destroyed populated place
   return if $t2 eq "PPLH"; # historical populated place
   return if $t2 eq "PPLQ"; # abandoned populated place

   # geonames has a lot of very long place names which aren't
   # actually place names, so ignore very long names
   60 > length $name
      or return;

   # we estimate a weight by dividing 25 by the radius of the place,
   # which we get by assuming a fixed population density of 5000 # people
   # per square km, # which is almost always a considerable over-estimate.
   # 25 and 5000 are pretty much made-up, feel free to improve and
   # send me the results.
   my $w = 25 / (1 + sqrt $pop / 5000);

   # administrative centers get a fixed low weight
   if ($t2 =~ /^PPLA(\d*)/) {
      $w = $1 || 1;
   }

   ($lat, $lon, $w, "$name, $cc")

AUTHOR

Marc Lehmann <schmorp@schmorp.de>