##-*- Mode: Change-Log; coding: utf-8; -*-
##
## Change log for perl distribution DiaColloDB
v0.12.011 Tue, 16 Apr 2019 12:31:38 +0200 moocow
* decode UTF-8 on dcdb-create.perl arguments
* fixed deep recursion typo in Utils::file_timestamp()
v0.12.010 Fri, 10 Nov 2017 10:31:29 +0100 moocow
* fixed native query syntax regex parsing with ddcmode=-1 (fallback)
v0.12.009 Thu, 09 Nov 2017 12:40:16 +0100 moocow
* fixed DDC::Any::Object::__dcvs_compile() fallback calls in Relation/TDF/Query.pm
- bogus parameter conventions were resulting in error messages
'Can't locate object method "toString" via package "DiaColloDB::Relation::TDF::Query"'
and 'no index term(s) matched user query' for simple WITH (&=) queries
v0.12.008 Thu, 22 Jun 2017 11:02:11 +0200 moocow
* trim perl-5.14 character set modifiers from qr()-stringified regexes
- including these breaks KWIC links, since DDC (PCRE) doesn't seem to support them
* die early in dcdb-create.perl if we forgot to specify an -output
v0.12.007 Tue, 13 Jun 2017 12:52:56 +0200 moocow
* added allow_nonref JSON-ification option to list-client 'extend' sub-calls
v0.12.006 Mon, 12 Jun 2017 13:00:09 +0200 moocow
* Document/TCF.pm convenience hack to support embedded TEI header in DTA TCF exports
v0.12.005 Wed, 31 May 2017 12:29:58 +0200 moocow
* Document/DDCTabs.pm fix for ddc_dump v2.1.x (updated DDC:BREAK format)
v0.12.004 Wed, 15 Mar 2017 16:04:39 +0100 moocow
* fix for extend() on list-URLs where all target attributes are missing from DB
* removed some painful global debug code in Profile::Multi::trim()
* made default eps=0 more consistent (dcdb-query.perl, DiaColloDB::profileOptions(), Profile::compile_xyz())
* added Profile::compile_clean() method as workaround for infinite score values (JSON module inserts e.g. '-inf', but browser can't parse it)
* added divide-by-zero checks to Profile::compile_SCORE() methods (for list URLs)
* fixed list-client global trimming
* added DiaColloDB::Client::dbOptions() hack to pass down logXYZ options through rcfile:// and list:// client-URLs
v0.12.003 Wed, 01 Mar 2017 15:47:02 +0100 moocow
* fixed default match-id =2 in ddc groupby
v0.12.002 Mon, 27 Feb 2017 10:44:04 +0100 moocow
* fixed f1 acquisition bug when using DDC back-end for "macro-event" diffs using e.g. -groupby='[@constant]'
v0.12.001 Tue, 24 Jan 2017 15:01:24 +0100 moocow
* made profile parameter N dependent on epoch size (fixes mantis bug #17036)
* adapted Profile::Multi::sumover() to heuristically guess whether to sum sub-profile N for diacollo <= v0.11 compatibility
* reverse-order from DDC::PP Descendants() fixed in DDC::Concordance v0.33
v0.11.002 Wed, 30 Nov 2016 15:38:39 +0100 moocow
* fixed bogus repsitory in Makefile.PL
* fixed meta-attribute parsing "doc.ATTR", "doc.ATTR=VAL" for native query syntax
* fixed buglet calling DDC::Any::Object::new() with temporary $1 argument (quote it to save it)
* TODO: figure out why DDC::PP Descendants() returns in reverse order vs. DDC::XS::Descendants() (--> diacollo item tuple order)
v0.11.001 Tue, 20 Sep 2016 14:14:18 +0200 moocow
* added DiaColloDB::Relation::extend() and wrappers DiaColloDB::extend(), DiaColloDB::Client::extend()
- high-level interface to independent f2 acquisition, e.g. for list-clients
- DDC-relation extend() support still experimental: large batch queries can cause server to choke
* fixed Client::list incorrect f2 acquisition bug
* added support for parallel sub-client processing in Client::list (requires 'forks' module)
* added dcdb-create.perl -lazy option for "lazy union" list-client creation
* renamed 'mi' score function to more accurate 'milf'
* added 'mi1' score function (raw PMI)
- not too useful without pre-filtered corpus: too sensitive to low-frequency outliers
v0.10.009 Thu, 25 Aug 2016 09:51:34 +0200 moocow
* updated Tie::File::Indexed dependency to v0.08
* updated DDC::Concordance dependency to v0.28
v0.10.008 Wed, 24 Aug 2016 14:12:21 +0200 moocow
* merged in debugging changes from v0.10.004 debugging branch (_v0.10.004_0[123])
- conditionally enabled by new dcdb-create.perl -debug option
* added Utils::fh_flush() and Utils::fh_reopen() methods
- fh_reopen() should simulate flush() even on systems which don't support flush()
* updated Persistent subclasses to call fh_reopen() from their flush() methods:
- EnumFile(+FixedLen +FixedMap +MMap), MultiMapFile(+MMap), PackedFile(+MMap)
v0.10.007 Wed, 24 Aug 2016 08:59:10 +0200 moocow
* removed "hard" pdl dependencies, moved to 'recommends'
* fixed default option inheritance for dcdb-create.perl hash-valued options -tdf-option, -option
* added use_ok(DiaColloDB::Upgrade) test: weird errors w/ DDC::Any
* fixed import DiaColloDB::Utils::packsize() in Relation.pm
* fixed native query-parsing for TDF, DDC relations
- direct ddc-parsing can be forced with "[QUERY]" or "(QUERY)"
v0.10.006 Mon, 11 Jul 2016 11:03:01 +0200 moocow
* better version dependency for v5.10.0
v0.10.005 Thu, 07 Jul 2016 14:40:04 +0200 moocow
* replaced DDC::XS query-parsing and -manipulation with DDC::Any from DDC::Concordance >= v0.25
- obviates troublesome Alien::DDC::Concordance dependency
- still only expected to run correctly on *NIX systems due to runtime calls to sort etc.
* commented out DiaColloDB::create() debugging code from v0.10.004_01
v0.10.004_03 Thu, 21 Jul 2016 13:43:49 +0200 moocow
* added dcdb-create -nommap option: see if mmap use in VirtualBox/MacOS is causing errors
v0.10.004_02 2017-07-15 moocow
* debugging test for PackedFile::MMap -- no joy
v0.10.004_01 Tue, 05 Jul 2016 09:27:52 +0200 moocow
* debugging release for un-reproducible 'undefined value' errors on Birmingham data
v0.10.004 Tue, 28 Jun 2016 09:30:42 +0100 moocow
* updated -nofilters option to dcdb-create.perl (alias -use-all-the-data, a la Mark Lauersdorf)
* added DDCTabs 'foreign' option (-dO=foreign=1)
* added (p|w|l)(good|bad)file options to DiaColloDB::create (stoplist files)
v0.10.003 Tue, 21 Jun 2016 15:37:23 +0200 moocow
* added -subclient-option to dcdb-query.perl (common options for list:// sub-clients)
* fixed stringification bug for ddc-diff queries introduced in v0.09.002
'Can't use string ("l") as a HASH ref while "strict refs" in use at DiaColloDB/Relation.pm line 281.'
v0.10.002 Mon, 13 Jun 2016 15:51:42 +0200 moocow
* native query syntax fix: identify CQOr queries and throw an error
v0.10.001 Thu, 12 May 2016 16:57:56 +0200 moocow
* added -log-level option to dcdb-info.perl
* removed dates from generic term-tuple vocabulary ("x-tuples" -> "t-tuples"), a la tdf relation
* changed db structure for more efficient 2-pass Cofreqs queries (f2 bug-fix)
- Cofreqs now 3-level (id1 -> (date -> (id2->f)))
- Unigrams now 2-level (id1 -> (date -> f))
- Relation::subprofile1() and subprofile2() calling conventions changed
- changed temporary file format for "tokens.dat" used by DiaColloDB::create(): added dates
* changed export text file formats
- Unigrams: added dates
- Cofreqs: added dates and un-collocated f1 lines
- "x-tuple" exports replaced by corresponding "t-tuple" exports xenum->tenum, ATTR_2x.*->ATTR_2t, etc.
* added upgrade package v0_10_x2t
- added compatibility wrappers Compat::v0_09::* for transparent use of old indices
* added auto-backup of changed files to upgrade framework
- upgraders are now instantiated as objects, not just packages: cache header & options
* added DiaColloDB::Upgrade::Base::revert() method and -revert option to dcdb-upgrade.perl
- default implementation relies on subclass revert_created() and revert_updated() methods
* added dcdb-upgrade.perl options -keep, -[no]backup
* added DiaColloDB::Utils functions copyto(), moveto(), copyto_a(), cp_a()
* added DiaColloDB::Persistent method-wrappers copyto(), moveto(), copyto_a()
* added optimized PackedFile::MMap::bsearch() method
- for faster v0.10.x Cofreqs 'onepass' mode; still not as fast as v0.09.x 1-pass but it's incorrect anyways
* removed unused methods Cofreqs::f1(), Cofreqs::f12()
* removed obsolete method DiaColloDB::xidsByDate()
* re-factored compatibility wrappers into DiaColloDB::Compat::vX_Y_Z::*
v0.09.004 Tue, 03 May 2016 14:03:13 +0200 moocow
* devel only, no CPAN release
* cofreqs (load|save)TextFh() idempotency tweaks for un-collocated f1
* mmap optimization for Cofreqs::subprofile2(): ca. 26% improvement
* PackedFile dump tweaks: better handling of non-singleton pack formats
* added Utils::packsingle(): better check for singleton pack formats
v0.09.003 Wed, 27 Apr 2016 09:55:14 +0200 moocow
* fixed 'undefined value in vec' warning in DiaColloDB/Relation.pm
v0.09.002 Tue, 26 Apr 2016 15:46:17 +0200 moocow
* fixed comparison profile stringification for new pack()-encoded profiles,
regression for v0.09.001 "f2 bug" fix
v0.09.001 Tue, 26 Apr 2016 14:49:29 +0200 moocow
* fixed double-counting f2 for multiple item1 targets with shared item2 collocates in Cofreqs::subprofile1() 1-pass mode
* added auto-upgrade framework
- DiaColloDB::Upgrade - top-level API
- DiaColloDB::Upgrade::Base - subclass API & defaults
- added subclass ::v0_08_to_v0_09_multimap for v0.09.x multimap format change
- dcdb-upgrade.perl : top-level auto-upgrade script
* added compatiblity mode for multimaps as DiaColloDB::MultiMapFile::v0_08
* fixed -nokeep option to dcdb-create.perl
* TDF union: avoid storage of non-persistent object keys qw(docmeta wdmfile logas reusedir)
* TDF union: fixed 'bus error' resulting from attempt to mmap() temporary data beyond EOF
- arose in dta+dwds trying to include 'pnd' metadata only indexed in dta
- temporary PackedFile tdf.d/mvals_pnd.pf had no entries for dwds data (pnd not indexed)
- readPdlFile(...,Dims=>[$NC]) choked with 'bus error'
* Client::list overhaul
- new default fudge=>10 should be safe (but rather expensive)
- re-factored Client::list::profile() and compare() methods
* improved Client and Client::list documentation
- added "incorrect independent collocate frequencies" section to Client::list documentation
- milder form of this bug applies even to single native CoFreqs indices ("f2 bug", see below)
* workaround for incorrect independent collocate frequency acquisition code in Cofreqs ("f2 bug")
- f2 were computed as marginals only over those (x1,x2,date) triples with f(x1,x2,date) > 0,
rather than over all (*,x2,date \in slice)
- result were in general underestimates of f2
- fix uses 2-pass acquisition strategy, ca. 10x slower for frequent targets (e.g. 'Mann')
~ old subprofile() method refactored into subprofile1() and subprofile2()
- todo: possibly re-factor db structure to use tdf-style {tenum} rather than {xenum},
minimize group-key lookup & optimize for serial cofreqs dba2 file access
- added 'onepass' query option for fast, old, incorrect f2 frequency acquisition (Cofreqs only)
v0.08.006 Thu, 10 Mar 2016 16:52:19 +0100 moocow
* added dbexport() support for TDF relations
* allow option pass-through for Profile::Multi::compile()
* fixed utf8 handling in TDF::qinfo() query templates
v0.08.005 Mon, 07 Mar 2016 10:02:12 +0100 moocow
* fixed pod =encoding typo in Profile.pod
* added 'verbose' option to Profile::(Multi)Diff::saveHtmlFile
- include sub-profile frequencies in diff html output, used by www wrappers if 'debug' flag is set.
* updated module-list and installation sketch in README
v0.08.004 Fri, 04 Mar 2016 13:25:20 +0100 moocow
* remove temporary PDL headers created by DiaColloDB::PackedFile::toPdl(), used by TDF::union()
* fixed buggy Profile::trim() call on undefined (empty) profiles in Profile::Diff::pretrim()
* updated PODs for command-line utilities
* updated & improved API module documentation
v0.08.003 Fri, 26 Feb 2016 15:14:43 +0100 moocow
* added missing PODs to MANIFEST
* added more DiaColloDB::Document subclasses:
- DiaColloDB::Document::JSON - raw JSON dump
- DiaColloDB::Document::TCF - CLARIN-D TCF (attributes {w,p,l} only; metadata from abused <source> element)
- DiaColloDB::Document::TEI - basic TEI-like XML (flexible but slow)
v0.08.002 Tue, 23 Feb 2016 10:51:02 +0100 moocow
* added Document::DDCTabs options trimGenre, trimAuthor
* added explicit PDL dependency in CONFIGURE_REQUIRES + PREREQ_PM: try to be cpantesters-friendly (see RT bug #112321)
* added manual check for PDL in Makefile.PL: disable PDL-Utils/ subdir build if PDL isn't installed
v0.08.001 Fri, 29 Jan 2016 12:35:44 +0100 moocow
* added co-occurrence profiles over (term x document) frequency matrix via DiaColloDB::Relation::TDF
- requires PDL, PDL::CCS, etc.: should be safe to omit, only loaded on demand
* re-worked compile-time filtering; new options to dcdb-create.perl:
-tfmin TFMIN : minimum global term frequency, regardless of DATE component (default=5)
-lfmin LFMIN : minimum global lemma frequency (default=5)
- prunes enums too, which keeps them smaller and speeds up access
v0.07.015 Wed, 04 Nov 2015 14:18:20 +0100 moocow
* added mi3 profiles a la Rychlý (2008)
* report log-log-likelihood scores (extra log() for better scaling)
* singularity checking for log-likelihood computations
v0.07.014 Tue, 03 Nov 2015 11:42:26 +0100 moocow
* added 1-sided log-likelihood ratio profiles a la Evert (2008)
v0.07.013 2015-11-02 12:52:56 +0100 moocow
* fix for Profile::empty(): a profile is empty if it contains no collocates, even if it has nonzero f1
v0.07.012 Wed, 28 Oct 2015 13:04:20 +0100 moocow
* omit {pgood},{pbad} restrictions in Relation::qinfoData()
- these are too expensive for large corpora, resulting in timeouts for KWIC-links
v0.07.011 Tue, 29 Sep 2015 09:10:33 +0200 moocow
* require perl >= v5.10.0 (for // operator)
v0.07.010 2015-09-24 moocow
* moved DDC dependency and include to new CPAN-friendly DDC::Concordance
* updated README
* distcheck fixes
* fixed fill/trim/alignment bug in ddc-diff ('fill' option wasn't being properly honored)
v0.07.009 2015-08-03 moocow
* relation-wise dbinfo
- merged -r 15066:15067 diacollo-0.07.006+vsem into DiaColloDB.pm, DiaColloDB/Relation.pm
v0.07.008 2015-07-31 moocow
* honor {xdmin},{xdmax} in DiaColloDB::xidsByDate()
- fixes 'cannot align non-trivial multi-profiles of unequal size' bug in corpora with bogus dates (e.g. zeitungen)
* ignore Makefile.old
v0.07.007 2015-07-23 moocow
* merged -r15021:15022 branch diacollo-0.07.006+vsem into Relation/DDC.pm
- fix for e.g. author-profiles
* allow ddc queries without primary targets (=1), for 'subcorpus comparison'
* merged -r 15013:15014 diacollo-0.07.006+vsem into DDC.pm
- fixes for pseudo-corpus comparison
v0.07.006 2015-07-20 moocow
* plots/*: pretty diff- and score-function plots
* documented -diff option to dcdb-query.perl
* Profile/Diff.pm pre-trimming tweaks, lavg fix
* doc fixes; lf, lfm score-funcs
* more diff documentation
* added, documented -diff=OP option (adiff,diff,sum,min,max,avg,havg)
v0.07.005 2015-07-08 moocow
* ddc groupby-request parsing tweak
* groupby without token attributes
* ddc tweak for groupby without a token field -- still not working (keys()-queries fail)
v0.07.004 2015-07-02 moocow
* fixed bogus $DiaColloDB::MMCLASS = "DiaColloDB::MultiMapFile::MMap" (not yet written)
* readme fixes
* distribution, docs, readme, htmlifypods
* fix mantis bug #804 : don't trim empty sub-profiles in diff mode
v0.07.003 2015-06-01 moocow
* renamed 'local' profiling option to 'global' (for better web-wrapper transparency and defaults)
v0.07.002 2015-05-29 moocow
* missing profile fix for diff (argh)
* added misc/ddc-sample.txt: notes on #SAMPLE keyword
* merged -r14464:HEAD diacollo-0.06+ddc intro trunk
v0.05.002 2015-04-23 moocow
* reverted trunk to current state of diacollo-0.05.001-pre-vsem branch
* benchmark -iters for dcdb-query.perl
* started trying to add DocClassify-based DSem to DiaColloDB: stuck on questions of modularity
* 'logwhich' option: log multiple sub-classes
v0.05.001 2015-03-24 moocow
* EnumFile fixes for missing keys
* EnumFile::Tied : tied interface to EnumFile
- EnumFile and friends (except for FixedLen::MMap) now allow in-memory cache to override file contents for i2s(), s2i()
v0.05 2015-03-23 moocow
* more verbose union messages
* added wvi-doc2terms.perl: not very encouraging
* woe is me: additive term-identities don't look kosher with word2vec
* work on topic-doc matrix (WAY TOO BIG sentence-based model with k=200)
* word2vec tweaks: a bit further along...
* union tweaks
* union() now uses temporary objects to map attribute indices (ai2u, xi2u)
- should improve memory usage a bit
- individual maps are still loaded to memory on a per-db basis
(at most 1 at any time) in Cofreqs::union and Unigrams::union
* stricter request handling (die on unsupported attributes)
* groupby and generic requests working via web-wrapper
- thought: should we model the query language on ddc (maybe even
use DDC::XS or similar) for max compatibility?
* updated MANIFEST
* parseRequest() for user queries working
* added {maxExpand} option to kludge memory-hogging queries
* factored out parseRequest() from groupby()
+ TODO: implement generic target query using parseRequest() rather than named parameters
* dbinfo for http (add url), list, file, http
* dbinfo, timestamp, disk usage
* remove MYMETA.yml from svn; ignore some other stuff
* EnumFile: more fixes for perl 5.18.2
* more groupby fixes
* attrs/groupby hack for shared arrays
* removed 'use bytes' pragmas almost everywhere
- deprecated in perl 5.18.2 (ubuntu 14.04.1 / kira)
- workaround is to use utf8::encode() and length(), if needed on a temporary
* delete empty records for test-check-enum
* added test-check-enum.perl
* buggy diacollo : taz
v0.04 2015-03-09 moocow
* 'having' filters, wip
* adopt xdmin,xdmax for union
* use lib qw(lib) for update-header
* merged -r r14008:14041 branch diacollo-0.03+attrs intro trunk : compile-time user-defined attributes
v0.03 2015-03-04 moocow
* metadata parsing for Document/DDCTabs.pm
* w2v test functionality now in w2v-compile.perl + w2v-query.perl
* removed cofreqs debugging log stuff
* utf8 parsing mode (improved filter regex matching)
* removed generated Makefile from svn
* tweaks for d* integration
* added dump.mak from old Makefile r13904
* export tweaks
* cofreqs loading tweaks, timing
* union tweaks and woes : seems basically working now
* dump DiaColloDB::Persistent subclass files
- toArray(), fromArray() for PackedFile
- work-in-progress: DiaColloDB::union()
* Client layer working and pretty much tested
* dcdb-query.perl added to MANIFEST
* added dcdb-query.perl : replaces dcdb-(profile|compare).perl
* moved Client/Distributed.pm -> Client/list.pm
v0.02 2015-02-24 moocow
* DiaColloDB/Client/Distributed.pm: error pass-through
* distributed client stuff
- functionality is basically in place, but NOT CORRECT
- getting (fudge*k)-best items from sub-corpora wonks up the
results (e.g. 'gnädig' doesn't appear for Mann vs Frau in
distributed kern), other frequencies and scores are off too
* Diff improvements: trimming via absolute value, add() support
* utf8 tweaks
* DiaColloDB::compare(): basically working ("diff" profiles)
v0.01 2015-02-20 moocow
* initial version