Changes for version 5.24.0 - 2026-09-05
- [BINARY RECORD SERIALIZATION ARCHITECTURE (ABR v5)] Native Pure Perl Binary Serialization:
- Introduced high-performance native pure Perl binary record serialization format (ABR v5 / Format 5) replacing legacy delimiter and regex text serialization (db_encode / db_decode), aligning format versioning with historical eras (v1: 2003 FlatDB, v2: 2005 \T, v3: 2021 <TAB>, v4: 2026 HTML entities, v5: 2026 ABR Binary).
- Zero CPAN Dependencies: Built strictly on core built-in Perl primitives (pack, unpack, substr, vec), completely eliminating version brittleness and security vulnerabilities associated with external serializers like Storable.
- Magic Header Architecture: Prefixes binary records with a 5-byte magic sequence (\x00ABR\x05); null-byte prefix guarantees zero collision with legacy plain text or data strings.
- Schema Type Coverage: Transparently maps all 9 AmberDB schema types into 1-byte typed nodes: UNDEF (0x00), SCALAR_RAW (0x01), SCALAR_UTF8 (0x02, via lossless utf8::encode/decode), ARRAY (0x03, 16-bit Big-Endian count), and HASH (0x04, 16-bit Big-Endian count).
- Nested Data Structures: Fully supports arbitrarily nested arrays, hashes, and repeat blocks with strict recursion depth guarding ($depth <= 32) to prevent stack overflow or circular reference hangs.
- Transparent Legacy Fallback: db_decode automatically falls back to _db_decode_legacy for non-ABR records, allowing mixed-version legacy tables to operate without downtime.
- Benchmarks: Achieves ~130,000 encodes/sec (+150% faster) and ~69,000 decodes/sec (+64% faster) on flat records; achieves ~4,300 decodes/sec (+35% faster) on complex deeply nested multi-level records.
- [BINARY INDEX ARCHITECTURE REFACTORING] Complete Migration of All Indexes to 8-Byte Packed Binary Buffers:
- Migrated all secondary index subsystems (.inx, .fld, .src, .fac, .slg, and Tier B Junk .jinx, .jfld, .jsrc) to pure 8-byte fixed-width packed binary buffers (pack "Q>", unpack "(Q>)*").
- Core Binary Primitives in AmberDB::Base: Implemented bin_add, bin_punch, bin_sort, bin_find, and bin_count operating directly on raw byte buffers via substr() and memory-aligned index(), achieving C-level execution speed.
- High-Level Cleanup: Completely eradicated high-level Perl array/hash manipulations (array_nodup, array_punch) from the core engine indexing path.
- Consolidated Pre-Sorted Indexing: Standalone .srt files are formally deprecated and eliminated; sort indexes are directly maintained inside .inx.
- Direct 64-bit Uint Indexing: Non-foreign key numeric fields in .fld bypass synthetic dictionary ID generation, indexing pure 64-bit unsigned integers directly into binary keys.
- Batch Foreign Key Pre-Fetching: Integrated batch RDBM ID pre-fetching in search and junk indexing pipelines, reducing disk I/O overhead to zero during keyword tokenization.
- [LEGACY TABLE MIGRATION & RECONSTRUCTION ENGINE] Automated update_table & update_all in AmberDB::Tools:
- Multi-Era Format Detection: Implemented _detect_record_format and decode_legacy_record recognizing and decoding all historical formats across AmberDB history: 2003 FlatDB (v1), 2004-2006 \T arrays (v2), 2019-2025 <TAB0>..<TAB3> hierarchical tabs (v3), 2026 HTML entities (v4), and modern ABR v5 (v5).
- Automated Timestamped Backups: Automatically backs up migrating tables as <table_name>-v<detected_ver>-<YYYY-MMDD>.db prior to rewriting.
- Authoritative Data Preservation:
- .unq (Unique & Synonym Dictionary): Identified as non-reconstructible authoritative master data; strictly exempted from derived index cleanup, preserved live, and snapshot-copied to <table_name>-v<ver>-<date>.unq.
- .del (Soft-Deleted Records Archive): Detected via exist_table($table, 'del'), backed up, and all archived records migrated to ABR v5.
- .aut (Audit Trail Log): Detected via exist_table($table, 'aut'), backed up, and user modification history migrated to ABR v5.
- .cnt (Read Counter): Detected via exist_table($table, 'cnt'), backed up, and live counter state preserved.
- Clean Re-indexing via insert_list: Cleans up all derived and legacy indexes (qw(inx src fld fac slg srt jinx jfld jsrc)) and rebuilds table data and indexes atomically through $adb->insert_list.
- [COMMAND LINE UTILITY] Automated Migration Script (bin/update_tables.pl):
- Added CLI migration script bin/update_tables.pl supporting --all, --table=<names>, --dbase=<dir>, --force, and --help options.
- Provides granular per-table progress reporting and comprehensive post-migration summary (tables processed, upgraded, already up-to-date, record counts, and companion file backups).
- [TRANSACTION SAFETY, SEARCH & CSV COMPATIBILITY]
- Transact Journal Escaping: Escaped \n, \r, \x1e, and \\ in binary record payloads logged to .txn journals, preventing binary length bytes from splitting WAL records or corrupting field boundaries.
- CSV Line Preservation: In tie2csv and vacuum, exported CSV records using _db_encode_legacy so exported files remain clean single-line human-readable text.
- Unindexed Search Word Extraction: Fixed search_table in unindexed mode to decode record fields before passing them to get_words, preventing binary length bytes from corrupting search tokens.
- [QUERY ENGINE & INDEX PERFORMANCE OPTIMIZATIONS] Adaptive Binary Intersect & Pure-Index Candidate Filtering:
- Adaptive Binary Search Pruning (bin_crop): Introduced dynamic thresholding between XS unpack probing and 8-byte aligned O(log N) binary search (substr). For massive posting lists (e.g. 600K records), eliminates huge Perl scalar allocations and byte-alignment collision issues, dropping multi-field pruning from 4.1 ms to 0.058 ms (58 us).
- Pure-Index Candidate Probing (search_table): Refactored multi-value/range filtering (e.g. 27-year date intervals) to evaluate candidate IDs directly against sorted 8-byte aligned index buffers via binary search, completely eliminating slow inverted unioning, zero .db reads, and zero massive 50,000-element Perl hash allocations.
- Batch Unindexed Fallback: Replaced iterative table_readid loops in unindexed search fallbacks with single-pass read_list batching, eliminating repetitive open/close file descriptor syscalls.
- [TEST COVERAGE & PACKAGING]
- Added comprehensive test suite t/amberdb_update_table.t covering multi-era format decoding, update_table migration, backup naming, .del/.aut/.cnt/.unq handling, and update_all batch discovery.
- Updated MANIFEST to include bin/update_tables.pl and new test suites.
- All 47 test files (440 assertions) passing with 100% success rate.
Modules
High-performance embedded NoSQL database engine for Perl
Array, matrix manipulation, and set operations utility
Core serialization, schema resolution, binary packing, file locking, and low-level I/O base class
Native .db and .inx RAM-Disk (tmpfs) unified cache and persistent staging buffer engine
Date manipulation, chronological ID generation, range calculation, and formatting utility
Inverted search, exact field match, binary sort, and URL slug rewrite indexing engine
Multilingual text processing, collation, number/currency formatting, and search normalization engine
ISO 4217 Currency Definition and Symbol Dictionary
String manipulation, HTML conversion, content detection, and sanitization utility
Database maintenance, CLI reindexing, and bulk conversion toolset
ACID-compliant transactions with Strict Two-Phase Locking (Strict 2PL) and undo journaling engine