Changes for version 5.26.0 - 2026-09-23
- [FAST INDEXED FIELD COUNTING (field_count)]
- Implemented field_count($table, $block, $val) for fast O(1) indexed counting across inverted match (.fld) indexes without reading or deserializing full records.
- Optimized for high-throughput facet counts, inventory status aggregation, and relational child counts.
- Added comprehensive unit test suite in t/amberdb_field_count.t and documentation in wiki/Method-field_count.md and wiki/TR-Method-field_count.md.
- [COLUMNAR FACETS & BIDIRECTIONAL DICTIONARY LOOKUP]
- Enhanced AmberDB::Base::Facet facet counting, facet menu generation, and tiered active/junk rule separation.
- Optimized .unq bidirectional string dictionary mapping between text labels and compact numeric dictionary IDs.
- [RELATIONAL PARENT/CHILD TABLES & MATCH INDEXING]
- Enforced transparent child table RAM-disk cache invalidation (_invalidate_child_ramdisk_tables) when parent_table receives writes or deletes.
- Validated match_block index construction and field_fetch O(1) lookups for child tables linked by parent_table (e.g., member_google).
- [ENCODER & BINARY BUFFER OPTIMIZATIONS]
- Optimized binary crop routines (t/amberdb_bin_crop.t) and range filtering algorithms in AmberDB::Base::Encoder and AmberDB::Array.
- [TOOLS REFACTOR & SUB-MODULE DECOMPOSITION (AmberDB::Tools)]
- Decomposed monolithic AmberDB::Tools (~2350 lines) into modular sub-modules under lib/AmberDB/Tools/:
- AmberDB::Tools::Index: Primary keys, full-text inverted search, columnar facets, match blocks, sort matrices, and URL slug indexing (set_index, set_readall, set_search, set_fields, set_filters, set_rwlnkall, set_sort, index_alltables, check_readall, check_search).
- AmberDB::Tools::Maintain: Database maintenance, backup dump/restore, compacting, table deletion, and TSV/CSV conversion (dump, restore, vacuum, del_table, tie2csv, csv2tie, dir_tables, all_tables, _copy_file).
- AmberDB::Tools::Update: Table upgrades, schema migrations, and centralized update routines (update_table, update_all, replace_tablename, replace_blockdata, db_simple, convert_tables, update_storage, update_version).
- Preserved AmberDB::Tools as the unified facade wrapper inheriting from all sub-modules (use parent qw(...)) with 100% backward compatibility.
- Enforced strict zero-POD rule across all sub-modules; all user-facing documentation is consolidated in AmberDB::Tools.
- Decomposed monolithic AmberDB::Tools (~2350 lines) into modular sub-modules under lib/AmberDB/Tools/:
- [CENTRALIZED CLI STORAGE & VERSION UPDATES (amberdb_cli.pl)]
- Integrated 'amberdb update storage' and 'amberdb update version' into bin/amberdb_cli.pl (and amberdb/amberdb.bat wrappers).
- Added CLI option parsing for --check, --force, --all, --no-backup, --tables, and --cpanm flags.
- Fully eliminated legacy bin/amberdb_setup.pl; all maintenance and provisioning tasks are routed through the main amberdb CLI.
- Registered bin/amberdb and bin/amberdb_cli.pl in Makefile.PL EXE_FILES.
- [RAM-DISK ZERO-FALLBACK & SHARED MEMORY ARCHITECTURE]
- Re-engineered RAM-disk mounting lifecycle using platform table mapping (%RM_DIR: Windows R:, Linux /dev/shm, macOS /Volumes).
- Fixed Windows ImDisk detection in AmberDB::Base::Ramdisk by appending a trailing slash ($rm_base/) in -d tests to resolve MSYS2/Windows drive root stat quirks.
- Enforced strict Zero-Fallback architecture: when unmounted or disconnected, ramdisk_dir, table_rdir, schema_rdir, and config_rdir are normalized strictly to "" (empty string) in both $cfg and {_path}.
- Completely removed shmem_dir and any fallback creation of dbstore/ramdisk directories on physical disk when unmounted; tables and routines never assume a RAM-disk is present.
- Removed table_rdir, schema_rdir, and config_rdir from default persistent @dirs initialization array in AmberDB::Base::set_datadir.
- Implemented pure RAM-disk Shared Memory API: get_shmem(), set_shmem(), del_shmem() with strict zero-fallback to disk when unmounted.
- Simplified Windows batch utility bin/setup_windows.bat using native imdisk directly, deleting obsolete shell and PowerShell helper scripts.
- Handled explicit AMBERDB_TEST_RAMDISK=0 as deterministic unmounted test override in AmberDB::Base::Ramdisk.
- Optimized ramdisk_is_mounted() for fast O(1) cached lookup.
- [TIERED STORAGE SYNCHRONIZATION & READ/WRITE CONSISTENCY (R0 - R4)]
- R1 (Tier 1 - Hybrid Storage: Primary Data on Disk, Indexes on RAM-Disk):
- Synchronous index dual-write: index_put(), index_del(), and recs_put()/recs_del() (for non-.db index targets like .slg, .fld, .inx, .fac, .src) now automatically write to both RAM-disk and physical disk.
- Guarantees complete index durability and survival across system reboots and power outages while preserving sub-millisecond query execution directly from RAM-disk.
- Primary data files (.db) remain strictly on physical disk without mirroring to RAM, eliminating memory bloat.
- R4 (Tier 4 - Write-Behind Async Storage):
- Fixed multi-record retrieval and query functions (read_list, read_all) and field scanning/filtering (field_fetch, field_filter, field_count, field_scan, search_table, table_keys) to include use_ramdisk == 4 in $data_path resolution.
- Guarantees immediate read-your-own-writes consistency directly from the active RAM-disk before the asynchronous background flush daemon writes dirty records to physical disk.
- R1 (Tier 1 - Hybrid Storage: Primary Data on Disk, Indexes on RAM-Disk):
- [STORAGE DIRECTORY LAYOUT & TEST SUITE ENHANCEMENTS]
- Modernized xt/amberdb_ramdisk.t and t/amberdb_ramdisk_tiers.t to completely eliminate all legacy "ramdisk" folder references, hardcoded subfolder paths, and regex expectations.
- Upgraded all test instances to pass explicit database names and isolated RAM-disk mounts ($mock_mount/amberdb_<dbname>).
- Hardened AmberDB::Base::Ramdisk test emulation mode ($ENV{AMBERDB_TEST_RAMDISK}) to strictly use temporary directories instead of defaulting to the host's physical RAM-disk (R:), completely preventing test artifacts from polluting production RAM-disks.
- Completely eliminated the legacy "$dbase_dir/ramdisk" fallback in AmberDB::Base::Ramdisk; when no database name is provided in production, RAM-disk mounting is strictly disabled (zero-fallback).
- Fixed test emulation fallback in AmberDB::Base::Ramdisk when database name is omitted under $ENV{AMBERDB_TEST_RAMDISK}, defaulting to isolated tmpdir()/amberdb_test to ensure clean schema inheritance testing (t/amberdb_dbase_inheritance.t) without host R: pollution.
- Updated AmberDB::Tools::Update::update_storage to exclude the obsolete 'ramdisk' folder from canonical directory layouts.
- Purged all orphaned physical dbstore/ramdisk directories across active projects.
- Standardized lock directory resolution in t/amberdb_lock.t using $adb->path('lock_dir').
- Added comprehensive R1 dual-write synchronization assertions in xt/amberdb_ramdisk.t verifying .inx, .fld, and .slg physical persistence.
- [CONNECT PARAMETER NORMALIZATION]
- Supported 'db' parameter alias in AmberDB constructor alongside database, dbase, and dbname.
- [DYNAMIC FIELD NORMALIZATION ENGINE (.norm)]
- Implemented field_normalize() and norm_info() in AmberDB::Base::Schema for domain-agnostic feature extraction from unstructured raw text strings.
- Normalization rules reside in dedicated <table_id>.norm schema files, eliminating runtime overhead for ordinary read/query workloads.
- Supports pipeline stages: preprocess boundary regexes, categorical mapping, single matches (integer, titlecase, trim, code callbacks), and multi-matches (min/max selection).
- Optional schema block array mapping via { as_array => 1 }.
- Registered _norm container in AmberDB core constructor with lock_keys and lock_value protection.
- Added comprehensive unit test suite in t/amberdb_field_normalize.t.
- [INDEX INTEGRITY & STRICT NUMERIC ID REINDEXING]
- Enforced strict positive integer filtering (/^\d+$/ && $_ > 0) across all binary index buffers (keys, A:keys, B:keys, $blk:keys, bin_encode, bin_decode, bin_add, bin_punch), completely preventing non-numeric or non-positive IDs from entering .inx files.
- Updated set_index in AmberDB::Tools::Index to close and unlink all existing index files (.inx, .src, .fld, .fac, .slg, .unq) on both physical disk and RAM-disk before rebuilding, ensuring clean and deterministic recreation.
- Added RAM-disk synchronization to set_index: freshly rebuilt index files are atomically mirrored to RAM-disk upon completion with safe filehandle management.
- Fixed UTF-8 double-encoding in $blk:vals across sort_add, sort_modify, and sort_del in AmberDB::Base::Index to prevent exponential byte bloat on multibyte character updates.
- Enhanced amberdb_cli.pl with database name auto-detection from connect.pl, core.conf (site_id), and project directories, ensuring CLI maintenance commands attach to active RAM-disk instances.
- Disallowed generic 'dbstore' as a valid database name across Ramdisk and CLI modules to eliminate unintended directory creation on RAM-disk.
- Enforced strict CLI connect argument validation: positional argument after 'connect' is strictly the database name (amberdb connect <database_name> [user] [pass]). Explicitly rejected 'connect dbstore' and directory paths with a descriptive error directing users to use --db=<dir> for custom data directories.
- Added non-numeric key and record ID enforcement in index_put(): when $type is not 'raw', non-numeric IDs in array references are filtered out and non-numeric scalar values are rejected from entering index files (.inx, .src, .fld, etc.).
- [FAST RANGE FILTER ENGINE]
- Implemented range_slice() using pre-sorted .inx keys ($blk:keys) and .fld boundaries for sub-millisecond range queries.
- Slices contiguous 8-byte packed binary record IDs directly via index() and substr(), replacing slow multi-key OR unions.
- Stored distinct block values under $blk:vals (and tiered A:$blk:vals, B:$blk:vals) in .inx for binary search boundary detection.
- Added candidate pruning for small survivor sets (<= 250 records) directly on the raw binary slice.
- Full lifecycle management for $blk:vals: automatically maintains and prunes values across sort_add, sort_modify, and sort_del.
- [FIELD QUERY ENHANCEMENTS]
- field_fetch now accepts named block arguments via resolve_block_idx().
- Fixed raw numeric value recovery for fallback queries on normalized .inx sort keys.
- [DATE ENGINE OPTIMIZATIONS]
- Added memoization caching to AmberDB::Date with rollover protection.
- Added reset_date() methods.
- Replaced direct hash access with fast accessor methods across dump, vacuum, and backup routines.
- [DIRECTORY ARCHITECTURE & LOCK PATH STANDARDIZATION]
- Standardized lock_dir and session_dir as single-key directories in AmberDB::Base::Ramdisk and AmberDB::Base.
- Removed obsolete fallback keys (lock_rdir) and legacy naming aliases (conf_dir, cfg_dir).
- In RAM-disk environments, lock_dir and session_dir are dynamically mounted under $ramdisk_dir/lock and $ramdisk_dir/session with 0755 permissions; when unmounted, they reside cleanly under $dbase_dir/lock and $dbase_dir/session.
- Aligned database lock directory structures with Misk 4.x state locking specifications.
- [SCHEMA LOADING RESILIENCE]
- Wrapped 'do $target_path' inside eval { do $target_path } in AmberDB::Base::Schema (table_info() and dbase_info()).
- Prevents fatal process/worker termination when user .table or .dbase schema files contain syntax errors under global $SIG{__DIE__} handlers.
- [SLUG INDEX ARCHITECTURE VERIFICATION (.slg)]
- Verified flat unified .slg index access model where block numbers (formerly _0 / _1 file suffixes) reside in key prefixes:
- Block 0: "0:$rid" -> URL slug value
- Block 1: "1:$slug" -> Record ID value
- Enforced direct .slg table access and recs_get key lookups without legacy .rwt fallbacks.
- Verified flat unified .slg index access model where block numbers (formerly _0 / _1 file suffixes) reside in key prefixes:
- [GROUP-LEVEL SCHEMA INHERITANCE (.dbase -> .table)]
- Database group schema files (.dbase) now cascade common operational flags to all member tables in the group.
- Added %INHERITABLE_DBASE_KEYS whitelist in AmberDB::Base::Schema (use_ramdisk, keep_deleted, use_counter, log_owner, table_dir, use_section, etc.) preventing table-specific blocks and rules from leaking.
- Granular overrides: attributes explicitly defined in a .table file take precedence over .dbase defaults.
- Upgraded version string to 5.25.3 across all modules in lib/.
Documentation
High-performance Command-Line Console & Embedded Management Utility for AmberDB
Modules
High-performance embedded NoSQL database engine for Perl
Array, matrix manipulation, and set operations utility
Date manipulation, chronological ID generation, range calculation, and formatting utility
Multilingual text processing, collation, number/currency formatting, and search normalization engine
Database maintenance, CLI reindexing, migrations, and bulk conversion toolset
Provides
in lib/AmberDB/Tools/Index.pm
in lib/AmberDB/Tools/Maintain.pm
in lib/AmberDB/Tools/Update.pm