NAME
AmberDB - High-performance, schema-driven NoSQL engine with ACID transactions and precomputed inverted indexing for Perl
SYNOPSIS
use AmberDB;
my $adb = AmberDB->new(
cfg => { language => "tr" },
path => { dbase_dir => "./dbstore" }
);
my @record = ( 0, "John Doe", "New York", 1980, 'john@example.com' );
# Insert record
$adb->insert_id("table_id", @record);
# Update record
$adb->modify_id("table_id", $record_id, @record_updated);
# Delete record
$adb->delete_id("table_id", $record_id);
# Read record by ID
my @record = $adb->read_id("table_id", $record_id);
# Read all records
my @records = $adb->read_all("table_id");
# Read list of records by IDs
my @records = $adb->read_list("table_id", \@id_list);
# Search for the string "New York" in field 2 using the match function.
my @records = $adb->field_fetch("table_id", 2, "New York");
# or if the New York ID is 142
my @records = $adb->field_fetch("table_id", 2, 142);
# Full-text string search
my @records = $adb->search_table("table_id", "search string");
# If `read_all`, `field_fetch`, and `search_table` take the `$limit` parameter, they will not read all records; they will read a specific range based on `start` and `limit`, and return the number of records at the beginning.
my ($count, @records) = $adb->read_all("table_id", $start, $limit);
my ($count, @records) = $adb->field_fetch("table_id", $field_no, $match_value, $start, $limit);
my ($count, @records) = $adb->search_table("table_id", "search string", $start, $limit);
# Direct low-level table access
$adb->table_write($file_path);
my $records = $adb->recs_get($file_path, @rec_ids);
my $ok = $adb->recs_put($file_path, @records);
$adb->table_close($file_path);
# Transaction example (Checkout / Stock operation)
my $res = $adb->transact_start();
my $order_id = $adb->insert_id("order", 0, $user_id, $item_id, $qty);
if ( !$order_id ) {
$adb->transact_rollback();
}
my $stock_id = $adb->modify_id("stock", $item_id, $user_id, $item_id, $new_qty);
$adb->transact_end();DESCRIPTION
AmberDB is a high-performance, flat-file NoSQL database engine for Perl built on top of Berkeley DB (DB_File). It combines the speed of flat-file storage with enterprise features: schema-driven multi-dimensional indexing, ACID-compliant transactions with Strict Two-Phase Locking (Strict 2PL), columnar faceted navigation, multilingual locale processing, and native RAM-disk caching.
SUBMODULE ARCHITECTURE & INHERITANCE
AmberDB is built as a unified coordinator that incorporates all functionality from specialized submodules via inheritance (use parent). When you instantiate an AmberDB object ($adb), all methods from the following submodules are directly available as methods on $adb:
AmberDB::Base — Core record serialization (
db_encode,db_decode), schema loading (table_info), path mapping (table_path), compact 8-byte binary packing (bin_encode,bin_decode), and file locking (flock_open,flock_close).AmberDB::Array — Array set operations (
array_nodup,array_crop,array_add,array_punch,array_substr), matrix transformations (inverse_matrix), filtering (array_filter), multi-dimensional sorting (array_sort), and deep copying (deep_copy).AmberDB::String — Smart string truncation (
sub_str,truncate_text,short_title), whitespace flattening (trim_space), bidirectional HTML conversion (text2html,html2text), content type detection (what_isthis), and HTML entity sanitization.AmberDB::Date — Compact chronological ID getters (
day_id,second_id,month_id), date string parsing (str2dateid,dateid2str), range generation (day_range), ISO week numbers (dateid2week), and relative offset calculation (offset2date).AmberDB::Locale — Multilingual text processing, locale-aware casing (
uc,lc,ucfirst), Unicode Collation (UCA) sorting (sort), ASCII transliteration (to_ascii), number-to-words / cheque conversion (num2text), number formatting (format_number), currency formatting (format_currency), and CLDR pluralization (plural).AmberDB::Locale::Currency — ISO 4217 currency definitions, symbols, and UI dropdown lists.
AmberDB::Cache — Unified RAM-disk (tmpfs / ImDisk) cache engine (
cache_read,cache_write,cache_preload) and persistent staging buffers (buffer_read,buffer_write).AmberDB::Transact — Multi-table ACID-compliant transaction engine with Strict Two-Phase Locking (Strict 2PL) and undo journaling (
transact_start,transact_end,transact_rollback,transact_recover).AmberDB::Index — Inverted full-text keyword indexing (
.src), exact field match indexing (.fld), binary pre-sorted indexing (.srt), and bidirectional SEO URL rewrite maps (.rwt).AmberDB::Index::Facet — Columnar forward indexing (
.fac), disjunctive count calculation, and dynamic scoped menu builder (facet_menu,field_fltkeys).AmberDB::Index::Junk — Schema-driven dual-tier cold record archiving (Hot Tier A vs. Cold Tier B) and query layer routing (
jnktype => 'A'|'AB'|'B'|'BA').AmberDB::Tools — Maintenance CLI, index rebuilding (
set_index,set_search,set_filters), and database-wide conversion.
All submodules (except AmberDB::Tools which takes an $adb handle) can also be instantiated and used independently in standalone scripts.
TABLE NAMING CONVENTIONS
AmberDB enforces a strict, deterministic lowercase snake_case table naming convention:
Format: All table identifiers must consist of lowercase alphanumeric characters in snake_case, structured as
<database>_<table_name>(e.g.catalog_product,member_address,orders_item).Database Prefix Resolution: The segment before the first underscore (
_) represents the logical database/schema group (mapped to<database>.dbase).Schema Files: A table
catalog_productautomatically resolves its schema fromcatalog_product.tableand its database group settings fromcatalog.dbase.Constraint: Uppercase or mixed-case table names (e.g.
Catalog_Product) are not supported and will fail database group extraction.
SCHEMA DEFINITION & CONFIGURATION (.table & IN-MEMORY)
AmberDB is schema-driven. Table schemas define primary key constraints, field blocks, multi-dimensional indexes, automatic SEO slug generation, facet filters, lifecycle junk rules, and repeating nested items.
Schemas can be defined in two ways:
- 1. Disk-Based Schema Files: Placed in the
dbstore/schema/<table_name>.tabledirectory. AmberDB loads and parses them automatically upon first access. - 2. Programmatic In-Memory Schemas: Defined directly on the AmberDB instance via
$adb->table_attr('table_id', { ... }).
Example Table Schema (catalog_product.table)
Defining blocks in the schema is not mandatory. However, `record_index`, `match_block`, `search_block`, and `sort_block` are crucial, especially for the automatic creation of indexes during record keeping. `record_index` only takes the value 0/1. `match_block` and `search_block` determine which blocks will be indexed, while `sort_block` determines both the blocks to be sorted and the sort type.
{
name => "Product Catalog",
record_index => 1, # Enable .inx primary record index
match_block => [1, 2, 3, 11], # .fld exact field match indexes (Category, Brand, etc.)
search_block => [4, 5, 7], # .src full-text search fields (Title, Subtitle, Description)
sort_block => [ 4, { blk => 10, type => 'num' } ], # .srt pre-sorted ID buffers
keep_deleted => 1, # Enable soft-delete audit log (.del)
log_owner => 1, # Enable change audit logging (.aut)
}Dynamic Runtime Schema Manipulation (table_attr)
Schemas can be dynamically reconfigured in-memory at runtime without modifying disk files or requiring table migrations:
# Dynamically change full-text search fields on the fly
$adb->table_attr("catalog_product", { search_block => [ 4, 9 ] });
# Toggle caching or soft-delete modes dynamically
$adb->table_attr("catalog_product", { use_cache => 0, keep_deleted => 0 });Expandable Records without SQL JOINs (Repeating Blocks)
AmberDB supports hierarchical, JSON-like extensible records without the need for child tables or relational JOIN queries. Multiple repeating child items (e.g., order lines, cart items, invoice lines) can be appended directly to the parent record. repeat_start should indicate the block number where the last repeating record started. The AmberDB engine writes the first ID of each row from repeat_start to the end, concatenated by commas, to the repeat_ids block. You must ensure that this block number also appears in match_block.
# Schema configuration for expanding order table
{
name => "Customer Orders",
record_index => 1,
match_block => [1, 2, 4], # Customer ID, Order Date, Products
repeat_ids => 4, # products field: item ids, separated by comma
repeat_start => 5, # repeat block begin at block 5
blocks => [
{ id => "id", name => "Order ID", type => "auto_id" },
{ id => "customer_id", name => "Customer ID", type => "text" },
{ id => "order_date", name => "Order Date", type => "text" },
{ id => "total_price", name => "Total Amount", type => "num" },
{ id => "products", name => "Products", type => "text" },
# Repeating line items:
{ id => "item_id", name => "Item ID", type => "text" },
{ id => "item_title", name => "Product Title",type => "text" },
{ id => "item_qty", name => "Quantity", type => "num" },
{ id => "item_price", name => "Unit Price", type => "num" },
],
}TRANSACTIONS
Transactions provide multi-table atomic updates backed by undo-log journals (.txn files). If a database error occurs (e.g. file lock failure, duplicate ID), or if custom business validation fails (e.g. insufficient stock), all base records and indexes across all affected tables are restored to their exact pre-transaction state.
Checkout / Stock Deduction Example
$adb->transact_start();
# 1. Check & update stock
my @product = $adb->read_id("product", $product_id);
my $current_stock = $product[4];
if ($current_stock < $quantity) {
# Custom business logic rollback (e.g. stock insufficient)
$adb->transact_rollback();
return { success => 0, error => "Out of stock" };
}
$product[4] -= $quantity;
$adb->modify_id("product", $product_id, @product);
# 2. Insert order record
my $order_id = $adb->insert_id("orders", 0, $user_id, $product_id, $quantity, time());
# 3. Finalize transaction (auto-rollbacks if base error occurred)
my $txn = $adb->transact_end();
if ($txn->{status} eq 'commit') {
return { success => 1, order_id => $order_id };
} else {
return { success => 0, error => "The operation failed, the changes were reverted." };
}Note / Limitations: Bulk/list operations (insert_list, modify_list, delete_list) do not support the transact operation. There is a fundamental reason for this. Junk operations are designed for loading, editing, or deleting a list containing records of the same type. Records in a list do not hierarchically affect each other. For example, when entering 1000 product records in bulk via XML, if one or more of them cannot be saved due to incorrect formatting, it does not cause a problem for the other records.
Furthermore, if the user truly wants to perform an operation on the list using transact, they can put it in a loop and use the individual insert_id, modify_id, delete_id operations.
METHODS
new(%options)
Instantiates a new AmberDB object.
config([$key], [%options])
Gets or sets runtime configuration flags deterministically with automatic hook/side-effect dispatching (e.g. locale reloading, table path invalidation):
# Single scalar getter
my $lang = $adb->config('language');
# Bulk getter (returns a safe shallow copy)
my $cfg = $adb->config();
# Key-value setter with method chaining
$adb->config( language => 'en', no_write => 1 );
# Hashref setter
$adb->config({ simple => 1, cache_size => '1024M' });insert_id($table_id, [$record_id], @record)
Inserts a new record into specified table. It automatically generates search, match, slug, and facet indexes if they are defined in the table schema. It supports transact operations. In normal records, there is no need to enter an ID value. It can be entered as empty, undef, or 0. The system automatically generates the ID using an incrementing counter and returns the ID value.
insert_list($table_id, @records)
Inserts multiple records in a single bulk operation. Aside from Transact, it processes records, search, match, SEO slug, and facet indexes all at once with high performance.
modify_id($table_id, $record_id, @record)
Updates existing record data. It automatically updates the search, match, slug, and facet indexes if they are defined in the table schema. It supports transact operations.
modify_list($table_id, @records)
Modifies multiple records in a single bulk operation. Aside from Transact, it processes records, search, match, SEO slug, and facet indexes all at once with high performance.
delete_id($table_id, $record_id)
Deletes specified record from table. Supports transaction logging.
delete_list($table_id, @records)
Deletes multiple records in a single bulk operation. Aside from Transact, it processes records, search, match, SEO slug, and facet indexes all at once with high performance.
read_id($table_id, $record_id)
Reads single record by primary key ID.
read_all($table_id, [$start], [$limit], [%options])
Reads active records from table. Supports pagination, binary index optimization (.inx, .srt), sorting, and keys_only.
IMPORTANT (Return Signature Convention): When $limit is passed and > 0 (paginated), read_all returns ($total_count, @records) where the first scalar is the total matching count integer. When $limit is omitted or 0 (unpaginated), it returns @records directly. Unpacking a paginated query into my @records causes $records[0] to be an integer scalar, which will crash if dereferenced as an array reference.
# 1. Unpaginated (returns array of record arrayrefs directly)
my @records = $adb->read_all("catalog_product");
# 1.1 Unpaginated with sorting / options (pass 0, 0 for start and limit)
my @sorted_desc = $adb->read_all("catalog_product", 0, 0, sort => 2);
my @sorted_asc = $adb->read_all("catalog_product", 0, 0, sort => -2);
my @sorted_full = $adb->read_all("catalog_product", 0, 0, sort => { blk => 2, reverse => 1 });
# 2. Paginated (limit > 0: first element is total matching count integer)
my ($total_count, @page_records) = $adb->read_all("catalog_product", 0, 20);
my ($total_count, @page_records) = $adb->read_all("catalog_product", start => 0, limit => 20);
# 2.1 Paginated with sorting
my ($total_count, @records) = $adb->read_all("catalog_product", 0, 20, sort => 2);
my ($total_count, @records) = $adb->read_all("catalog_product", 0, 20, sort => -2);
my ($total_count, @records) = $adb->read_all("catalog_product", 0, 20, sort => { blk => 2, reverse => 1 });
my ($total_count, @records) = $adb->read_all("catalog_product", 0, 20, sort => { reverse => 1 });Tiered query mode: 'A' (Active only), 'B' (Junk only), 'AB' (Active first, then Junk)
my @active_only = $adb->read_all("catalog_product", jnktype => 'A');
my ($total_count, @all_tiered) = $adb->read_all("catalog_product", 0, 20, jnktype => 'AB');Return only scalar record IDs (memory-efficient pipeline)
my ($count, @ids) = $adb->read_all("catalog_product", 0, 50, keys_only => 1);
my @all_ids = $adb->read_all("catalog_product", keys_only => 1);Forcing non-indexed reading (no_index)
my @all_ids = $adb->read_all("catalog_product", 0, 0, no_index => 1);read_list($table_id, \@id_list)
Reads multiple records matching provided ID list while preserving exact list ordering.
# Read the entire active order list.
my @records = $adb->read_all("order_active");
# Extract customer IDs from block 1 using the map.
my %customer_ids = map { $_->[1] => 1 } @records;
# You've found the customer ID keys, now read them using read_list.
my @customers = $adb->read_list("customers", [ keys %customer_ids ]);field_fetch($table_id, $block, $value, [$start], [$limit], [%options])
Fetches records matching one or more block values using the .fld match index (or sequential table scan fallback if unindexed). Supports multi-value queries, automatic deduplication, sorting, pagination, and keys_only.
IMPORTANT (Return Signature Convention): When $limit is passed and > 0 (paginated), field_fetch returns ($total_count, @records) where the first scalar is the total matching count integer. When $limit is omitted or 0 (unpaginated), it returns @records directly. Unpacking a paginated query into my @records causes $records[0] to be an integer scalar, which will crash if dereferenced as an array reference.
# 1. Unpaginated (returns array of record arrayrefs directly)
my @records = $adb->field_fetch("products", 1, "5");
my @sorted_asc = $adb->field_fetch("products", 1, "5", 0, 0, sort => -10);
# 2. Paginated (first element is total matching count integer)
my ($total_count, @records) = $adb->field_fetch(
"products", 1, "5",
0, 20,
sort => { blk => 10, reverse => 1 } # Or shorthand: sort => -10 (ascending)
);
# Multi-value matching (comma string, semicolon, or ARRAY ref)
my @records = $adb->field_fetch("products", 1, ["5", "8"]);
my @records = $adb->field_fetch("products", 1, "5, 8");
# Return only record IDs: keys_only flag
my @all_ids = $adb->field_fetch("products", 1, "5", keys_only => 1);
my ($total_count, @ids) = $adb->field_fetch("products", 1, "5", 0, 20, keys_only => 1);
# Tiered Junk query mode
my @active = $adb->field_fetch("products", 1, "5", jnkmode => 'A'); # Only Active recordsfield_fetch uses the match_block definition in the schema and accesses inverted match index files (.fld), providing $O(1)$ average-time lookup per indexed key (total retrieval cost scales with the number of requested values and matching record IDs). If match_block is not defined or if running in simple mode, field_fetch falls back to a sequential table scan.
search_table($table_id, $query, [$start], [$limit], [$mode], [%options])
It performs searches matching query terms using the full-text .src index (or a sorted table scan backup method if unindexed). search_table uses the AmberDB::Locale module. It features advanced language normalization according to the selected language (apostrophe stop words, accent normalization, phonetic silencing as in Turkish b/d/g -> p/t/k, circumflex vowels â/î/û), block filtering, tier mode selection (jnktype => 'A' | 'AB' | 'B' | 'BA'), sorting, pagination, and keys_only features.
IMPORTANT (Return Signature Convention): When $limit is passed and > 0 (paginated), search_table returns ($total_count, @records) where the first scalar is the total matching count integer. When $limit is omitted or 0 (unpaginated), it returns @records directly. Unpacking a paginated query into my @records causes $records[0] to be an integer scalar, which will crash if dereferenced as an array reference.
# 1. Unpaginated (returns array of record arrayrefs directly)
my @records = $adb->search_table("catalog_product", "kablosuz kulaklık");
my @sorted_records = $adb->search_table("catalog_product", "kulaklık", 0, 0, sort => -5);
# 2. Paginated (first element is total matching count integer)
my ($total_count, @search) = $adb->search_table( "catalog_product", "kulaklık", 0, 20 );
my ($total_count, @search) = $adb->search_table(
"catalog_product", "kulaklık",
start => 0,
limit => 20,
sort => -5,
filter => { field => 6, value => 12 },
jnktype => 'AB',
);
# Return only scalar record IDs
my @all_ids = $adb->search_table("catalog_product", "kulaklık", keys_only => 1);
my ($total_count, @ids) = $adb->search_table("catalog_product", "kulaklık", 0, 50, keys_only => 1);field_filter($table_id, \%filter_options)
Performs multi-block filtered queries (AND / OR) with support for multi-value filters, tier mode selection (jnktype), sorting, and pagination:
my $res = $adb->field_filter("catalog_product", {
type => "and",
filter => { 1 => "5", 6 => ["12", "14"] },
sort => { blk => 5, reverse => 1 },
jnktype => "AB",
start => 0,
limit => 20,
});
# Returns: { count => $total, ids => \@matching_ids }exist_id($table_id, $record_id)
Checks if a single record exists in the specified table. Returns 1 if present, 0 otherwise:
my $exists = $adb->exist_id("catalog_product", 101);exist_list($table_id, @record_ids)
Queries the presence of multiple record IDs in a single pass. Returns a hash reference { id => 1/0 }:
my $map = $adb->exist_list("catalog_product", 101, 102, 103);exist_table($table_id, [$ext])
Checks whether the physical database table or index file exists on disk. $ext defaults to $self->{db_ext} ('db'):
my $has_table = $adb->exist_table("catalog_product");
my $has_index = $adb->exist_table("catalog_product", "inx");table_count($table_id)
Returns the total number of records in the specified table. Reads from the primary .inx index if enabled, or scans the main table:
my $total_records = $adb->table_count("catalog_product");table_keys($table_id)
Returns an array of all record IDs present in the table (retrieved from memory cache, .inx index, or sequential table scan):
my @all_ids = $adb->table_keys("catalog_product");table_lastid($table_id)
Returns the highest / auto-increment primary key ID currently allocated in the table:
my $last_id = $adb->table_lastid("catalog_product");table_attr($table_id, [$key_or_attributes])
Reads or dynamically customizes table schema attributes in-memory at runtime without altering schema files on disk:
# 1. Single attribute getter (scalar)
my $id_type = $adb->table_attr("catalog_product", "id_type");
# 2. Bulk attribute getter (returns a safe shallow copy)
my $attrs = $adb->table_attr("catalog_product");
# 3. Key-value setter (automatically recalculates paths if year/section/lang changes)
$adb->table_attr("catalog_product", id_type => "ascii", keep_deleted => 1);
# 4. Hashref setter
$adb->table_attr("catalog_product", { search_block => [ 4, 9 ], use_cache => 0 });table_create($table_id)
Creates an empty physical database file (.db) on disk. If a table is accessed with table_write and does not exist, it is created automatically. table_create is useful to prevent file-not-found errors before initial read operations on new tables:
$adb->table_create("catalog_product");table_read($file_path)
Opens a DB_File database file in read-only mode (O_RDONLY). No exclusive lock is applied. table_read and table_write are used for both base data files (.db) and index files (note that internal encodings differ):
my $db_obj = $adb->table_read("/path/to/table.db");table_write($file_path)
Opens a DB_File database file in read-write mode (O_RDWR | O_CREAT) and acquires an exclusive write lock (flock LOCK_EX). Uses the file path as the handle key:
my $db_obj = $adb->table_write("/path/to/table.db");table_close($file_path)
Syncs, unlocks, and closes the specified DB_File handle, releasing its file lock and removing it from the internal connection pool:
$adb->table_close("/path/to/table.db");recs_exist($file_path, @record_ids)
Low-level existence check directly on an open DB_File handle. Returns boolean 1/0 for a single ID, or a hash reference { id => 1/0 } for multiple IDs:
my $is_found = $adb->recs_exist($file_path, "101");
my $id_map = $adb->recs_exist($file_path, "101", "102");recs_keys($file_path)
Extracts all raw keys directly from an open DB_File handle in sequential order using C-level seq:
my @raw_keys = $adb->recs_keys($file_path);recs_scan($file_path, [$mode_or_callback])
Scans key-value pairs sequentially directly from an open DB_File handle using C-level seq. Supports multiple modes: - \&callback: Invokes callback->($key, $val) for each pair. - 'keys': Returns list/arrayref of all keys. - 'value' / 'values': Returns list/arrayref of all raw values. - 'each' / 'pairs': Returns list/arrayref of [$key, $val] pairs. - 'count': Returns total number of records. - 'hash' (default): Returns key-value hash (or hashref).
# Examples
my @keys = $adb->recs_scan($file_path, "keys");
my @values = $adb->recs_scan($file_path, "values");
my @each = $adb->recs_scan($file_path, "each");
# Custom iterator
$adb->recs_scan($file_path, sub {
my ($key, $val) = @_;
print "Key: $key, Val: $val\n";
});recs_get($file_path, @record_ids)
Direct raw record retrieval for specific record IDs from an open DB_File handle. Returns { id => raw_val }:
my $raw_data = $adb->recs_get($file_path, 101, 102);recs_put($file_path, @records)
Writes records in bulk directly to an open DB_File write handle. Each item must be in [$rid, @fields] or [$rid, $val] format:
$adb->recs_put($file_path, [ 101, "Category", "Brand", "Title" ]);recs_del($file_path, @record_ids)
Deletes specified record IDs directly from an open DB_File write handle:
$adb->recs_del($file_path, 101, 102);transact_start()
Starts a new transaction for atomic multi-table operations.
transact_end()
Transact terminates the process. If any errors occur in the underlying database during the process, it performs a LIFO rollback by executing transact_rollback. If no errors are found, it commits the transact_commit operation.
transact_rollback()
It forces a manual rollback of the active operation immediately. It doesn't need to be called in the normal flow. transact_end calls transact_rollback if it receives a transact_error log.
flock_open($table_id, [$mode], [$record_id])
Acquires a record-level (if $record_id specified) or table-level (if $record_id omitted) lock. $mode can be "write" (exclusive lock, default) or "read" (shared lock).
flock_close($table_id, [$record_id])
Releases a record-level or table-level lock previously acquired via flock_open().
AUTHOR
Maruf Cetin <marufcetin@gmail.com>
LICENSE AND COPYRIGHT
Copyright (C) 2005-2026 Maruf Cetin.
This library is free software; you can redistribute it and/or modify it under the terms of the Artistic License 2.0.