Changes for version 0.002 - 2026-11-19

  • cpanfile declares the core modules the code loads but did not list: Archive::Tar, File::Temp, JSON::PP, Text::ParseWords
  • Scenarios: a case may carry scenario (an id of [a-z0-9-]); cases with the same scenario are variants of one task. The report keeps the field, and every verdict of a suite with scenarios carries scenarios { total, solved, rate, unsolved }: a scenario is solved only when every variant passed in all k repetitions (AppWorld's scenario goal completion). The text notes it per suite; it does not decide ok
  • journald speaks about itself: journalctl -u systemd-journald shows at the boot time "Journal started" and "Runtime Journal (/run/log/journal/ ID) is …, max …, … free." (or "System Journal" in /var), and a restart logs the new process's journal with its limit and the flush, after "Started". journalctl --header lists path, state and time span of every journal file; -t, SYSLOG_IDENTIFIER=, _COMM= (cut to 15 characters as the kernel does) and _SYSTEMD_UNIT= filter by sender. --list-boots starts at the boot time, as uptime -s
  • RouterOS simulator: /ip pool used lists the addresses the DHCP leases hold in their server's pool (print, where, count-only; read only)
  • Monitoring simulator: Prometheus answers /api/v1/targets (static targets of the running configuration; health from the service's scrape_health or the connection probe) and /api/v1/query for a single selector, optionally with time() - before and a comparison after it (up from the targets, other series only from the service's metrics). Alertmanager answers /api/v2/alerts, /api/v2/silences and /api/v2/receivers from the same data amtool uses. Anything else stays a gap
  • Shell filters: python3 -m json.tool behind a pipe formats JSON like the real module (key order kept, indent 4, \uXXXX), without a sandbox
  • PostgreSQL simulator: the near relatives of the slot query answer truly instead of ending as a gap -- plugin, database and confirmed_flush_lsn of a physical slot are NULL, a WAL difference to a NULL LSN is NULL, pg_stat_replication is empty while no slot is active, DROP REPLICATION SLOT is the syntax error of an SQL session, and psql takes -Atc
  • A wget that no observation matches is answered from the case's curl observation for the same request, in wget's words, when the document goes to standard output (wget -qO- URL). Before, such a call fell through to the simulator and could get "Connection refused" for a service the case describes as running (ADR 0013)
  • Linux simulator: the world is closed on a device's own network. An address in a directly connected network that no device carries and the state names nowhere does not exist: /dev/tcp, nc, wget and curl fail with "No route to host" and ping with "Destination Host Unreachable", without a sim_gap. Named addresses and everything beyond a gateway answer as before. A device reaches its own networks even when the case lists no routes (ADR 0012)
  • `holmgang run --sandbox docker|podman [--sandbox-image IMAGE]`: a real program the Linux simulator does not model (python3 -c, perl -e, an awk script past the filter grammar) runs in a locked-down container on a copy of the device's files instead of ending as a sim_gap; what it wrote comes back into the state. Opt-in per run, never in loading or lint (ADR 0011). New: Langertha::Holmgang::Sandbox::Docker, the image in share/sandbox/Dockerfile
  • A call whose programs ran for real carries `sandbox` in the report, and `generator.sandbox` names runtime, image and image id; both only when a sandbox was used

Documentation

Lint Holmgang suites, run them against a Langertha engine, evaluate the reports

Modules

Deterministic tool-use benchmarks for language models via Langertha
Suites against targets and their variants, repeated, with bounded concurrent requests per target, resumable
The holmgang command: lint suites, run them against a Langertha engine, evaluate reports
One case of a suite: its own format 2 values and what derives from them
A loud refusal of input: the field path and the reason
Per-topic evaluation of Holmgang reports: weights, length tiers, verdicts, comparison
Excerpts from a suite's statute texts under quellen/, set into messages
The expectations on a value, shared by rules, when conditions and check.expect
Grades a case deterministically: rules at the end state and on the calls
Long, deterministic operational data for long-context cases, generated from a short spec
The self-check every case passes on every load
Loads suite format 2 from search directories, lints every case
The seeded private random source for case generators
The JSON report of a Holmgang invocation, format holmgang_report 1
The plugin role of a world with its own tools
The plugin role of a device simulator
The model loop of one case over a Langertha engine: turns, turn limit, tokens per turn, lint mock runs
Runs one real program on a copy of a device's files in a locked-down container
The bash grammar of the simulators — words, items, heredocs, blocks, expansion, test
The text filters behind a pipe — grep, head, tail, wc, sort, uniq, cut, awk, tr, fold, python3 -m json.tool
The answer world denken — reasoning tasks, the answer handed in by tool, one checker per task like score_answer of Reasoning Gym
The simulator of device type ios — a Cisco IOS switch CLI
The answer world kueche — recipe collection, pantry, shopping list, week plan, units
The simulator of device type linux — a Debian shell with ip, nft, wg, sysctl, systemctl, nmcli, files
Accounts and sudo on a simulated Linux server — passwd, chage, deluser, userdel, visudo, sudo -l
Backup and restore on a simulated Linux server — tar, rsync, restic, sha256sum
systemd calendar expressions (OnCalendar) for the Linux simulator — check, normalise, next elapse
BIND 9 on a simulated Linux server — named.conf, zones, rndc, dig, secondaries
Docker Engine with Compose v2 on a simulated Linux server — projects, containers, derived state
git on simulated Linux hosts — working copy, index, commits, push to a Forgejo server with hooks
helm on the admin host of a simulated cluster — releases, values, revisions, rendering
HTTP between simulated devices: curl, applications behind ports, probes at the end state
Kubernetes cluster behind kubectl on a simulated Linux server — stored resources, derived status
logrotate 3.21 (Debian 12) for the Linux simulator — reads the configuration, checks (-d), rotates (-f)
MariaDB on a simulated Linux server — mysql/mariadb client, variables, users, binlogs, replica
Monitoring on a simulated Linux server — checks Prometheus/Alertmanager, promtool, amtool
Evaluates the nftables ruleset of a simulated Linux device against a packet
nginx on a simulated Linux server: checking and loading the configuration, answering requests
Certificates and keys as data: openssl, update-ca-certificates, verification in the client
Postfix on a simulated Linux server — postconf, postmap, newaliases, queue
PostgreSQL on a simulated Linux server — psql, an SQL subset, pg_hba, settings, slots
Redis on a simulated Linux server — redis-cli, CONFIG GET/SET/REWRITE, INFO, BGSAVE
Redis from the service alone (services.redis-server.running) — redis-cli connection, auth, PING, CONFIG GET and redis-server -v
Whether a user can log in over SSH: port, firewall, Match, account, authorized_keys, StrictModes
OpenSSH server on a simulated Linux server — sshd -t/-T, Include, effect of reload/restart, login
Disks and file systems on a simulated Linux server — LVM, mount, fstab, swap, mdadm
The answer world roost — an invented dragon roost with perches and a larder, for the example suite demo-roost
The simulator of device type routeros — a RouterOS (MikroTik) console
The answer world schichtfehler — a web application from recorded pages: sources, fetches, console, reported findings
The answer world sehen — hand in what an image shows: graph, sequence, fields, table, values, finding
The answer world uebersetzung — a translation job: hand in segments, report ambiguity, look up the glossary
Registry of simulators by device type, and dispatch of calls to them
The shape checks of suite format 2, shared by the loader and the DSL
A suite: head, tools, defaults and its cases in load order
A Langertha engine as a benchmark target: variant settings, one model turn, failed turns classified
Kolon output templates of handlers and observations
The JSON Schema subset of tool parameters, checked at load and per call
The simulated world of a case — state, tool dispatch, handlers, end-state diff