Changes for version 0.320 - 2026-09-24
- read_table(): sep may be a qr// regex
- sep (and its synonym delim) was only ever a literal string. A qr// was stringified to "(?^:\s+)", matched byte for byte, never found, and every line came back as a single field keyed by the whole header line, with no warning. A qr// is now a pattern: sep => qr/\s+/ reads whitespace-aligned columns, qr/\s*,\s*/ trims around each comma, and qr/[;,]/ takes either character. A string sep is the literal it always was, so sep => '\s+' still means those three characters.
- qr/\s+/ is whitespace-delimited, as sep=r"\s+" is in pandas and sep = "" in R's read.table: leading and trailing whitespace on a line make no field. Any other pattern cuts as split() does, so a separator at the start of a line leaves an empty first field. Capture groups in the pattern are not fields, and a pattern that can match an empty string, such as qr/\s*/, is refused rather than cutting between every character. The pattern is matched as written, so its group numbers are its own and a backreference works: qr/(:)\1/ splits on "::".
- Quoted fields, comments and commented-out headers, blank lines, a byte-order mark, CRLF, filter, row.names, auto.row.names, na.strings and all three output types behave exactly as with a literal separator: the same XS parser reads both, and only how it finds a separator differs. pandas drops quote handling for a regex separator; read_table keeps it. An .xlsx still ignores sep.
- The separators are found by perl's regex engine, called from C on the parser's own line buffer, so a regex read keeps the C fast path: 0.17 s rather than 0.14 s for a literal sep on a 300,000 x 5 CSV. A literal sep is untouched and as fast as before.
- read_table(): header => 0, col.names and quote => ''
- read_table always took the first line as the header, so a file without one -- an NCBI taxonomy dump, R's write.table(col.names = FALSE) -- lost its first row to the column names. header => 0 (R's header = FALSE, pandas' header=None), or perl's false '', reads it as data, and the columns are named by col.names or, as R names them, V1, V2, ... col.names with a header renames its columns, warning as R does when the lengths differ. With no header there is none to rescue, so a line starting with the comment marker is a comment even when text hugs it ("#comment"), as in R.
- A '"' anywhere in an unquoted field opened a quoted one, so a file whose quotes are not CSV quoting had every line up to the next '"' read into one cell, silently. NCBI's fullnamelineage.dmp has a name ending in 'Beach rock 4+5"', and about 950,000 of its 3,015,956 lines went into one field. quote => '' (R's quote = "", pandas' quoting=QUOTE_NONE) makes '"' ordinary text, with a literal sep and a regex one alike; that file now reads in full. The default is unchanged.
- write_table(): a hash of hashes keeps its outer keys
- A HoH's outer keys were written only as row names, and row.names has been off by default since it stopped following R's write.table. For every other shape that label is a 1..n index nobody needs, but a HoH's keys are the only place its row identifiers exist, so a taxid-keyed HoH came out with no taxids in it at all. A HoH now defaults row.names on, under an empty header cell, which read_table reads back as row_name. row.names => 0 still drops the keys, and every other shape is unchanged.
- row.names => 'name' on a HoH was accepted and did nothing a 1 would not: the key column's header stayed empty. It now heads that column, so row.names => 'taxid' writes "taxid,genus,species", in delimited, LaTeX and .xlsx output alike. A name that is also a column being written -- a col.names entry, or else a key of any inner hash -- dies before the file is opened, rather than writing two columns of one name.
- read_table(): output.type => 'aoa'
- read_table could return an AoH, a HoA or a HoH, but not the AoA that write_table, agg, csort and melt all take. output.type => 'aoa' returns the header row, then one array per data row, every row in file column order, with an empty or na.strings cell undef as in every other shape. The header row is what write_table reads an AoA's first row as, so a read and a write round-trip a delimited file byte for byte.
- It is the one shape that keeps every field when the header repeats a name, so it gives no "later values win" warning. Nothing labels an AoA's rows, so row.names with it is an error. It goes through the same C fast path as the other three shapes when there is no filter, and through the per-row closure when there is one.
- Tests
- t/read_table.regex_sep.t takes its whitespace and multi-character cases from pandas 3.0.4's parser tests, and checks that every CSV under t/, and a tab-separated copy of each, reads the same through qr/\Q$sep\E/ as through the literal separator, warnings and errors included.
- t/read_table.header_quote.R.pandas.t takes its cases from R 4.6.1's tests/reg-IO2.R and reg-tests-1a, 1b, 1d and 2, with the values R itself gives frozen from t/read_table.header_quote.R, and from pandas 3.0.4's test_header.py, test_quoting.py, test_dialect.py and test_na_values.py. The one cell where R and read_table differ -- R ends a line at a comment character anywhere in it -- is pinned there.
Modules
Get basic statistical functions, like in R, but with Perl using XS for performance