NAME
yq - Filter YAML through a command-line program
VERSION
version 0.037
SYNOPSIS
yq [-vx] <script> [<file>...]
yq -h|--help|--version
DESCRIPTION
This program takes a stream of YAML documents (on STDIN or file arguments), applies a filter, then writes the results to STDOUT.
ARGUMENTS
script
The script to run. For the script syntax, see SYNTAX.
<file>
A YAML file to filter. The special file "-" refers to STDIN. If no files are specified, filter STDIN.
OPTIONS
-v | --verbose
Set verbose mode to print out some internal program messages on STDERR to help with debugging.
-x | --xargs
xargs mode. When the filter returns only a single item, simply print it out without using the serializer. This allows single values to be piped into other programs that may not know how to deal with serialized data, like xargs.
-h | --help
Show this help document.
--version
Print the current yq and Perl versions.
SYNTAX
EXPRESSIONS
An EXPRESSION
is allowed to be either a FILTER, VALUE, or a COMPARISON.
FILTERS
Filters select a portion of the incoming documents. Filters can be combined to reach deep inside the documents you're working with.
- .
-
Returns the entire document, unfiltered. Useful in if/then statements.
# INPUT foo: bar baz: fuzz $ yq . foo: bar baz: fuzz
- .key
-
Extract a single item out of a hash.
# INPUT foo: bar: baz fizz: fuzz $ yq .foo bar: baz fizz: fuzz $ yq .foo.fizz fuzz
- .[#]
-
Extract a single item out of an array.
# INPUT - foo: fuzz - bar: buzz - baz: good: bad new: old $ yq .[1] bar: buzz $ yq .[2] baz: good: bad new: old $ yq .[2].baz good: bad new: old $ yq .[2].baz.new old
- []
-
Use [] with no index to flatten an array.
# INPUT - foo: fuzz - bar: buzz $ yq '.[]' foo: fuzz --- bar: buzz
- $.
-
$.
is the whole original document before any pipe operators changed what part of the document we're working with. - Assignment
-
Filters can be assigned new values. The new value can be another filter or any "VALUES".
# INPUT foo: fuzz bar: baz --- foo: buzz bar: taz $ yq '.foo = 2' foo: 2 bar: baz --- foo: 2 bar: taz
Multiple assignments can be combined with
|
:$ yq '.foo = .bar | .bar = "foo"' foo: baz bar: foo --- foo: taz bar: foo
VALUES
- 'STRING' "STRING"
-
Both single- and double-quoted strings are allowed. Using \ will escape the string delimiter.
- { KEY: EXPRESSION, ... }
-
The hash constructor.
KEY
may be anyFILTER
or a bare value.# INPUT foo: bar baz: fuzz --- foo: 1 baz: 2 $ yq '{ bar: .foo, .baz: foo }' bar: bar fuzz: foo --- 2: foo bar: 1
- [ EXPRESSION, ... ]
-
The array constructor.
# INPUT foo: bar baz: fuzz --- foo: 1 baz: 2 $ yq '[ .foo, .baz ]' - bar - fuzz --- - 1 - 2
- empty
-
The special value empty suppresses printing of a document. Normally, an undefined document will show up in the output as "--- ~". If your filter instead yields empty, the document will not be printed at all.
This is especially useful in conditionals:
# INPUT foo: bar baz: fuzz $ yq 'if .foo eq bar then . else empty' foo: bar baz: fuzz $ yq 'if .foo eq buzz then . else empty' $
... though see the
grep()
function for a shorter way of writing this. - Values
-
Any bareword that is not recognized as a syntax element is treated as a value. These barewords may only contain letters, numbers, and underscore.
NOTE: This may be subject to change to only allow quoted strings and bare numbers in a future version.
COMPARISONS
- eq
-
String equals comparison. Returns true if both sides are equal to each other when treated as a string.
The two sides may be FILTERS or VALUES.
# INPUT foo: bar baz: fuzz buzz: fuzz $ yq '.foo eq bar' true $ yq '.baz eq .buzz' true $ yq '.baz eq bar' false
YAML treats the string "true" as a true value, and the string "false" as a false value.
- ne
-
String not equals comparison. Returns true if one side is not equal to the other side when compared as a string.
The two sides may be FILTERS or VALUES.
# INPUT foo: bar baz: fuzz buzz: fuzz $ yq '.foo eq bar' true $ yq '.baz eq .buzz' true $ yq '.baz eq bar' false
YAML treats the string "true" as a true value, and the string "false" as a false value.
- ==
-
Numeric equals comparison. Returns true if both sides are equal to each other when treated as numbers. If one of the items is not a number, will print a warning to STDERR but try to compare anyway.
The two sides may be FILTERS or VALUES.
# INPUT one: 1 two: 2 uno: 1 $ yq '.one == 1' true $ yq '.one == 2' false $ yq '.one == .uno' true
- !=
-
Numeric not equals comparison. Returns true if both sides are equal to each other when treated as numbers. If one of the items is not a number, will print a warning to STDERR but try to compare anyway.
The two sides may be FILTERS or VALUES.
# INPUT one: 1 two: 2 uno: 1 $ yq '.two != 1' true $ yq '.two != 2' false $ yq '.one != .uno' false
- > / >=
-
Numeric greater-than (or equal-to) comparison. Returns true if the left side is greater than (or equal-to) the right side. If one of the items is not a number, will print a warning to STDERR but try to compare anyway.
The two sides may be FILTERS or VALUES.
# INPUT one: 1 two: 2 uno: 1 $ yq '.two >= 1' true $ yq '.two > 2' false $ yq '.one >= .uno' true
- < / <=
-
Numeric less-than (or equal-to) comparison. Returns true if the left side is less than (or equal-to) the right side. If one of the items is not a number, will print a warning to STDERR but try to compare anyway.
The two sides may be FILTERS or VALUES.
# INPUT one: 1 two: 2 uno: 1 $ yq '.two <= 1' false $ yq '.two < 2' false $ yq '.one <= .uno' true
FUNCTIONS
- length( EXPRESSION )
-
Returns the length of the thing returned by EXPRESSION. Depending on what type of thing that is:
string/number - Returns the number of characters array - Returns the number of items hash - Returns the number of pairs
If EXPRESSION is missing, gives the length of the entire document (
length(.)
). Returns a number suitable for assignment.Although length() takes an expression, certain constructs are redundant:
length( keys( EXPRESSION ) ) -> length( EXPRESSION ) # length() works on hashes
A future version may optimize these away, or warn you of their redundancy.
# INPUT foo: one: 1 two: onetwothreefourfive three: 3 baz: [ 3, 2, 1 ], $ yq 'length(.)' 2 $ yq 'length' 2 $ yq 'length( .foo )' 3 $ yq 'length( .baz )' 3 $ yq 'length( .foo.two )' 19 $ yq '{ l: length( .foo.two ) }' l: 19
- keys( EXPRESSION )
-
Return the keys of the hash or the indicies of the array returned by EXPRESSION. If EXPRESSION is missing, gives the keys of the entire document (
keys(.)
).Returns an array suitable for assignment.
# INPUT foo: one: 1 two: 2 three: 3 baz: [ 3, 2, 1 ] $ yq 'keys( .foo )' - one - two - three $ yq 'keys( .baz )' - 0 - 1 - 2 $ yq 'keys( . )' - foo - baz $ yq 'keys' - foo - baz $ yq '{ k: keys( .foo ) }' k: - one - two - three
- each( EXPRESSION )
-
Return a list of key/value pairs for the hash or array given by EXPRESSION. If EXPRESSION is missing, gives the key/value pairs of the entire document (
each(.)
).# INPUT foo: one: 1 two: 2 three: 3 baz: [ 3, 2, 1 ] $ yq 'each' --- key: foo value: one: 1 two: 2 three: 3 --- key: baz value: [ 3, 2, 1 ] $ yq 'each( .foo )' --- key: one value: 1 --- key: two value: 2 --- key: three value: 3 $ yq 'each( .baz )' --- key: 0 value: 3 --- key: 1 value: 2 --- key: 2 value: 1
The documents created by each can be piped to further filters.
- grep( EXPRESSION )
-
If
EXPRESSION
is true, return the current document. Otherwise, returnempty
.This is exactly the same as:
if EXPRESSION then . else empty
- select( EXPRESSION )
-
Another name for
grep()
to matchjq
's syntax. - group_by( EXPRESSION )
-
Group incoming documents based on the result of
EXPRESSION
, yielding a single document containing a hash of arrays.# INPUT --- foo: 'bar' baz: 1 --- foo: 'bar' baz: 2 --- foo: 'baz' baz: 3 $ yq 'group_by( .foo )' bar: - foo: bar baz: 1 - foo: bar baz: 2 baz: - foo: baz baz: 3
NOTE: If you are filtering a lot of documents, this will consume a lot of memory.
- parse_time( EXPRESSION, FORMAT )
-
Parse the date/time string in
EXPRESSION
and return the number of seconds since the UNIX epoch (1970-01-01 00:00:00).FORMAT
is optional, and will try to guess what format the date/time is in.FORMAT
may be one of the following:- iso
-
An ISO8601 date/time string with a 4-digit year. 2-digit years, week numbers, and year days are not (yet) supported.
2017-01-01T00:00:00 2017-01-01 00:00:00 2017-01-01 20170101000000 201701010000 20170101
- apache
-
The date/time format used by the Common Log Format (Apache HTTP logs).
01/Jan/2017:00:00:00
# INPUT timestamp: 2017-01-01 00:00:00 $ yq '.timestamp = parse_time( .timestamp, "iso" )' timestamp: 1483228800 $ yq '.timestamp = parse_time( .timestamp )' timestamp: 1483228800
CONDITIONALS
- if EXPRESSION then TRUE_FILTER else FALSE_FILTER
-
If the
EXPRESSION
is true, return the result ofTRUE_FILTER
, otherwise return the result ofFALSE_FILTER
.# INPUT foo: bar baz: fuzz $ yq 'if .foo eq bar then .baz else .foo' fuzz $ yq 'if .foo eq buzz then .baz else .foo' bar $ yq 'if .foo then .baz' fuzz $ yq 'if .herp then .derp else .' foo: bar baz: fuzz
The
else FALSE_FILTER
is optional and defaults to returning undefined.
COMBINATORS
Combinators combine multiple expressions to yield one or more documents in the output stream.
- ,
-
Multiple EXPRESSIONS may be separated by commas to yield multiple documents in the output.
# INPUT foo: bar baz: fuzz $ yq '.foo, .baz' bar --- fuzz
- |
-
Multiple EXPRESSIONS may be separated by pipes to give the output of the left expression as the input of the right expression (much like how shell pipes work).
# INPUT foo: bar baz: fuzz pop: more --- foo: buzz baz: fizz pop: jump $ yq '{ foo: .foo, val: .pop } | group_by( .foo )' bar: - foo: bar val: more buzz: - foo: buzz val: jump
The above example can be useful to avoid
group_by
memory issues when dealing with very large streams: Reduce the size of the working document by keeping only the keys you want, then group those documents.
ENVIRONMENT VARIABLES
- YERTL_FORMAT
-
Specify the default format Yertl uses between commands. Defaults to
yaml
. Can be set tojson
for interoperability with other programs. - YQ_VERBOSE
-
Set the verbosity level. Useful when running the tests.
AUTHOR
Doug Bell <preaction@cpan.org>
COPYRIGHT AND LICENSE
This software is copyright (c) 2016 by Doug Bell.
This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.
1 POD Error
The following errors were encountered while parsing the POD:
- Around line 644:
You forgot a '=back' before '=head2'