Changes for Test::Nginx

0.24 - 2014-12-07
 *   feature: added support for the "benchmark" testing mode enabled
     by the TEST_NGINX_BENCHMARK environment variable.

 *   feature: added support for the TEST_NGINX_BENCHMARK_WARMUP
     environment to specify the number of warm-up requests before the
     actual benchmark testing.

 *   feature: added support for the "--- curl" section that generates
     a "curl" command line for the corresponding test request.

 *   feature: added new section "--- server_name" to control the
     value of the server_name nginx config directive used for the
     default server {}.

 *   feature: added new section "--- error_log_file".

 *   feature: automatically print out warnings for [emerg] messages
     in nginx's error.log to stderr if it is not mentioned in "---
     error_log" nor "--- no_error_log".

 *   feature: Test::Nginx::Socket: "--- request" now supports leading
     comment lines prefixed by "#".

 *   feature: the "--- user_files" section supports raw Perl data
     structures for specifying the user files.

 *   feature: the "--- more_headers" section now supports perl
     array-ref typed values for pipelined requests.

 *   feature: exported the "add_response_body_check" and "is_str"
     utility functions.

 *   feature: the "--- udp_reply" section now accepts a Perl
     subroutine as its value which can be used to generate dynamic
     replies based on the actual query. thanks blablacio for the
     original patch.

 *   feature: Test::Nginx::Socket: exported new Perl utility
     functions "add_cleanup_handler", "add_block_preprocessor", and

 *   feature: made the special $LIBxxx_PATH variable more general in
     the "--- stap" section.

 *   feature: the "--- no_error_log" section now tries to find all
     the matched lines in error logs instead of stopping on the first

 *   feature: added new section "--- tcp_shutdown" for shutting down
     read, write, or both parts of the connection in the embedded TCP
     server immedately after the connection is established.

 *   feature: the value of "--- tcp_query_len" now defaults to the
     lenghth of the value of "--- tcp_query" (if any).

 *   feature: added new section "--- must_die" for testing the cases
     that nginx fail to start (like invalid nginx configurations and
     etc). thanks Markus Linnala for the patch.

 *   bugfix: when user specifies "Host" and "Connection" in "---
     more_headers", then we should not add our own Host request

 *   bugfix: made the default value of the "Connection" request
     header to be "close" instead of "Close".

 *   bugfix: the "check leak" testing mode might send a bogus request
     header via ab or weighttp.

 *   bugfix: we should also wait for the time specified by the "---
     abort" section before checking error logs for the repeated
     requests that are not the last one.

 *   optimize disabled accept_mutex by default in the auto-generated

 *   bugfix: "--- skip_nginx" and "--- skip_nginx2" did not skip
     tests related to the mocked TCP and UDP servers.

 *   bugfix: we did not automatically remove the .stp and .stp-out
     temp files created by the "--- stap" sections in the systemtap
     testing mode.

 *   bugfix: "--- request" did not provide the guilty request line
     that it failed to parse.

 *   bugfix: Test::Nginx::Socket: "--- error_code" might take bad

 *   bugfix: we incorrectly discarded all the query data read in the
     mocked TCP server when timeout errors happen (due to the actual
     query is shorter than the expected data specified "---
     tcp_query", for example).

 *   bugfix: the "check leak" testing mode could not parse raw HTTP
     requests using LF instead of CRLF for line terminators.

 *   doc: documented the section "--- init".

 *   doc: added info for the openresty and openresty-en mailing lists
     for discussions.

 *   doc: spelling fixes from Markus Linnala.

0.23 - 2014-04-05
 *   feature: added new section --- response_body_unlike. thanks
     Rickey Visinski for the patch.

 *   feature: allow use of the $ServerPort and $ServerPortForClient
     variables in "--- response_headers".

 *   feature: now we automatically kill the process listening on the
     mocked TCP port when we have tried for more than 20 times.

 *   feature: added new section "--- http09" to Test::Nginx::Socket
     to indicate use of HTTP 0.9 requests.

 *   feature: added support for the $LIBLUA_PATH variable in "---
     stap", which evaluates to the absolute path of the liblua*.so
     DSO loaded by the nginx server.

 *   feature: we skipped reading the response body for 101 responses.

 *   feature: added new section "--- post_main_config" to add custom
     top-level configuration snippets right after the http {} block
     generated by Test::Nginx.

 *   feature: added unix domain socket support for tcp_listen. thanks
     Jiale Zhi for the patch.

 *   feature: added new sections "--- grep_error_log" and "---

 *   feature: added utility function check_accum_error_log for
     checking the accumulated error logs in "--- error_log" and "---

 *   feature: added utility Perl function server_port_for_client.
     thanks doujiang for the patch.

 *   feature: added configuration Perl function server_name(). thanks
     doujiang for the original patch.

 *   feature: added new module Test::Nginx::Socket::Lua.

 *   feature: automatically test if there is any assertion failures
     in the nginx error logs.

 *   feature: when nginx process is gone unexpectedly, we
     automatically tail the nginx error log file.

 *   feature: added the "--- no_check_leak" section to skip test
     blocks in the "check leak" testing mode.

 *   feature: "--- no_error_log" accepts a single Perl Regexp object
     now, just like "--- error_log".

 *   bugfix: Test::Nginx::Socket tried to read response bodies for
     304 responses.

 *   bugfix: we did not try hard enough before killing the nginx
     process by force.

 *   bugfix: improved the error handling when recv() fails on the
     HTTP connection.

 *   bugfix: subtests for "--- tcp_query", "--- tcp_query_len", and
     "--- udp_query" disabled test plan checks.

 *   bugfix: we did not pass the LD_LIBRARY_PATH environment to nginx
     when systemtap is used.

 *   bugfix: "--- error_log" should work on each test request's own
     logs only.

 *   bugfix: server address should default to instead of

 *   bugfix: made the test skip messages better.

 *   change: removed "env LUA_PATH" and "env LUA_CPATH" from the
     default nginx configuration generated.

 *   change: set --num-callers=100 to valgrind by default.

 *   doc: Test::Nginx::Socket: documented the various exported Perl
     functions like "run_tests", "repeat_each", and "no_shuffle".

0.22 - 2013-06-10
 *   feature: added new sections --- abort and --- shutdown to test
     premature client abortions.

 *   feature: implemented new section "---

 *   feature: added support for custom http_config filters for

 *   feature: now we automatically check [alert] and [crit] messages
     in the nginx error logs and print them as warnings if they do
     not appear in either --- error_log or --- no_error_log.

 *   feature: added new section "--- stap_out_unlike" for testing
     patterns which should not appear in the output systemtap.

 *   feature: added support for the special notation "$LIBPCRE_PATH"
     within --- stap, which is always substituted by the absolute
     path to libpcre used by the current nginx.

 *   feature: now we check if the nginx process is gone in the "check
     leak" testing mode (i.e., when TEST_NGINX_CHECK_LEAK=1).

 *   feature: added new section "--- skip_eval: <subtest-count>
     <perl-code>" for skipping tests according to the running result
     of a piece of Perl code.

 *   feature: now we allow a (unix domain) socket file path specified
     in the "--- udp_listen" directive value.

 *   feature: in --- stap values, now we support the notation

 *   feature: made the environment TEST_NGINX_SERVER_PORT always take
     the value so that the test cases can always reference it.

 *   feature: added new section "--- wait" to always wait a specified
     amount of time before checking the systemtap outputs and/or
     error logs (if any).

 *   bugfix: no longer retry connecting to the nginx server when the
     child process running nginx is already gone.

 *   bugfix: no longer bail out the up to 3 test cases fail all their
     connecting attempts.

 *   bugfix: TEST_NGINX_USE_HUP=1 could not work with

 *   bugfix: the output of systemtap took the value undef when it was
     in fact empty.

 *   bugfix: protect against division by zero in the "check leak"
     testing mode.

 *   bugfix: disabled the test report ending when failing to parse
     chunked response body because the test count is surely out of

 *   bugfix: the code that searches executables in the PATH
     environment would always look up the current working directory
     first and might pick up an executable directory with the same

 *   bugfix: conflicting testing modes may conflict together.

 *   bugfix: added File::Path 2.06_05 as a minimum requirement.
     thanks Andreas J. König for reporting this issue.

 *   bugfix: pass --show-possibly-lost=no to valgrind by default
     because nginx makes clever use of pointers almost everywhere.

 *   bugfix: when --- abort was not specified, http client timeout
     was not treated as a subtest failure as specified.

 *   bugfix: the child process running ab or weighttp might not be
     reaped in extreme conditions.

 *   bugfix: we did not close STDOUT and STDERR in the child
     processes forked, which could confuse the "prove" harness and
     lead to hang.

 *   optimize: disabled the access log altogether in the "check leak"
     testing mode.

 *   docs: documented the TEST_NGINX_NO_CLEAN environment.

 *   docs: fixed a small issue in the code sample. thanks smallfish.

0.21 - 2012-09-27
 *   feature: added support for SystemTap: the SystemTap white-box
     testing mode is enabled by setting TEST_NGINX_USE_STAP=1 and by
     using new test sections "--- stap", "--- stap_out", and "---

 *   feature: added support for new sections "--- udp_listen" and
     "--- udp_reply" for mocking an internal UDP server.

 *   feature: added new section "--- udp_reply_delay" and also make
     "--- udp_reply_delay" and "--- timeout" sections accept time
     values with a time unit which can be either "s" or "ms".

 *   feature: added new section "--- udp_query" to test if the UDP
     query sent to the embeded UDP server is expected.

 *   feature: added new section "--- tcp_query_len" to work with
     mockeagain testing modes.

 *   feature: added support for embedded TCP server by introducing
     new sections "--- tcp_listen", "--- tcp_query", "--- tcp_reply",
     and "--- tcp_reply_delay".

 *   feature: added new section "--- tcp_no_close" to tell the
     embedded TCP server (if any) never close the connection
     actively. this is feature is mainly for mocking upstream reply
     read time out in the middle.

 *   feature: added new section "--- log_level" to specify the error
     log level on a test-by-test basis.

 *   bugfix: we might lose logs at config-time when
     TEST_NGINX_USE_HUP=1 because we sent HUP before USR1.

 *   bugfix: be more tolarent when removing the nginx pid file while
     the nginx process is not running.

 *   optimize: now we can automatically check the version of the
     current nginx configuration and retry dozens of times when
     TEST_NGINX_USE_HUP=1. this speed up the test suite in this mode
     a lot and also avoid accidental failures due to slow nginx

 *   bugfix: try harder when failing to connect to the Nginx server.

 *   refactor: now we use the standard "return" directive instead of
     "echo" for detecting nginx config reload completion for the "HUP
     reload" testing mode.

 *   refactor: use waitpid() wherever possible.

 *   bugfix: Parallel::ForkManager does not allow the user calling
     waitpid directly, or inifinite looping might happen when calling
     its start() method. now we no longer use this module any more
     and use the fork() builtin function exclusively.

 *   bugfix: disabled the ending output "Looks like you ..." when the
     client socket timed out.

 *   refactor: now we disabled the access log by default.

 *   updated the bundled versions of Test::Builder and Test::More

 *   docs: documented the "--- skip_nginx" and "--- skip_nginx2"
     sections. thanks Weibin Yao for requesting it.

0.20 - 2012-06-25
* feature: now "--- error_log" allows a single regexp object as its value.

* feature: made client socket timeout as a test failure.

* feature: added support for the environment TEST_NGINX_CHECK_LEAK=1
  that enables the most general memory leak check by calling
  ab/weighttp + ps (for the nginx process's RSS).

* feature: when the TEST_NGINX_USE_HUP environment is set to true,
  then HUP signal will be used to update the nginx config between
  different test blocks and there will be no HUP reload between
  every two successive test cases within the same test block when
  "repeat_each" is set to a number more than 1.

* bugfix: fixed the TEST_NGINX_SLEEP environment support.

* bugfix: we should always restore user files before each
  "repeat_each" iteration because some test case may modify
  the user files.

* bugfix: setting environment TEST_NGINX_USE_HUP could not work
  with TEST_NGINX_USE_VALGRIND=1 properly.

* bugfix: print out every test block's name when

* bugfix: the "response body truncated" warning was a false alarm
  for HEAD requests. thanks Piotr Sikora for reporting this in
  github issue #1.

* bugfix: do not add --gen-suppressions and --suppressions options
  to the valgrind command line when custom non-number
  TEST_NGINX_USE_VALGRIND env value is specified.

* docs: documented the "--- timeout" section.

* docs: added a link to António P. P. Almeida's debian package for this module.

0.19 - 2012-04-29
* feature: added new section "--- error_code_like".
* bugfix: tests could hang on slow machines after maximal number of connecting
  attempts (as well as other exceptions). Now we eliminate dying directly from
  within the test scaffold. Instead, we always use Test::More::BAIL_OUT. Also,
  we always politely clean up all the forked child processes before we quit.
  Finally, we add timeout protection to the connect operation.
* bugfix: prevented blocking indefinitely on ForkManager's wait_all_children call.
  also made timeout an API provided by Test::Nginx::Util.
* feature: now we preserve environment "MOCKEAGAIN" as well.
* bugfix: the timestamps for the "--- user_files" directive should be GMT rather
  than localtime.
* feature: now section "no_error_log" also supports multi-line values to mean
  multiple literal patterns (per line).
* feature: the "--- user_files" sections now support absolute file paths and creating
  arbitrarily nested directories in the file path. thanks @penjin2012 for suggesting it.
* feature: the "--- error_log" section now supports multiple lines to mean multiple
  literal patterns.
* bugfix: fixed places that crash for bad responses.
* bugfix: fixed another crash when no response object is constructed.

0.18 - 2012-02-03
* feature: added new section "--- no_error_log" which could be used to specify
  patterns of lines that do not appear in error.log at all.
  environments that should be kept through nginx reload and restart.
* doc: added Naxsi and ngx_rds_csv to the user module list in the POD documentation.
* feature: now we can use the environment TEST_NGINX_USE_VALGRIND to specify the
  valgrind command-line options, when set to 1 or other non-zero numbers, it is
  effectively equivalent to the value "--tool=memcheck --leak-check=full". To
  use Valgrind SGCheck to run the tests, just specify the environment
  TEST_NGINX_USE_VALGRIND="--tool=exp-sgcheck" first.
* feature: added support for environment TEST_NGINX_VERBOSE.
* feature: added support for environment TEST_NGINX_USE_HUP.
* feature: added support for the TEST_NGINX_POSTPONE_OUTPUT environment.
* feature: now we'll retry sending the QUIT signal for at most 5 times when the
  nginx process is refusing to quit. after 5 trials, KILL signal will be sent.
* feature: exposes the "master_off" API which turns off the nginx master process.
* feature: added new section "--- ignore_response" to test bad responses.
* bugfix: Test::Nginx::Socket will retry 10 times if fail to connect because nginx
  with valgrind can be very slow to start in some systems.
* feature: added new section name "--- error_log" to test error lines. requested
  by Piotr Sikora in the last year or so.
* feature: added the TEST_NGINX_EVENT_TYPE environment to specify an event API type
  (like "epoll" or "select") when running the test suite.

0.17 - 2011-11-04
* bugfix: do not strip trailing CRCF for response headers in
  Test::Nginx::Socket's response parser.
* bugfix: fixed issues in Content-Length response header handling in case of
  pipelined reqeusts.
* bugfix: fixed the pipelined_requests section support.
* feature: added support for user file timestamp.

0.16 - 2011-07-07
* fixed a bug that length(undef) produced lots of warnings on perl 5.10.x.
  thanks Guo Ying (shrimp).
* now we automatically retry connecting for 3 times (1 sec delay)
  in Test::Nginx::Socket::send_request. this will help automatically recovering
  from random slow valgrind loading problems.
* when the directories specified in --- user_files do not exist, Test::Nginx will create it
* now we'll issue warning when there is extra bytes after the last chunk in the chunked response parser.

0.15 - 2011-05-23
* fixed a bug when using arrays in the "response_body" (and other similar) section when there was only one request.
* fixed presentation problems in the HTML generated from the POD.
* added documentation for the TEST_NGINX_* variables.
* added "local $?" in Test::Nginx::Util::END to avoid confusing Test::Builder::_ending()
* removed "use Test::More" from Test::Nginx::Socket because Test::Base already loads it.

0.14 - 2011-05-12
* now we require Parallel::ForkManager, Text::Diff, and Test::Base.
* the --- request section now supports multiple requests and even "packeting"
  (see documentation for details). And it should be backward compatible.
  Of course, the "expected" sections have been upgraded to expect
 (and even require) arrays in this situation.
* updated quite a bit the documentation.
* deprecated request_eval and response_body_eval. If you use them, the
  test will pass but you will be issued a "diag" statement telling you
  to move to "request eval" and "response_body eval" respectively.
* added new implementation for pipelined requests and raw requests.
  Rewrote SKIP statement to be less verbose and more in line with what
  is recommended on Test::Nginx.
* added support for environment variable TEST_NGINX_FORCE_RESTART_ON_TEST.
  Defaults to 1. Must be explicitly set to 0 to avoid restarting the nginx
  server when the config does not change between two tests.
  This also brings support for allowing config not to be set.
  and having a config section only on the first test, you can have all tests
  use the same config.
* added support for environment variable TEST_NGINX_NO_NGINX_MANAGER (defaults
    to 0) which will disable the nginx management code (stop/config/start).
    Very useful when you want to run tests on an already running NGINX (set
    port your running nginx is listening on).

0.13 - 2011-03-08
* documented the TEST_NGINX_PORT environment.

0.12 - 2011-03-08
* documented the etcproxy and valgrind integration support.
* implemented the no_nginx_manager utility function
* documented the difference between Test::Nginx::Socket and Test::Nginx::LWP.
* allow numbers in TEST_NGINX_BLAH_BLAH_BLAH env names.
* added support for --- raw_response_headers_like.
* fixed a bug in the can_run sub.

0.11 - 2010-10-09
* add TEST_NGINX_BUILDSLAVE and --- skip_slave.
* implemented the TEST_NGINX_IGNORE_MISSING_DIRECTIVES which can
  force test cases to skip when the directive is not
  found in the current nginx.
* added support for env TEST_NGINX_TIMEOUT.
* implemented TEST_NGINX_XXX env expansion in --- config and --- http_config sections.

0.10 - 2010-08-19
* we use our own can_run function rather than Module::Install::Can's.
* added TEST_NGINX_MASTER_PROCESS, rename TEST_NGINX_ROOT to less confusing TEST_NGINX_SERVROOT and reorder priority of TEST_NGINX_{CLIENT_,SERVER_,}PORT environment options. (Piotr Sikora)
* added support for TEST_NGINX_NO_SHUFFLE env as requested by Piotr Sikora :)
* added support for TEST_NGINX_LOG_LEVEL env but it can be overriden on the script level by calling the log_level function.
* added support for TEST_NGINX_ERROR_LOG: if this env is specified to some file path, then all the error log by all the test cases will be accumulated to that file.
* Now the test script driven by Test::Nginx will tell nginx to quit by default. The TEST_NGINX_NO_CLEAN=1 env setting will give back the old behavior :) Thanks Piotr Sikora for requesting this.
* added support for --- main_config.
* Make Test::Nginx more friendly to the environment (Piotr Sikora)
    Added environment variables:
    TEST_NGINX_BINARY - full path to the nginx binary,
    TEST_NGINX_ROOT - path where server root should be created,
    TEST_NGINX_PORT - port number that should be used by client and server
                      (takes precedence over _CLIENT_PORT and _SERVER_PORT),
    TEST_NGINX_SERVER_PORT - port number that should be used by the server.
* add environment variable TEST_NGINX_SLEEP
* added support for ngx_openresty in nginx version parser.

0.09 - Wed 07 Jul 2010 02:57:13 PM CST
* added new section --- user_files which will create user files under html/.
* applied the patch from Piotr Sikora to add the --- skip_nginx2 directive.
* use the latest nginx version as the default if the version number can not be parsed from "nginx -V".
* added support for ! HEADER syntax to --- response_headers such that we can test if a response header exists.
* fixed some bugs in --- response_headers.
* now TEST_NGINX_USE_VALGRIND=1 works out of the box. one can ualso use test/valgrind.suppress file to serve as a default suppression file.
* better diagnostics for socket timeout outputs.
* added support for the --- charset section.
* use localhost as the server_name in the test nginx.conf.
* added support for the server_addr sub as well as the TEST_NGINX_CLIENT_PORT environment.
* always use Time::HiRes's sleep. this issue was reported by leicheng++.
* added new config function no_root_location as well as the new section --- init.

0.08 - Tue 26 Jan 2010 12:20:03 PM CST
* fixed a bug in the chunked parser in Test::Nginx::Socket:
  now we immediately stop reading more data chunks when
  seeing the last chunk.
* added func no_long_string to Test::Nginx::Socket and
  Test::Nginx::LWP so that we can disable
  Test::LongString in the test scaffold.
* added utility config functions "workers", "log_level", and
  "master_on" to Test::Nginx::Socket.
* added ngx_rds_json and ngx_xss modules to the module
  list that uses Test::Nginx::Socket.

0.07 - Tue 05 Jan 2010 10:06:56 AM CST
* we now use IO::Select in Test::Nginx::Socket to save CPU resources
  while polling the non-blocking socket.
* POD: added ngx_drizzle to the module list that use Test::Nginx::Socket.
* added utility config functions "worker_connections" and

0.06 - Thu 24 Dec 2009 10:59:29 AM CST
* we now disable the nginx daemon mode in the config file by default.
* we now generate a default index.html page under the root /.
* added support for the new --- http_config section so that you can
  put upstream foo { ... } in it.

0.05 - Tue 22 Dec 2009 10:17:44 AM CST
* added support for gcc profiling via the
  $Nginx::Util::Profiling variable.
* fixed the chunked decoder for large chunks.

0.04 - Fri 18 Dec 2009 04:09:02 PM CST
* added new sections --- raw_request and
  --- raw_request_middle_delay to Test::Nginx::Socket.
* added new functions config_preamble and repeat which
  are exported by default.

0.03 - Tue 15 Dec 2009 11:35:07 AM CST
- refactored a lot of duplicate code between
  Test::Nginx::LWP and Test::Nginx::Socket to
  a separate module named Test::Nginx::Util.
- now Test::Nginx::Socket also restart the nginx
  server at every test case run.
- added support for --- request_eval and
  --- response_eval to Test::Nginx::Socket.

0.02 - Wed 09 Dec 2009 09:50:11 AM CST
- skipped the POD coverage tests for now.

0.01 - Tue 08 Dec 2009 11:07:51 AM CST
- initial release.