1
0
Files
2016-11-30 09:03:17 +08:00

149 lines
4.7 KiB
Plaintext

The Automake test suite
User interface
==============
Running all tests
-----------------
make check
You can use `-jN' for faster completion (it even helps on a
uniprocessor system, due to unavoidable sleep delays, as
noted below).
Interpretation
--------------
Successes:
PASS - success
XFAIL - expected failure
Failures:
FAIL - failure
XPASS - unexpected success
Other:
SKIP - skipped tests (third party tools not available)
Getting details from failures
-----------------------------
Each test is a script. In a non-VPATH build you can run them
directly, they will be verbose. By default, verbose output of
a test foo.test is retained in the log file foo.log. A summary
log is created in the file test-suite.log.
You can limit the set of files using the TESTS variable, and
enable detailed test output at the end of the test run with the
VERBOSE variable:
env VERBOSE=x TESTS='first.test second.test ...' make -e check
Reporting failures
------------------
Send verbose output, i.e., the contents of test-suite.log, of failing
tests to <bug-automake@gnu.org>, along with the usual version numbers
(which Automake, which Autoconf, which operating system, which make
version, which shell, etc.)
Writing test cases
==================
Do
--
If you plan to fix a bug, write the test case first. This way you'll
make sure the test catches the bug, and that it succeeds once you have
fixed the bug.
Add a copyright/license paragraph.
Explain what the test does.
Cite the PR number (if any), and the original reporter (if any), so
we can find or ask for information if needed.
Use `required=...' for required tools.
Include ./defs (see other tests) for normal tests, ./defs-p for tests
that use the `parallel-tests' option. For the latter, use a name that
ends in `-p.test' and does not clash with any generated tests in the
suite.
Use `set -e' to catch failures you might not have thought of.
./defs sets a skeleton configure.in. If possible, append to this
file. In some cases you'll have to overwrite it, but this should
be the exception. Note that configure.in registers Makefile.in
but do not output anything by default. If you need ./configure
to create Makefile, append AC_OUTPUT to configure.in.
Use $ACLOCAL, $AUTOMAKE, $AUTOCONF, $AUTOUPDATE, $AUTOHEADER,
$PERL, $MAKE, $EGREP, and $FGREP, instead of the corresponding
commands.
Use $sleep when you have to make sure that some file is newer
than another.
Use `cat' or `grep' to display (part of) files that may be
interesting for debugging, so that when a user send a verbose
output we don't have to ask him for more details. Display stderr
output on the stderr file descriptor. If some redirected command
is likely to fail, and `set -e' is in effect, display its output
even in the failure case, before exiting.
Use `Exit' rather than `exit' to abort a test.
It's more important to make sure that a feature works, than
make sure that Automake's output looks correct. It might look
correct and still fail to work. In other words, prefer
running `make' over grepping `Makefile.in' (or do both).
If you run $AUTOMAKE or $AUTOCONF several times in the same test
and change `configure.in' by the meantime, do
rm -rf autom4te.cache
before the following runs. On fast machines the new `configure.in'
could otherwise have the same timestamp as the old `autom4te.cache'.
Alternatively, use `--force' for subsequent runs of the tools.
Use filenames with two consecutive spaces when testing that some
code preserves filenames with spaces. This will catch errors like
`echo $filename | ...`.
Before commit: make sure the test is executable, add the tests to
TESTS in Makefile.am, add it to XFAIL_TESTS in addition if needed,
write a ChangeLog entry, send the diff to <automake-patches@gnu.org>.
Do not
------
Do not test an Automake error with `$AUTOMAKE && Exit 1', or in three
years we'll discover that this test failed for some other bogus reason.
This happened many times. Better use something like
AUTOMAKE_fails
grep 'expected diagnostic' stderr
(Note this doesn't prevent the test from failing for another
reason, but at least it makes sure the original error is still
here.)
Do not override Makefile variables using make arguments, as in
$MAKE ANSI2KNR=./ansi2knr U=_ all
this is not portable for recursive targets (targets that
call a sub-make may not pass `ANSI2KNR=./ansi2knr U=_' along).
Use the following instead.
ANSI2KNR=./ansi2knr U=_ $MAKE -e all
Do not send a test case without signing a copyright disclaimer.
See http://sources.redhat.com/automake/contribute.html or
ask <automake@gnu.org> for details.