149 lines
4.7 KiB
Plaintext
149 lines
4.7 KiB
Plaintext
The Automake test suite
|
|
|
|
|
|
User interface
|
|
==============
|
|
|
|
|
|
Running all tests
|
|
-----------------
|
|
|
|
make check
|
|
|
|
You can use `-jN' for faster completion (it even helps on a
|
|
uniprocessor system, due to unavoidable sleep delays, as
|
|
noted below).
|
|
|
|
Interpretation
|
|
--------------
|
|
|
|
Successes:
|
|
PASS - success
|
|
XFAIL - expected failure
|
|
|
|
Failures:
|
|
FAIL - failure
|
|
XPASS - unexpected success
|
|
|
|
Other:
|
|
SKIP - skipped tests (third party tools not available)
|
|
|
|
|
|
Getting details from failures
|
|
-----------------------------
|
|
|
|
Each test is a script. In a non-VPATH build you can run them
|
|
directly, they will be verbose. By default, verbose output of
|
|
a test foo.test is retained in the log file foo.log. A summary
|
|
log is created in the file test-suite.log.
|
|
|
|
You can limit the set of files using the TESTS variable, and
|
|
enable detailed test output at the end of the test run with the
|
|
VERBOSE variable:
|
|
|
|
env VERBOSE=x TESTS='first.test second.test ...' make -e check
|
|
|
|
|
|
Reporting failures
|
|
------------------
|
|
|
|
Send verbose output, i.e., the contents of test-suite.log, of failing
|
|
tests to <bug-automake@gnu.org>, along with the usual version numbers
|
|
(which Automake, which Autoconf, which operating system, which make
|
|
version, which shell, etc.)
|
|
|
|
|
|
|
|
Writing test cases
|
|
==================
|
|
|
|
|
|
Do
|
|
--
|
|
|
|
If you plan to fix a bug, write the test case first. This way you'll
|
|
make sure the test catches the bug, and that it succeeds once you have
|
|
fixed the bug.
|
|
|
|
Add a copyright/license paragraph.
|
|
|
|
Explain what the test does.
|
|
|
|
Cite the PR number (if any), and the original reporter (if any), so
|
|
we can find or ask for information if needed.
|
|
|
|
Use `required=...' for required tools.
|
|
|
|
Include ./defs (see other tests) for normal tests, ./defs-p for tests
|
|
that use the `parallel-tests' option. For the latter, use a name that
|
|
ends in `-p.test' and does not clash with any generated tests in the
|
|
suite.
|
|
|
|
Use `set -e' to catch failures you might not have thought of.
|
|
|
|
./defs sets a skeleton configure.in. If possible, append to this
|
|
file. In some cases you'll have to overwrite it, but this should
|
|
be the exception. Note that configure.in registers Makefile.in
|
|
but do not output anything by default. If you need ./configure
|
|
to create Makefile, append AC_OUTPUT to configure.in.
|
|
|
|
Use $ACLOCAL, $AUTOMAKE, $AUTOCONF, $AUTOUPDATE, $AUTOHEADER,
|
|
$PERL, $MAKE, $EGREP, and $FGREP, instead of the corresponding
|
|
commands.
|
|
|
|
Use $sleep when you have to make sure that some file is newer
|
|
than another.
|
|
|
|
Use `cat' or `grep' to display (part of) files that may be
|
|
interesting for debugging, so that when a user send a verbose
|
|
output we don't have to ask him for more details. Display stderr
|
|
output on the stderr file descriptor. If some redirected command
|
|
is likely to fail, and `set -e' is in effect, display its output
|
|
even in the failure case, before exiting.
|
|
|
|
Use `Exit' rather than `exit' to abort a test.
|
|
|
|
It's more important to make sure that a feature works, than
|
|
make sure that Automake's output looks correct. It might look
|
|
correct and still fail to work. In other words, prefer
|
|
running `make' over grepping `Makefile.in' (or do both).
|
|
|
|
If you run $AUTOMAKE or $AUTOCONF several times in the same test
|
|
and change `configure.in' by the meantime, do
|
|
rm -rf autom4te.cache
|
|
before the following runs. On fast machines the new `configure.in'
|
|
could otherwise have the same timestamp as the old `autom4te.cache'.
|
|
Alternatively, use `--force' for subsequent runs of the tools.
|
|
|
|
Use filenames with two consecutive spaces when testing that some
|
|
code preserves filenames with spaces. This will catch errors like
|
|
`echo $filename | ...`.
|
|
|
|
Before commit: make sure the test is executable, add the tests to
|
|
TESTS in Makefile.am, add it to XFAIL_TESTS in addition if needed,
|
|
write a ChangeLog entry, send the diff to <automake-patches@gnu.org>.
|
|
|
|
|
|
Do not
|
|
------
|
|
|
|
Do not test an Automake error with `$AUTOMAKE && Exit 1', or in three
|
|
years we'll discover that this test failed for some other bogus reason.
|
|
This happened many times. Better use something like
|
|
AUTOMAKE_fails
|
|
grep 'expected diagnostic' stderr
|
|
(Note this doesn't prevent the test from failing for another
|
|
reason, but at least it makes sure the original error is still
|
|
here.)
|
|
|
|
Do not override Makefile variables using make arguments, as in
|
|
$MAKE ANSI2KNR=./ansi2knr U=_ all
|
|
this is not portable for recursive targets (targets that
|
|
call a sub-make may not pass `ANSI2KNR=./ansi2knr U=_' along).
|
|
Use the following instead.
|
|
ANSI2KNR=./ansi2knr U=_ $MAKE -e all
|
|
|
|
Do not send a test case without signing a copyright disclaimer.
|
|
See http://sources.redhat.com/automake/contribute.html or
|
|
ask <automake@gnu.org> for details.
|