From 58c182ae59f585c5687104e0112d163274c6c732 Mon Sep 17 00:00:00 2001 From: Stefano Lattarini Date: Sun, 3 Jul 2011 19:20:22 +0200 Subject: [PATCH] tap: add experimental TAP-aware driver * doc/automake.texi (Using the TAP test protocol): New section. (Overview of Custom Test Drivers Support): Minor updates. * lib/tap-driver: New script, TAP-aware test driver for Automake; implemented in perl and based on TAP::Parser. * lib/Makefile.am (dist_script_DATA): Add it. * tests/tap-autonumber.test: New test. * tests/tap-bailout.test: Likewise. * tests/tap-basic.test: Likewise. * tests/tap-deps.test: Likewise. * tests/tap-diagnostic.test: Likewise. * tests/tap-empty.test: Likewise. * tests/tap-empty-diagnostic.test: Likewise. * tests/tap-escape-directive.test: Likewise. * tests/tap-exit.test: Likewise. * tests/tap-fancy.test: Likewise. * tests/tap-fancy2.test: Likewise. * tests/tap-global-result.test: Likewise. * tests/tap-html.test: Likewise. * tests/tap-log.test: Likewise. * tests/tap-merge-stdout-stderr.test: Likewise. * tests/tap-more.test: Likewise. * tests/tap-more2.test: Likewise. * tests/tap-no-merge-stdout-stderr.test: Likewise. * tests/tap-no-spurious-summary.test: Likewise. * tests/tap-no-spurious.test: Likewise. * tests/tap-not-ok-skip.test: Likewise. * tests/tap-numeric-description.test: Likewise. * tests/tap-out-of-order.test: Likewise. * tests/tap-passthrough.test: Likewise. * tests/tap-plan.test: Likewise. * tests/tap-plan-errors.test: Likewise. * tests/tap-plan-corner.test: Likewise. * tests/tap-realtime.test: Likewise. * tests/tap-recheck-logs.test: Likewise. * tests/tap-recheck.test: Likewise. * tests/tap-skip-whole.test: Likewise. * tests/tap-summary.test: Likewise. * tests/tap-todo-skip.test: Likewise. * tests/tap-todo-skip-together.test: Likewise. * tests/tap-todo-skip-whitespace.test: Likewise. * tests/tap-skipall-whitespace.test: Likewise. * tests/tap-unplanned.test: Likewise. * tests/tap-whitespace-normalization.test: Likewise. * tests/tap-with-and-without-number.test: Likewise. * tests/tap-xfail-tests.test: Likewise. * tests/tap-bad-prog.test: New xfailing test. * tests/tap-color.test: Likewise. * tests/tap-plan-corner2.test: Likewise. * tests/tap-message-0.test: Likewise. * tests/tap-signal.test: Likewise. * tests/Makefile.am (TESTS, XFAIL_TESTS): Update. --- ChangeLog | 55 +++ doc/automake.texi | 94 ++++- lib/Makefile.am | 2 +- lib/Makefile.in | 2 +- lib/tap-driver | 363 ++++++++++++++++++ tests/Makefile.am | 51 +++ tests/Makefile.in | 51 ++- tests/tap-autonumber.test | 94 +++++ tests/tap-bad-prog.test | 69 ++++ tests/tap-bailout.test | 155 ++++++++ tests/tap-basic.test | 177 +++++++++ tests/tap-color.test | 178 +++++++++ tests/tap-deps.test | 98 +++++ tests/tap-diagnostic.test | 142 +++++++ tests/tap-empty-diagnostic.test | 59 +++ tests/tap-empty.test | 73 ++++ tests/tap-escape-directive.test | 57 +++ tests/tap-exit.test | 70 ++++ tests/tap-fancy.test | 151 ++++++++ tests/tap-fancy2.test | 147 +++++++ tests/tap-global-result.test | 128 ++++++ tests/tap-html.test | 93 +++++ tests/tap-log.test | 174 +++++++++ tests/tap-merge-stdout-stderr.test | 85 ++++ tests/tap-message-0.test | 84 ++++ tests/tap-more.test | 160 ++++++++ tests/tap-more2.test | 93 +++++ tests/tap-no-merge-stdout-stderr.test | 66 ++++ tests/tap-no-spurious-summary.test | 65 ++++ tests/tap-no-spurious.test | 114 ++++++ tests/tap-not-ok-skip.test | 57 +++ tests/tap-numeric-description.test | 86 +++++ tests/tap-out-of-order.test | 98 +++++ tests/tap-passthrough-exit.test | 79 ++++ tests/tap-passthrough.test | 202 ++++++++++ tests/tap-plan-corner.test | 81 ++++ tests/tap-plan-corner2.test | 67 ++++ tests/tap-plan-errors.test | 110 ++++++ tests/tap-plan.test | 69 ++++ tests/tap-realtime.test | 115 ++++++ tests/tap-recheck-logs.test | 132 +++++++ tests/tap-recheck.test | 217 +++++++++++ tests/tap-signal.test | 65 ++++ tests/tap-skip-whole-whitespace.test | 68 ++++ tests/tap-skip-whole.test | 100 +++++ tests/tap-summary.test | 363 ++++++++++++++++++ tests/tap-todo-skip-together.test | 57 +++ tests/tap-todo-skip-whitespace.test | 118 ++++++ tests/tap-todo-skip.test | 109 ++++++ tests/tap-unplanned.test | 177 +++++++++ tests/tap-whitespace-normalization.test | 181 +++++++++ tests/tap-with-and-without-number.test | 68 ++++ tests/tap-xfail-tests.test | 85 ++++ .../test-driver-custom-multitest-recheck.test | 1 + 54 files changed, 5845 insertions(+), 10 deletions(-) create mode 100755 lib/tap-driver create mode 100755 tests/tap-autonumber.test create mode 100755 tests/tap-bad-prog.test create mode 100755 tests/tap-bailout.test create mode 100755 tests/tap-basic.test create mode 100755 tests/tap-color.test create mode 100755 tests/tap-deps.test create mode 100755 tests/tap-diagnostic.test create mode 100755 tests/tap-empty-diagnostic.test create mode 100755 tests/tap-empty.test create mode 100755 tests/tap-escape-directive.test create mode 100755 tests/tap-exit.test create mode 100755 tests/tap-fancy.test create mode 100755 tests/tap-fancy2.test create mode 100755 tests/tap-global-result.test create mode 100755 tests/tap-html.test create mode 100755 tests/tap-log.test create mode 100755 tests/tap-merge-stdout-stderr.test create mode 100755 tests/tap-message-0.test create mode 100755 tests/tap-more.test create mode 100755 tests/tap-more2.test create mode 100755 tests/tap-no-merge-stdout-stderr.test create mode 100755 tests/tap-no-spurious-summary.test create mode 100755 tests/tap-no-spurious.test create mode 100755 tests/tap-not-ok-skip.test create mode 100755 tests/tap-numeric-description.test create mode 100755 tests/tap-out-of-order.test create mode 100755 tests/tap-passthrough-exit.test create mode 100755 tests/tap-passthrough.test create mode 100755 tests/tap-plan-corner.test create mode 100755 tests/tap-plan-corner2.test create mode 100755 tests/tap-plan-errors.test create mode 100755 tests/tap-plan.test create mode 100755 tests/tap-realtime.test create mode 100755 tests/tap-recheck-logs.test create mode 100755 tests/tap-recheck.test create mode 100755 tests/tap-signal.test create mode 100755 tests/tap-skip-whole-whitespace.test create mode 100755 tests/tap-skip-whole.test create mode 100755 tests/tap-summary.test create mode 100755 tests/tap-todo-skip-together.test create mode 100755 tests/tap-todo-skip-whitespace.test create mode 100755 tests/tap-todo-skip.test create mode 100755 tests/tap-unplanned.test create mode 100755 tests/tap-whitespace-normalization.test create mode 100755 tests/tap-with-and-without-number.test create mode 100755 tests/tap-xfail-tests.test diff --git a/ChangeLog b/ChangeLog index 1a0569955..c7e1cc403 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,58 @@ +2011-07-18 Stefano Lattarini + + tap: add experimental TAP-aware driver + * doc/automake.texi (Using the TAP test protocol): New section. + (Overview of Custom Test Drivers Support): Minor updates. + * lib/tap-driver: New script, TAP-aware test driver for Automake; + implemented in perl and based on TAP::Parser. + * lib/Makefile.am (dist_script_DATA): Add it. + * tests/tap-autonumber.test: New test. + * tests/tap-bailout.test: Likewise. + * tests/tap-basic.test: Likewise. + * tests/tap-deps.test: Likewise. + * tests/tap-diagnostic.test: Likewise. + * tests/tap-empty.test: Likewise. + * tests/tap-empty-diagnostic.test: Likewise. + * tests/tap-escape-directive.test: Likewise. + * tests/tap-exit.test: Likewise. + * tests/tap-fancy.test: Likewise. + * tests/tap-fancy2.test: Likewise. + * tests/tap-global-result.test: Likewise. + * tests/tap-html.test: Likewise. + * tests/tap-log.test: Likewise. + * tests/tap-merge-stdout-stderr.test: Likewise. + * tests/tap-more.test: Likewise. + * tests/tap-more2.test: Likewise. + * tests/tap-no-merge-stdout-stderr.test: Likewise. + * tests/tap-no-spurious-summary.test: Likewise. + * tests/tap-no-spurious.test: Likewise. + * tests/tap-not-ok-skip.test: Likewise. + * tests/tap-numeric-description.test: Likewise. + * tests/tap-out-of-order.test: Likewise. + * tests/tap-passthrough.test: Likewise. + * tests/tap-plan.test: Likewise. + * tests/tap-plan-errors.test: Likewise. + * tests/tap-plan-corner.test: Likewise. + * tests/tap-realtime.test: Likewise. + * tests/tap-recheck-logs.test: Likewise. + * tests/tap-recheck.test: Likewise. + * tests/tap-skip-whole.test: Likewise. + * tests/tap-summary.test: Likewise. + * tests/tap-todo-skip.test: Likewise. + * tests/tap-todo-skip-together.test: Likewise. + * tests/tap-todo-skip-whitespace.test: Likewise. + * tests/tap-skipall-whitespace.test: Likewise. + * tests/tap-unplanned.test: Likewise. + * tests/tap-whitespace-normalization.test: Likewise. + * tests/tap-with-and-without-number.test: Likewise. + * tests/tap-xfail-tests.test: Likewise. + * tests/tap-bad-prog.test: New xfailing test. + * tests/tap-color.test: Likewise. + * tests/tap-plan-corner2.test: Likewise. + * tests/tap-message-0.test: Likewise. + * tests/tap-signal.test: Likewise. + * tests/Makefile.am (TESTS, XFAIL_TESTS): Update. + 2011-07-07 Stefano Lattarini test defs: new auxiliary function 'count_test_results' diff --git a/doc/automake.texi b/doc/automake.texi index 16804cdb8..0c75a22c6 100644 --- a/doc/automake.texi +++ b/doc/automake.texi @@ -320,6 +320,7 @@ Support for test suites * Simple Tests:: Listing programs and scripts in @code{TESTS} * Simple Tests using parallel-tests:: More powerful test driver * Custom Test Drivers:: Writing and using custom test drivers +* Using the TAP test protocol:: Integrating test scripts that use the TAP protocol * DejaGnu Tests:: Interfacing with the external testing framework * Install Tests:: Running tests on installed packages @@ -8656,6 +8657,7 @@ are very similar. * Simple Tests:: Listing programs and scripts in @code{TESTS} * Simple Tests using parallel-tests:: More powerful test driver * Custom Test Drivers:: Writing and using custom test drivers +* Using the TAP test protocol:: Integrating test scripts that use the TAP protocol * DejaGnu Tests:: Interfacing with the external testing framework * Install Tests:: Running tests on installed packages @end menu @@ -9037,12 +9039,9 @@ or work around the issue with inference rules to generate the tests. @subsection Overview of Custom Test Drivers Support Starting form Automake version 1.12, the @option{parallel-tests} harness -allows the package authors to use third-party custom test drivers, in case -the -@c FIXME: this should become "default ones" once we have TAP and Subunit -default one is inadequate for their purposes. -@c FIXME: add this once we have TAP and Subunit -@c "or do not support their testing protocol of choice." +allows the package authors to use third-party custom test drivers, in +case the default ones are inadequate for their purposes, or do not support +their testing protocol of choice. A custom test driver is expected to properly run the test programs passed to it, including the command-line arguments passed to it, @@ -9336,6 +9335,89 @@ is not the case, the HTML generation will not work, although all the other functionalities of the Automake testsuite harness should remain untouched, and continue to work correctly. +@node Using the TAP test protocol +@section Using the TAP test protocol + +Brief introduction to TAP and its philosophy. + +Real-word uses of TAP: testsuite of @command{perl} and of many +perl modules, and of @command{git}. Third-party libraries and +utilities that can generate TAP (tell also to look at links +below). + +@quotation +A harness must only read TAP output from standard output and not +from standard error. Lines written to standard output matching +@code{/^(not )?ok\b/} must be interpreted as test lines. All other +lines must not be considered test output. +@end quotation + +@noindent +Here are some links to more extensive official or third-party documentation +and resources: +@itemize @bullet +@item +@uref{http://search.cpan.org/~petdance/Test-Harness/lib/Test/Harness/TAP.pod, + @samp{Test::Harness::TAP}}, +the (mostly) official documentation about the TAP format and protocol. +@item +@uref{http://search.cpan.org/~andya/Test-Harness/bin/prove, + @command{prove}}, +the most famous command-line TAP test driver, included in the distribution +of @command{perl} and +@uref{http://search.cpan.org/~andya/Test-Harness/lib/Test/Harness.pm, + @samp{Test::Harness}}. +@item +The @uref{http://testanything.org/wiki/index.php/Main_Page,TAP wiki}. +@item +A ``gentle introduction'' to testing for perl coders: +@uref{http://search.cpan.org/dist/Test-Simple/lib/Test/Tutorial.pod, + @samp{Test::Tutorial}}. +@item +@uref{http://search.cpan.org/~mschwern/Test-Simple/lib/Test/Simple.pm, + @samp{Test::Simple}} +and +@uref{http://search.cpan.org/~mschwern/Test-Simple/lib/Test/More.pm, + @samp{Test::More}}, +the standard perl testing libraries, which are based on TAP. +@item +@uref{http://www.eyrie.org/~eagle/software/c-tap-harness/,C TAP Harness}, +a C-based project implementing both a TAP producer and a TAP consumer. +@end itemize + +Give example of output from @command{prove} command (when used in +``verbose'' mode, with option @option{-v}), to which we'd like to +remain somewhat compatible compatible, while also (and foremost) +being compatible with the default @option{parallel-tests} driver. + +Example of output from out own TAP testsuite driver. Point out +similarities with the output from @command{prove --verbose} and +from the default @option{parallel-tests} driver. + +Option @option{--ignore-exit} causes the driver to ignore the exit +status of the test scripts; by default, the driver will report an +error if the script exit with status != 0. +@emph{TODO}: also add a @option{--no-ignore-exit} option, for +completeness? + +By default, TAP diagnostic (i.e., lines beginning with the @samp{#} +character) are copied only in the @file{.log} file. The option +@option{--comments} causes the driver to display them in the testsuite +progress output too (@emph{TODO}: give example). The option +@option{--no-comments} restore the default behaviour. + +@noindent +Differences and incompatibilities with other TAP parsers and drivers: +@itemize @bullet +@item +A @code{Bail out!} directive doesn't stop the whole testsuite, but only +the test script it occurs into. This doesn't follows TAP specifications, +but on the other hand maximize compatibility (and code sharing) with +the ``hard error'' concept of the default @option{parallel-tests} driver. +@item +@emph{TODO}: there's surely something else ... +@end itemize + @node DejaGnu Tests @section DejaGnu Tests diff --git a/lib/Makefile.am b/lib/Makefile.am index a26d6633e..a9467f1f1 100644 --- a/lib/Makefile.am +++ b/lib/Makefile.am @@ -29,7 +29,7 @@ dist_pkgvdata_DATA = COPYING INSTALL texinfo.tex ansi2knr.c ansi2knr.1 \ scriptdir = $(pkgvdatadir) dist_script_DATA = config.guess config.sub install-sh mdate-sh missing \ mkinstalldirs elisp-comp ylwrap acinstall depcomp compile py-compile \ - symlink-tree ar-lib test-driver + symlink-tree ar-lib test-driver tap-driver EXTRA_DIST = gnupload diff --git a/lib/Makefile.in b/lib/Makefile.in index 5f8aa913b..b60040270 100644 --- a/lib/Makefile.in +++ b/lib/Makefile.in @@ -242,7 +242,7 @@ dist_pkgvdata_DATA = COPYING INSTALL texinfo.tex ansi2knr.c ansi2knr.1 \ scriptdir = $(pkgvdatadir) dist_script_DATA = config.guess config.sub install-sh mdate-sh missing \ mkinstalldirs elisp-comp ylwrap acinstall depcomp compile py-compile \ - symlink-tree ar-lib test-driver + symlink-tree ar-lib test-driver tap-driver EXTRA_DIST = gnupload all: all-recursive diff --git a/lib/tap-driver b/lib/tap-driver new file mode 100755 index 000000000..acfcd5971 --- /dev/null +++ b/lib/tap-driver @@ -0,0 +1,363 @@ +#! /usr/bin/env perl +# Temporary/experimental TAP test driver for Automake. +# TODO: should be rewritten portably (e.g., in awk or shell). + +# ---------------------------------- # +# Imports, static data, and setup. # +# ---------------------------------- # + +use warnings FATAL => 'all'; +use strict; +use Getopt::Long (); +use TAP::Parser; + +my $ME = "tap-driver"; + +my $USAGE = <<'END'; +Usage: + tap-driver [--help|--version] --test-name=NAME --log-file=PATH + [--expect-failure={yes|no}] [--color-tests={yes|no}] + [--enable-hard-errors={yes|no}] [--merge|--no-merge] + [--ignore-exit] [--comments|--no-comments] [--] TEST-COMMAND +The `--test-name' and `--log-file' options are mandatory. +END + +my $HELP = "$ME: TAP-aware test driver for Automake testsuite harness." . + "\n" . $USAGE; + +my $VERSION = '(experimental version)'; + +# ------------------- # +# Global variables. # +# ------------------- # + +my $lineno = 0; # Number of input lines seen so far. +my $testno = 0; # Number of test results seen so far. +my $plan_seen = 0; # Whether the TAP plan has been seen or not. +my $parser; # TAP parser object (will be initialized later). + +# ----------------- # +# Option parsing. # +# ----------------- # + +my %cfg = ( + "color-tests" => 0, + "expect-failure" => 0, + "enable-hard-errors" => 1, + "merge" => 0, + "comments" => 0, + "ignore-exit" => 0, +); + +my $test_script_name = undef; +my $log_file = undef; + +Getopt::Long::GetOptions ( + 'help' => sub { print $HELP; exit 0; }, + 'version' => sub { print "$ME $VERSION"; exit 0; }, + 'test-name=s' => \$test_script_name, + 'log-file=s' => \$log_file, + 'color-tests=s' => \&bool_opt, + 'expect-failure=s' => \&bool_opt, + 'enable-hard-errors=s' => \&bool_opt, + 'comments' => sub { $cfg{"comments"} = 1; }, + 'no-comments' => sub { $cfg{"comments"} = 0; }, + 'merge' => sub { $cfg{"merge"} = 1; }, + 'no-merge' => sub { $cfg{"merge"} = 0; }, + 'ignore-exit' => sub { $cfg{"ignore-exit"} = 1; }, + ) or exit 1; + +# -------------- # +# Subroutines. # +# -------------- # + +sub bool_opt ($$) +{ + my ($opt, $val) = @_; + if ($val =~ /^(?:y|yes)\z/i) + { + $cfg{$opt} = 1; + } + elsif ($val =~ /^(?:n|no)\z/i) + { + $cfg{$opt} = 0; + } + else + { + die "invalid argument '$val' for option '$opt'\n"; + } +} + +TAP_PEEKING : +{ + my @tap_lines = (); + + sub get_tap_line () + { + @tap_lines > 0 ? pop @tap_lines : $parser->next; + } + + sub unget_tap_line ($) + { + push @tap_lines, @_; + } + + sub peek_tap_line () + { + my $res = get_tap_line; + unget_tap_line ($res); + return $res; + } + +} + +TEST_RESULTS : +{ + my (@test_results, %test_results); + + sub add_test_result ($) + { + my $res = shift; + push @test_results, $res; + $test_results{$res} = 1; + } + + sub get_test_results () + { + return @test_results; + } + + # FIXME: this can certainly be improved ... + sub get_global_test_result () + { + my @results = keys %test_results; + return "ERROR" if exists $test_results{"ERROR"}; + return "SKIP" if @results == 1 && $results[0] eq "SKIP"; + return "FAIL" if exists $test_results{"FAIL"}; + return "FAIL" if exists $test_results{"XPASS"}; + return "PASS"; + } + +} + +sub start (@) +{ + # Redirect stderr and stdout to a temporary log file. Save the + # original stdout stream, since we need it to print testsuite + # progress output. + open OLDOUT, ">&STDOUT" or die "duplicating stdout: $!\n"; + open TMP, ">$log_file-t" or die "opening $log_file-t: $!\n"; + open STDOUT, ">&TMP" or die "redirecting stdout: $!\n"; + open STDERR, ">&TMP" or die "redirecting stderr: $!\n"; + $parser = TAP::Parser->new ({ exec => \@_, merge => $cfg{merge} }); + $parser->ignore_exit(1) if $cfg{"ignore-exit"}; +} + +sub finish () +{ + open LOG, ">", $log_file or die "opening $log_file: $!\n"; + + # We need to declare a global test result in order to have + # "make recheck" working. + my $global_result = get_global_test_result; + my $global_result_line = "$global_result: $test_script_name"; + print LOG "$global_result_line\n"; + print LOG "=" x length ($global_result_line) . "\n"; + print LOG "\n"; + + foreach (get_test_results) + { + print LOG ":test-result: $_\n"; + # Add extra trailing empty lines to support reStructuredText -> HTML + # conversion. + print LOG "\n"; + } + + # So that the output from the test script won't be parsed for test + # results (which would potentially expose us to false positives). + print LOG ":test-result: END\n"; + print LOG "\n"; + + close TMP or die "closing $log_file-t: $!\n"; + # FIXME: remove this hack! + my $test_output = `cat $log_file-t && rm -f $log_file-t`; + print LOG $test_output; + close LOG or die "closing $log_file: $!\n"; + + exit 0; +} + +sub stringify_test_result ($) +{ + my $result = shift; + my $PASS = $cfg{"expect-failure"} ? "XPASS": "PASS"; + my $FAIL = $cfg{"expect-failure"} ? "XFAIL": "FAIL"; + if ($result->is_unplanned || $result->number != $testno) + { + return "ERROR"; + } + elsif (!$result->directive) + { + return $result->is_ok ? $PASS: $FAIL; + } + elsif ($result->has_todo) + { + return $result->is_actual_ok ? "XPASS" : "XFAIL"; + } + elsif ($result->has_skip) + { + return $result->is_ok ? "SKIP" : $FAIL; + } + die "INTERNAL ERROR"; # NOTREACHED +} + +sub console_output (@) +{ + return unless @_ > 0; + my $msg = join ("\n", @_) . "\n"; + print OLDOUT $msg; + # Log the result in the log file too, to help debugging (this is + # especially true when said result is a TAP error or "Bail out!"). + print $msg; +} + +sub testuite_error ($) +{ + add_test_result "ERROR"; + console_output "ERROR: $test_script_name - $_[0]"; +} + +sub handle_tap_test ($) +{ + $testno++; + my $test = shift; + + my $test_result = stringify_test_result $test; + my $string = "$test_result: $test_script_name " . $test->number; + + if (my $description = $test->description) + { + $string .= " $description"; + } + if ($test->is_unplanned) + { + $string .= " # UNPLANNED"; + } + elsif ($test->number != $testno) + { + $string .= " # OUT-OF-ORDER (expecting $testno)"; + } + elsif (my $directive = $test->directive) + { + $string .= " # $directive"; + if (my $explanation = $test->explanation) + { + $string .= " $explanation"; + } + } + add_test_result $test_result; + console_output $string; +} + +sub handle_tap_plan ($) +{ + my $plan = shift; + testuite_error "multiple test plans" if $plan_seen; + $plan_seen = 1; + # TAP plan must be either in the first or in the last line. + if ($lineno > 1 && peek_tap_line) + { + testuite_error "test plan in middle of output"; + return; + } + # Nothing more to do, unless the plan contains a SKIP directive. + return + if not defined $plan->directive && length ($plan->directive) > 0; + my $string = "SKIP: $test_script_name"; + if (my $explanation = $plan->explanation) + { + $string .= " - $explanation"; + } + add_test_result "SKIP"; + console_output $string; + finish; +} + +sub handle_tap_bailout ($) +{ + my ($bailout, $msg) = ($_[0], "Bail out!"); + $msg .= " " . $bailout->explanation if $bailout->explanation; + testuite_error $msg; + finish; +} + +sub handle_tap_comment ($) +{ + return unless $cfg{comments}; + my $comment = $_[0]->comment; + console_output "# $test_script_name: $comment" + if length $comment; +} + +sub main (@) +{ + start @_; + + while (defined (my $cur = get_tap_line)) + { + # Verbatim copy any input line into the log file. + print $cur->raw . "\n"; + $lineno++; + if ($cur->is_plan) + { + handle_tap_plan ($cur); + } + elsif ($cur->is_test) + { + handle_tap_test ($cur); + } + elsif ($cur->is_comment) + { + handle_tap_comment ($cur); + } + elsif ($cur->is_bailout) + { + handle_tap_bailout ($cur); + } + } + if (!$plan_seen) + { + testuite_error "missing test plan"; + } + elsif ($parser->tests_planned != $parser->tests_run) + { + my ($planned, $run) = ($parser->tests_planned, $parser->tests_run); + my $bad_amount = $run > $planned ? "many" : "few"; + testuite_error (sprintf "too %s tests run (expected %d, got %d)", + $bad_amount, $planned, $run); + } + # TODO: we should probably use $parser->wait here, to catch signals too + if ($parser->exit != 0) + { + my $msg = sprintf "exited with status %d", $parser->exit; + if ($cfg{"ignore-exit"}) + { + # Log the exit status of the script anyway, even if it is not + # considered to be an error, to help debugging. + print "INFO: $test_script_name - $msg\n"; + } + else + { + testuite_error $msg;; + } + } + finish; +} + +# ----------- # +# Main code. # +# ----------- # + +main @ARGV; + +# vim: ft=perl ts=4 sw=4 et diff --git a/tests/Makefile.am b/tests/Makefile.am index 0d41a1c1e..2ef4d7005 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -27,6 +27,11 @@ gcj6.test \ override-conditional-2.test \ pr8365-remake-timing.test \ yacc-dist-nobuild-subdir.test \ +tap-bad-prog.test \ +tap-color.test \ +tap-plan-corner2.test \ +tap-message-0.test \ +tap-signal.test \ txinfo5.test @@ -980,6 +985,52 @@ suffix13.test \ symlink.test \ symlink2.test \ syntax.test \ +tap-autonumber.test \ +tap-bad-prog.test \ +tap-bailout.test \ +tap-basic.test \ +tap-color.test \ +tap-deps.test \ +tap-diagnostic.test \ +tap-empty.test \ +tap-empty-diagnostic.test \ +tap-escape-directive.test \ +tap-exit.test \ +tap-signal.test \ +tap-fancy.test \ +tap-fancy2.test \ +tap-global-result.test \ +tap-html.test \ +tap-log.test \ +tap-more.test \ +tap-more2.test \ +tap-message-0.test \ +tap-merge-stdout-stderr.test \ +tap-no-merge-stdout-stderr.test \ +tap-no-spurious-summary.test \ +tap-no-spurious.test \ +tap-not-ok-skip.test \ +tap-numeric-description.test \ +tap-out-of-order.test \ +tap-passthrough.test \ +tap-passthrough-exit.test \ +tap-plan.test \ +tap-plan-errors.test \ +tap-plan-corner.test \ +tap-plan-corner2.test \ +tap-realtime.test \ +tap-recheck-logs.test \ +tap-recheck.test \ +tap-summary.test \ +tap-skip-whole.test \ +tap-skip-whole-whitespace.test \ +tap-todo-skip.test \ +tap-todo-skip-whitespace.test \ +tap-todo-skip-together.test \ +tap-unplanned.test \ +tap-whitespace-normalization.test \ +tap-with-and-without-number.test \ +tap-xfail-tests.test \ tags.test \ tags2.test \ tagsub.test \ diff --git a/tests/Makefile.in b/tests/Makefile.in index c72b1cf25..c5e100951 100644 --- a/tests/Makefile.in +++ b/tests/Makefile.in @@ -273,8 +273,9 @@ EXTRA_DIST = ChangeLog-old gen-parallel-tests instspc-tests.sh \ extract-testsuite-summary XFAIL_TESTS = all.test auxdir2.test cond17.test gcj6.test \ override-conditional-2.test pr8365-remake-timing.test \ - yacc-dist-nobuild-subdir.test txinfo5.test \ - $(instspc_xfail_tests) + yacc-dist-nobuild-subdir.test tap-bad-prog.test tap-color.test \ + tap-plan-corner2.test tap-message-0.test tap-signal.test \ + txinfo5.test $(instspc_xfail_tests) parallel_tests = backcompat5-p.test check-exported-srcdir-p.test \ check-tests-in-builddir-p.test check-tests_environment-p.test \ check-p.test check11-p.test check12-p.test check2-p.test \ @@ -1227,6 +1228,52 @@ suffix13.test \ symlink.test \ symlink2.test \ syntax.test \ +tap-autonumber.test \ +tap-bad-prog.test \ +tap-bailout.test \ +tap-basic.test \ +tap-color.test \ +tap-deps.test \ +tap-diagnostic.test \ +tap-empty.test \ +tap-empty-diagnostic.test \ +tap-escape-directive.test \ +tap-exit.test \ +tap-signal.test \ +tap-fancy.test \ +tap-fancy2.test \ +tap-global-result.test \ +tap-html.test \ +tap-log.test \ +tap-more.test \ +tap-more2.test \ +tap-message-0.test \ +tap-merge-stdout-stderr.test \ +tap-no-merge-stdout-stderr.test \ +tap-no-spurious-summary.test \ +tap-no-spurious.test \ +tap-not-ok-skip.test \ +tap-numeric-description.test \ +tap-out-of-order.test \ +tap-passthrough.test \ +tap-passthrough-exit.test \ +tap-plan.test \ +tap-plan-errors.test \ +tap-plan-corner.test \ +tap-plan-corner2.test \ +tap-realtime.test \ +tap-recheck-logs.test \ +tap-recheck.test \ +tap-summary.test \ +tap-skip-whole.test \ +tap-skip-whole-whitespace.test \ +tap-todo-skip.test \ +tap-todo-skip-whitespace.test \ +tap-todo-skip-together.test \ +tap-unplanned.test \ +tap-whitespace-normalization.test \ +tap-with-and-without-number.test \ +tap-xfail-tests.test \ tags.test \ tags2.test \ tagsub.test \ diff --git a/tests/tap-autonumber.test b/tests/tap-autonumber.test new file mode 100755 index 000000000..745a6398f --- /dev/null +++ b/tests/tap-autonumber.test @@ -0,0 +1,94 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - unnumbered tests are OK, as long as their final total number +# agrees with the plan +# - test results without number get automatically numbered in the +# console progress output + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = all.test +EXTRA_DIST = $(TESTS) +END + +cat > all.test <<'END' +1..14 +ok +ok foo +ok - foo2 +not ok +not ok bar +not ok - bar2 +; See that we can intermingle different kind of results without +; messing up the autonumbering +ok +ok # TODO +not ok # TODO who cares? +ok +not ok +ok muuu # SKIP +not ok +ok +END + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +$MAKE check >stdout && { cat stdout; Exit 1; } +cat stdout +count_test_results total=14 pass=6 fail=5 xpass=1 xfail=1 skip=1 error=0 + +cat > exp <<'END' +PASS: all.test 1 +PASS: all.test 2 foo +PASS: all.test 3 - foo2 +FAIL: all.test 4 +FAIL: all.test 5 bar +FAIL: all.test 6 - bar2 +PASS: all.test 7 +XPASS: all.test 8 # TODO +XFAIL: all.test 9 # TODO who cares? +PASS: all.test 10 +FAIL: all.test 11 +SKIP: all.test 12 muuu # SKIP +FAIL: all.test 13 +PASS: all.test 14 +END + +$FGREP ': all.test' stdout > got + +cat exp +cat got +diff exp got + +: diff --git a/tests/tap-bad-prog.test b/tests/tap-bad-prog.test new file mode 100755 index 000000000..af54b2e98 --- /dev/null +++ b/tests/tap-bad-prog.test @@ -0,0 +1,69 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - missing, unreadable, or not-executable test scripts cause proper +# error reports + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TESTS = none.test noread.test noexec.test +none.test: +END + +cat > noexec.test <<'END' +#!/bin/sh +echo 1..1 +ok 1 +END + +./noexec.test && skip_ "can't have non-executable files" + +cp noexec.test noread.test +chmod a-r noread + +if test -r noread.test || cat noread.test; then + skip_ "can't have readonly files" +fi + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +$MAKE check >stdout && { cat stdout; Exit 1; } +cat stdout + +# FIXME: maybe better error messages? +grep '^ERROR: none.test' stdout +grep '^ERROR: noexec.test' stdout +grep '^ERROR: noread.test' stdout + +count_test_results total=3 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=3 + +: diff --git a/tests/tap-bailout.test b/tests/tap-bailout.test new file mode 100755 index 000000000..6c0a588f9 --- /dev/null +++ b/tests/tap-bailout.test @@ -0,0 +1,155 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# Basic TAP test protocol support: +# - "Bail out!" magic + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = a.test b.test c.test d.test e.test +END + +: > exp + +#------------------------------------------------------------------ + +# Bailout without explanation. + +cat > a.test <> exp < b.test <> exp < c.test <> exp < d.test <> exp + +# error += 1 + +#------------------------------------------------------------------ + +# TAP input comprised only of a bailout directive. + +cat > e.test <> exp + +# error += 1 + +#------------------------------------------------------------------ + +# Doing the sums above, we have: +test_counts='total=12 pass=3 fail=1 xpass=1 xfail=1 skip=1 error=5' + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +$MAKE check >stdout && { cat stdout; Exit 1; } +cat stdout + +count_test_results $test_counts + +LC_ALL=C sort exp > t +mv -f t exp + +# We need the sort below to account for parallel make usage. +grep ': [abcde]\.test' stdout | LC_ALL=C sort > got + +cat exp +cat got +diff exp got + +: diff --git a/tests/tap-basic.test b/tests/tap-basic.test new file mode 100755 index 000000000..26a749d4a --- /dev/null +++ b/tests/tap-basic.test @@ -0,0 +1,177 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# Basic TAP support: +# - LOG_COMPILER support; +# - basic support for TODO and SKIP directives, and "Bail out!" magic; +# - testsuite progress output on console; +# - runtime overriding of TESTS and TEST_LOGS; +# - correct counts of test results (both in summary and in progress +# output on console). +# Note that some of the features checked here are checked in other +# test cases too, usually in a more thorough and detailed way. + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +## Defining LOG_COMPILER should work and not intefere with the +## tap-driver script. +TEST_LOG_COMPILER = cat +TESTS = success.test + +ok.test: + echo '1..3' > $@-t + echo 'ok 1' >> $@-t + echo 'not ok 2 # TODO' >>$@-t + echo 'ok 3 # SKIP' >>$@-t + cat $@-t ;: For debugging. + mv -f $@-t $@ +END + +cat > success.test << 'END' +1..20 +ok 1 +ok 2 two +ok 3 - three +ok 4 four four +not ok 5 +not ok 6 six +not ok 7 - seven +not ok 8 eight eight +ok 9 # TODO +ok 10 ten # TODO +ok 11 - eleven # TODO +ok 12 twelve twelve # TODO +not ok 13 # TODO +not ok 14 fourteen # TODO +not ok 15 - fifteen # TODO +not ok 16 sixteen sixteen # TODO +ok 17 # SKIP +ok 18 eighteen # SKIP +ok 19 - nineteen # SKIP +ok 20 twenty twenty # SKIP +END + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +# Basilar usage and testsuite progress output. + +$MAKE check >stdout && { cat stdout; Exit 1; } +cat stdout + +count_test_results total=20 pass=4 fail=4 xpass=4 xfail=4 skip=4 error=0 + +test -f success.log +test -f test-suite.log + +cat > exp << 'END' +PASS: success.test 1 +PASS: success.test 2 two +PASS: success.test 3 - three +PASS: success.test 4 four four +FAIL: success.test 5 +FAIL: success.test 6 six +FAIL: success.test 7 - seven +FAIL: success.test 8 eight eight +XPASS: success.test 9 # TODO +XPASS: success.test 10 ten # TODO +XPASS: success.test 11 - eleven # TODO +XPASS: success.test 12 twelve twelve # TODO +XFAIL: success.test 13 # TODO +XFAIL: success.test 14 fourteen # TODO +XFAIL: success.test 15 - fifteen # TODO +XFAIL: success.test 16 sixteen sixteen # TODO +SKIP: success.test 17 # SKIP +SKIP: success.test 18 eighteen # SKIP +SKIP: success.test 19 - nineteen # SKIP +SKIP: success.test 20 twenty twenty # SKIP +END + +$FGREP ': success.test' stdout > got + +cat exp +cat got +diff exp got + +# Override TESTS from the command line. + +rm -f *.log *.test + +cat > bail.test <<'END' +1..1 +Bail out! +ok 1 +END + +TESTS=bail.test $MAKE -e check >stdout && { cat stdout; Exit 1; } +cat stdout + +count_test_results total=1 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=1 + +test ! -f success.log +test -f bail.log +test -f test-suite.log + +grep '^ERROR: bail\.test - Bail out!' stdout +grep '^PASS:' stdout && Exit 1 +test `$FGREP -c ': bail.test' stdout` -eq 1 +$FGREP 'success.test' stdout && Exit 1 + +# Override TEST_LOGS from the command line, making it point to a test +# (ok.test) that have to be generated at make time. + +rm -f *.log *.test + +TEST_LOGS=ok.log $MAKE -e check >stdout || { cat stdout; Exit 1; } +cat stdout + +count_test_results total=3 pass=1 fail=0 xpass=0 xfail=1 skip=1 error=0 + +test -f ok.test +test -f ok.log +test ! -f success.log +test ! -f bail.log +test -f test-suite.log + +$EGREP '(bail|success)\.test' stdout && Exit 1 + +cat > exp << 'END' +PASS: ok.test 1 +XFAIL: ok.test 2 # TODO +SKIP: ok.test 3 # SKIP +END + +$FGREP ': ok.test' stdout > got + +cat exp +cat got +diff exp got + +: diff --git a/tests/tap-color.test b/tests/tap-color.test new file mode 100755 index 000000000..82c0b5162 --- /dev/null +++ b/tests/tap-color.test @@ -0,0 +1,178 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - colorization of TAP results and diagnostic messages + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +TERM=ansi; export TERM + +esc='' +# Escape `[' for grep, below. +red="$esc\[0;31m" +grn="$esc\[0;32m" +lgn="$esc\[1;32m" +blu="$esc\[1;34m" +mgn="$esc\[0;35m" +std="$esc\[m" + +# Check that grep can parse nonprinting characters. +# BSD 'grep' works from a pipe, but not a seekable file. +# GNU or BSD 'grep -a' works on files, but is not portable. +case `echo "$std" | grep .` in + "$std") ;; + *) echo "$me: grep can't parse nonprinting characters" >&2; Exit 77;; +esac + +cat >> configure.in < Makefile.am << 'END' +AUTOMAKE_OPTIONS = color-tests +AM_TEST_LOG_DRIVER_FLAGS = --comments +TEST_LOG_COMPILER = cat +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TESTS = all.test skip.test bail.test badplan.test noplan.test \ + few.test many.test order.test +END + +cat > all.test << 'END' +1..5 +ok 1 - foo +# Hi! I shouldn't be colorized! +not ok 2 - bar # TODO td +ok 3 - baz # SKIP sk +not ok 4 - quux +ok 5 - zardoz # TODO +END + +cat > skip.test << 'END' +1..0 # SKIP whole script +END + +cat > bail.test << 'END' +1..1 +ok 1 +Bail out! +END + +cat > badplan.test << 'END' +# foo +1..1 +ok 1 +END + +cat > noplan.test << 'END' +ok 1 +END + +cat > few.test << 'END' +1..2 +ok 1 +END + +cat > many.test << 'END' +1..1 +ok 1 +ok 2 +END + +cat > order.test << 'END' +1..1 +ok 5 +END + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +AM_COLOR_TESTS=always $MAKE check >stdout && { cat stdout; Exit 1; } +cat stdout + +test_color () +{ + # Not a useless use of cat; see above comments about grep. + cat stdout | grep "^${grn}PASS${std}: all\.test 1 - foo$" + cat stdout | grep "^${lgn}XFAIL${std}: all\.test 2 - bar # TODO td$" + cat stdout | grep "^${blu}SKIP${std}: all\.test 3 - baz # SKIP sk$" + cat stdout | grep "^${red}FAIL${std}: all\.test 4 - quux$" + cat stdout | grep "^${red}XPASS${std}: all\.test 5 - zardoz # TODO$" + cat stdout | grep "^${blu}SKIP${std}: skip\.test - whole script$" + cat stdout | grep "^${grn}PASS${std}: bail\.test 1$" + cat stdout | grep "^${mgn}ERROR${std}: bail\.test - Bail out!$" + cat stdout | grep "^${mgn}ERROR${std}: badplan\.test - test plan in middle of output$" + cat stdout | grep "^${mgn}ERROR${std}: noplan\.test - missing test plan$" + cat stdout | grep "^${mgn}ERROR${std}: few.test - too few tests run (expected 2, got 1)$" + cat stdout | grep "^${mgn}ERROR${std}: many.test - too many tests run (expected 1, got 2)$" + cat stdout | grep "^${mgn}ERROR${std}: many.test 2 # UNPLANNED$" + cat stdout | grep "^${mgn}ERROR${std}: order.test 5 # OUT-OF-ORDER (expecting 1)$" + # Diagnostic messages shouldn't be colorized. + cat stdout | grep "^# all\.test: Hi! I shouldn't be colorized!$" + : +} + +test_no_color () +{ + # With make implementations that, like Solaris make, in case of errors + # print the whole failing recipe on standard output, we should content + # ourselves with a laxer check, to avoid false positives. + # Keep this in sync with lib/am/check.am:$(am__color_tests). + if $FGREP '= Xalways || test -t 1 ' stdout; then + # Extra verbose make, resort to laxer checks. + # But we also want to check that the testsuite summary is not unduly + # colorized. + ( + set +e # In case some grepped regex below isn't matched. + # Not a useless use of cat; see above comments about grep. + cat stdout | grep "TOTAL.*:" + cat stdout | grep "PASS.*:" + cat stdout | grep "FAIL.*:" + cat stdout | grep "SKIP.*:" + cat stdout | grep "XFAIL.*:" + cat stdout | grep "XPASS.*:" + cat stdout | grep "ERROR.*:" + cat stdout | grep "^#" + cat stdout | grep 'test.*expected' + cat stdout | grep 'test.*not run' + cat stdout | grep '====' + cat stdout | grep '[Ss]ee .*test-suite\.log' + cat stdout | grep '[Tt]estsuite summary' + ) | grep "$esc" && Exit 1 + : For shells with broken 'set -e' + else + cat stdout | grep "$esc" && Exit 1 + : For shells with broken 'set -e' + fi +} + +AM_COLOR_TESTS=always $MAKE check >stdout && { cat stdout; Exit 1; } +cat stdout +test_color + +$MAKE -e check >stdout && { cat stdout; Exit 1; } +cat stdout +test_no_color + +: diff --git a/tests/tap-deps.test b/tests/tap-deps.test new file mode 100755 index 000000000..ec744857b --- /dev/null +++ b/tests/tap-deps.test @@ -0,0 +1,98 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# Basic TAP test protocol support: +# - dependencies between test scripts + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +# The tests are *deliberately* listed in inversed order here. +TESTS = c.test b.test a.test +b.log: a.log +c.log: b.log +END + +cat > a.test << 'END' +#!/bin/sh +echo 1..2 +echo ok 1 +sleep '3' # FIXME: quotes to please maintainer-check +echo ok 2 +: > a.run +END + +cat > b.test << 'END' +#!/bin/sh +echo 1..2 +if test -f a.run; then + echo ok 1 +else + echo not ok 1 +fi +sleep '3' # FIXME: quotes to please maintainer-check +echo ok 2 +: > b.run +END + +cat > c.test << 'END' +#!/bin/sh +echo 1..1 +test -f b.run || { echo 'Bail out!'; exit 1; } +echo ok 1 +rm -f a.run b.run +END + +chmod a+x *.test + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +$MAKE check >stdout || { cat stdout; Exit 1; } +cat stdout + +count_test_results total=5 pass=5 fail=0 xpass=0 xfail=0 skip=0 error=0 + +cat > exp << 'END' +PASS: a.test 1 +PASS: a.test 2 +PASS: b.test 1 +PASS: b.test 2 +PASS: c.test 1 +END + +grep ': [abc]\.test' stdout > got + +cat exp +cat got +diff exp got + +# TODO: it would be nice to also redo the checks forcing parallel make... + +: diff --git a/tests/tap-diagnostic.test b/tests/tap-diagnostic.test new file mode 100755 index 000000000..caef65119 --- /dev/null +++ b/tests/tap-diagnostic.test @@ -0,0 +1,142 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - diagnostic messages (TAP lines with leading "#") +# - flags '--comments' and '--no-comments' of the TAP test driver + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = all.test +END + +metacharacters=\''"\$!&()[]<>#;^?*' + +cat > all.test < exp <stdout || { cat stdout; Exit 1; } +cat stdout +$EGREP -i "#.*all\\.test|a comment|(Tests|Shell) " stdout && Exit 1 +count_test_results total=4 pass=2 fail=0 xpass=0 xfail=1 skip=1 error=0 + +echo 'AM_TEST_LOG_DRIVER_FLAGS = --comments' >> Makefile +$MAKE check >stdout || { cat stdout; Exit 1; } +cat stdout +$FGREP ' all.test' stdout > got +cat exp +cat got +diff exp got +count_test_results total=4 pass=2 fail=0 xpass=0 xfail=1 skip=1 error=0 + +TEST_LOG_DRIVER_FLAGS="--no-comments" $MAKE -e check >stdout \ + || { cat stdout; Exit 1; } +cat stdout +$EGREP -i "#.*all\\.test|a comment|(Tests|Shell) " stdout && Exit 1 +count_test_results total=4 pass=2 fail=0 xpass=0 xfail=1 skip=1 error=0 + +# The "#"-prepended lines here shouldn't be parsed as test results. +cat > all.test <stdout || { cat stdout; Exit 1; } +cat stdout +count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0 + +# Diagnostic without leading whitespace, or with extra leading whitespace, +# is ok. Be laxer in the grepping checks, to allow for whitespace +# normalization by the TAP driver. + +ws="[ $tab]" +ws0p="${ws}*" +ws1p="${ws}${ws0p}" + +cat > all.test <stdout || { cat stdout; Exit 1; } +cat stdout +count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0 + +grep "^# all.test:${ws0p}foo$" stdout +grep "^# all.test:${ws0p}bar${ws0p}$" stdout +grep "^# all.test:${ws1p}zardoz${ws0p}$" stdout +grep "^# all.test:${ws1p}foo${ws1p}bar${ws1p}baz${ws0p}$" stdout + +test `grep -c '^# all\.test:' stdout` -eq 4 + +: diff --git a/tests/tap-empty-diagnostic.test b/tests/tap-empty-diagnostic.test new file mode 100755 index 000000000..0edea1392 --- /dev/null +++ b/tests/tap-empty-diagnostic.test @@ -0,0 +1,59 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - empty diagnostic messages are discarder + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +AM_TEST_LOG_DRIVER_FLAGS = --comments +TESTS = all.test +END + +sed 's/\$$//' > all.test <stdout || { cat stdout; Exit 1; } +cat stdout + +grep '#.*all\.test' stdout && Exit 1 +grep "all\.test[ $tab]*:[ $tab]*$" stdout && Exit 1 + +: diff --git a/tests/tap-empty.test b/tests/tap-empty.test new file mode 100755 index 000000000..565c1347f --- /dev/null +++ b/tests/tap-empty.test @@ -0,0 +1,73 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: the following situations should be flagged as errors: +# - empty TAP input +# - blank TAP input + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = all.test +END + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +# Empty TAP input. +: > empty.test + +# Blank TAP input (one empty line). +echo > blank.test + +# Blank TAP input (one whitespace-only line). +cat > white.test < white2.test <stdout && { cat stdout; Exit 1; } + cat stdout + count_test_results total=1 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=1 + grep '^ERROR: all\.test - missing test plan$' stdout +done + +: diff --git a/tests/tap-escape-directive.test b/tests/tap-escape-directive.test new file mode 100755 index 000000000..fb333baa6 --- /dev/null +++ b/tests/tap-escape-directive.test @@ -0,0 +1,57 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - "escape" TODO and SKIP directives (by escaping the "#" character) + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = all.test +END + +cat > all.test <<'END' +1..2 +ok \# TODO +ok \# SKIP +END + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +$MAKE check >stdout || { cat stdout; Exit 1; } +cat stdout + +count_test_results total=2 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=0 + +grep '^PASS: all\.test 1 .*TODO' stdout +grep '^PASS: all\.test 2 .*SKIP' stdout + +: diff --git a/tests/tap-exit.test b/tests/tap-exit.test new file mode 100755 index 000000000..07209d1b8 --- /dev/null +++ b/tests/tap-exit.test @@ -0,0 +1,70 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - an exit status != 0 of a test script causes an hard error +# - the `--ignore-exit' option causes the TAP test driver to ignore +# exit statuses of the test scripts. + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TESTS = +END + +for st in 1 2 77 99; do + unindent > exit${st}.test <> Makefile.am +done + +chmod a+x *.test + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +$MAKE check >stdout && { cat stdout; Exit 1; } +cat stdout +count_test_results total=8 pass=4 fail=0 xpass=0 xfail=0 skip=0 error=4 + +grep '^ERROR: exit1\.test - exited with status 1$' stdout +grep '^ERROR: exit2\.test - exited with status 2$' stdout +grep '^ERROR: exit77\.test - exited with status 77$' stdout +grep '^ERROR: exit99\.test - exited with status 99$' stdout + +echo TEST_LOG_DRIVER_FLAGS = --ignore-exit >> Makefile +$MAKE check >stdout || { cat stdout; Exit 1; } +cat stdout +count_test_results total=4 pass=4 fail=0 xpass=0 xfail=0 skip=0 error=0 + +: diff --git a/tests/tap-fancy.test b/tests/tap-fancy.test new file mode 100755 index 000000000..68612dec0 --- /dev/null +++ b/tests/tap-fancy.test @@ -0,0 +1,151 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: some unusual forms for valid TAP input. +# See also related test 'tap-fancy2.test'. + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = all.test +END + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +# +# From manpage Test::Harness::TAP(3): +# +# ``Lines written to standard output matching /^(not )?ok\b/ must be +# interpreted as test lines. All other lines must not be considered +# test output.'' +# +# Unfortunately, the exact format of TODO and SKIP directives is not as +# clearly described in that manpage; but a simple reverse-engineering of +# the prove(1) utility shows that it is probably given by the perl regex +# /#\s*(TODO|SKIP)\b/. +# + +cat > all.test <stdout && { cat stdout; Exit 1; } +cat stdout + +count_test_results total=21 pass=6 fail=6 xfail=4 xpass=1 skip=4 error=0 + +# +# "Weird" characters support. +# + +# The "#" character might cause confusion w.r.t. TAP directives (TODO, +# SKIP), so we don't attempt to use it. +weirdchars=\''"$!&()[]<>;^?*/@%=,.:' + +cat > all.test <stdout && { cat stdout; Exit 1; } +cat stdout + +count_test_results total=6 pass=1 fail=1 xfail=1 xpass=1 skip=1 error=1 +$FGREP "PASS: all.test 1 $weirdchars" stdout +$FGREP "FAIL: all.test 2 $weirdchars" stdout +$FGREP "XPASS: all.test 3 $weirdchars" stdout +$FGREP "XFAIL: all.test 4 $weirdchars" stdout +$FGREP "SKIP: all.test 5 $weirdchars" stdout +$FGREP "ERROR: all.test - Bail out! $weirdchars" stdout + +# +# Trailing backslashes does not confuse the parser. +# + +bs='\' + +cat > all.test <stdout && { cat stdout; Exit 1; } +cat stdout + +count_test_results total=6 pass=1 fail=1 xfail=1 xpass=1 skip=1 error=1 +grep '^PASS: all\.test 1 \\$' stdout +grep '^FAIL: all\.test 2 \\$' stdout +grep '^XPASS: all\.test 3 # TODO \\$' stdout +grep '^XFAIL: all\.test 4 # TODO \\$' stdout +grep '^SKIP: all\.test 5 # SKIP \\$' stdout +grep '^ERROR: all\.test - Bail out! \\$' stdout + +: diff --git a/tests/tap-fancy2.test b/tests/tap-fancy2.test new file mode 100755 index 000000000..b97d787ea --- /dev/null +++ b/tests/tap-fancy2.test @@ -0,0 +1,147 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: more unusual forms for valid TAP input. +# See also related test 'tap-fancy.test'. + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = all.test +END + +# +# From manpage Test::Harness::TAP(3): +# +# ``Lines written to standard output matching /^(not )?ok\b/ must be +# interpreted as test lines. All other lines must not be considered +# test output.'' +# +# Unfortunately, the exact format of TODO and SKIP directives is not as +# clearly described in that manpage; but a simple reverse-engineering of +# the prove(1) utility shows that it is probably given by the perl regex +# /#\s*(TODO|SKIP)\b/. +# + +# There are 34 values for $str ... +for str in \ + \' \ + '"' \ + '`' \ + '#' \ + '$' \ + '!' \ + '\' \ + '/' \ + '&' \ + '%' \ + '(' \ + ')' \ + '|' \ + '^' \ + '~' \ + '?' \ + '*' \ + '+' \ + '-' \ + ',' \ + ':' \ + ';' \ + '=' \ + '<' \ + '>' \ + '@' \ + '[' \ + ']' \ + '{' \ + '}' \ + '\\' \ + '...' \ + '?[a-zA-Z0-9]*' \ + '*.*' \ +; do + # ... each of them add 1 pass, 1 fail, ... + echo "ok${str}" + echo "not ok${str}" + # ... and (generally) 4 skips, 4 xfails, and 4 xpasses ... + for settings in \ + 'result="ok" directive=SKIP' \ + 'result="not ok" directive=TODO' \ + 'result="ok" directive=TODO' \ + ; do + eval "$settings" + echo "${result}# ${directive}${str}" + # ... but 6 skips, 6 xpasses and 6 xfails are to be removed, since + # they might not work with $str = '#' or $str = '\' ... + if test x"$str" != x'#' && test x"$str" != x'\'; then + echo "${result}${str}#${directive}" + echo "${result}${str}# ${tab}${tab} ${directive}" + echo "${result}${str}#${directive}${str}" + fi + done +done > all.test + +# ... so that we finally have: +pass=34 +fail=34 +xfail=130 # = 4 * 34 - 6 +xpass=130 # = 4 * 34 - 6 +skip=130 # = 4 * 34 - 6 +error=0 +total=`expr $pass + $fail + $xfail + $xpass + $skip` + +# Even nastier! But accordingly to the specifics, it should still work. +for result in 'ok' 'not ok'; do + echo "${result}{[(<#${tab}TODO>)]}" >> all.test +done +echo "ok{[(<#${tab}SKIP>)]}" >> all.test + +# We have to update some test counts. +xfail=`expr $xfail + 1` +xpass=`expr $xpass + 1` +skip=`expr $skip + 1` +total=`expr $total + 3` + +# And add the test plan! +echo 1..$total >> all.test + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +$MAKE check >stdout && { cat stdout; Exit 1; } +cat stdout + +$EGREP '^(PASS|FAIL|SKIP).*#.*TODO' stdout && Exit 1 +$EGREP '^X?(PASS|FAIL).*#.*SKIP' stdout && Exit 1 + +count_test_results total=$total pass=$pass fail=$fail skip=$skip \ + xpass=$xpass xfail=$xfail error=$error + +: diff --git a/tests/tap-global-result.test b/tests/tap-global-result.test new file mode 100755 index 000000000..d2e593ecf --- /dev/null +++ b/tests/tap-global-result.test @@ -0,0 +1,128 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - which global test result derives from different test results +# mixed in a single script? + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = \ + ok.test \ + skip.test skipall.test \ + fail.test fail2.test \ + xpass.test xpass2.test \ + error.test error2.test \ + hodgepodge.test hodgepodge-all.test +END + +cat > ok.test < skip.test <<'END' +1..3 +ok 1 # SKIP +ok 2 # SKIP +ok 3 # SKIP +END + +cat > skipall.test <<'END' +1..0 # SKIP +not ok 1 +END + +cat > fail.test <<'END' +1..1 +not ok 1 +END + +(sed '1s/.*/1..4/' ok.test && echo 'not ok 4') > fail2.test + +cat > xpass.test <<'END' +1..1 +ok 1 # TODO +END + +(sed '1s/.*/1..4/' ok.test && echo 'ok 4 # TODO') > xpass2.test + +echo 'Bail out!' > error.test +(cat ok.test && echo 'Bail out!') > error2.test + +cat > hodgepodge.test <<'END' +1..2 +not ok 1 +ok 2 # TODO +Bail out! +END + +cat > hodgepodge-all.test <<'END' +1..4 +ok 1 +ok 2 # SKIP +not ok 2 # TODO +not ok 3 +ok 4 # TODO +Bail out! +END + +# TODO: add scripts with TAP errors (multiple plans, out-of-order +# tests, etc). + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +$MAKE check >stdout && { cat stdout; Exit 1; } +cat stdout + +for tst in ok skip skipall fail fail2 xpass xpass2 error error2 \ + hodgepodge hodgepodge-all; do + sed -e 2q $tst.log > $tst.res +done + +cat *.res # For debugging. + +grep '^PASS:' ok.res +grep '^SKIP:' skip.res +grep '^SKIP:' skipall.res +grep '^FAIL:' fail.res +grep '^FAIL:' fail2.res +grep '^FAIL:' xpass.res +grep '^FAIL:' xpass2.res +grep '^ERROR:' error.res +grep '^ERROR:' error2.res +grep '^ERROR:' hodgepodge.res +grep '^ERROR:' hodgepodge-all.res + +: diff --git a/tests/tap-html.test b/tests/tap-html.test new file mode 100755 index 000000000..12dcb3689 --- /dev/null +++ b/tests/tap-html.test @@ -0,0 +1,93 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - "check-html" and "recheck-html" targets +# - reStructuredText -> HTML conversion + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = ok.test ko.test +END + +cat > ok.test << 'END' +1..3 +ok 1 - expected success +not ok 2 - expectred failure # TODO +ok 3 - skipped test # SKIP +END + +cat > ko.test << 'END' +1..2 +not ok 1 - unexpected failure +ok 2 - spurious success # TODO +Bail out! hard error +END + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +no_rst2html_error () +{ + $EGREP '(test-suite\.log:[0-9]|WARNING|SEVERE)' output && Exit 1 + : +} + +# Overriding TESTS should work with check-html. +TESTS=ok.test $MAKE -e check-html >output 2>&1 || { cat output; Exit 1; } +cat output +no_rst2html_error +test -f test-suite.html + +mv output stdout +count_test_results total=3 pass=1 fail=0 xpass=0 xfail=1 skip=1 error=0 + +: > older +$sleep +$MAKE check-html >output 2>&1 && { cat output; Exit 1; } +cat output +no_rst2html_error +# test-suite.html should be remake if if it was up-to-date. +test -f test-suite.html +is_newest test-suite.html older + +mv output stdout +count_test_results total=6 pass=1 fail=1 xpass=1 xfail=1 skip=1 error=1 + +rm -f ok.test +$MAKE recheck-html >output 2>&1 && { cat output; Exit 1; } +cat output +no_rst2html_error + +mv output stdout +count_test_results total=3 pass=0 fail=1 xpass=1 xfail=0 skip=0 error=1 + +: diff --git a/tests/tap-log.test b/tests/tap-log.test new file mode 100755 index 000000000..37b5956ad --- /dev/null +++ b/tests/tap-log.test @@ -0,0 +1,174 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - log file creation +# - log file removal +# - stdout and stderr of a TAP-generating test script go in its log file +# - TEST_SUITE_LOG redefinition, at either automake or make time +# - VERBOSE environment variable support + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TESTS = pass.test skip.test xfail.test fail.test xpass.test error.test +TEST_SUITE_LOG = global.log +END + +# Custom markers, for use in grepping checks. +cmarker=::: # comment marker +pmarker=%%% # plain maker + +cat > pass.test <&2 +echo "# $cmarker pass $cmarker" >&2 +echo "ok 1" +END + +cat > skip.test < xfail.test <&2 +echo "# $cmarker xfail $cmarker" >&2 +echo "not ok 1 # TODO" +END + +cat > fail.test < xpass.test <&2 +echo "# $cmarker xpass $cmarker" >&2 +echo "ok 1 # TODO" +END + +cat > error.test <stdout && { cat stdout; Exit 1; } +cat stdout +cat global.log +test ! -f my.log +test ! -f test-suite.log +# Check that VERBOSE causes the global testsuite log to be +# emitted on stdout. +out=`cat stdout` +log=`cat global.log` +case $out in *"$log"*) ;; *) Exit 1;; esac + +touch error2.log test-suite.log my.log +$MAKE mostlyclean +ls -l # For debugging. +test ! -f global.log +test ! -f pass.log +test ! -f fail.log +test ! -f xfail.log +test ! -f xpass.log +test ! -f skip.log +test ! -f error.log +test -f error2.log +test -f test-suite.log +test -f my.log + +rm -f *.log + + +: diff --git a/tests/tap-merge-stdout-stderr.test b/tests/tap-merge-stdout-stderr.test new file mode 100755 index 000000000..e73f0ec05 --- /dev/null +++ b/tests/tap-merge-stdout-stderr.test @@ -0,0 +1,85 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - The Automake TAP driver has an option that instruct it to read TAP +# input also from the stderr of the test command, not only its stdout. + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +AM_TEST_LOG_DRIVER_FLAGS = --comments --merge +TESTS = all.test +END + +: > all.test # Updated later. +chmod a+x all.test + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +cat > all.test <&2 +echo ok 2 +echo "not ok 3 # TODO" >&2 +echo "ok 4 # SKIP" +echo "# foo foo foo" >&2 +END + +$MAKE check >stdout || { cat stdout; Exit 1; } +cat stdout + +count_test_results total=4 pass=2 fail=0 xpass=0 xfail=1 skip=1 error=0 +grep '^# all\.test: foo foo foo' stdout + +cat > all.test <&2 +END + +$MAKE check >stdout && { cat stdout; Exit 1; } +cat stdout + +count_test_results total=2 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=1 + +# See that the option `--no-merge' can override the effect of `--merge'. + +TEST_LOG_DRIVER_FLAGS=--no-merge $MAKE -e check >stdout \ + || { cat stdout; Exit 1; } +cat stdout + +count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0 + + + +: diff --git a/tests/tap-message-0.test b/tests/tap-message-0.test new file mode 100755 index 000000000..9f16171cf --- /dev/null +++ b/tests/tap-message-0.test @@ -0,0 +1,84 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - having "0" as a test description (or TODO or SKIP message) should +# be supported +# Note that a bug in some versions of TAP::Parser causes this not to be +# generally true! + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = all.test +END + +cat > all.test << 'END' +1..10 +ok 1 0 +ok - 0 +not ok 3 0 # TODO +not ok - 0 # TODO +ok 5 0 # SKIP +ok - 0 # SKIP +not ok 7 # TODO 0 +ok 8 # SKIP 0 +not ok 9 0 # TODO 0 +ok 10 0 # SKIP 0 +END + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +$MAKE check >stdout || { cat stdout; Exit 1; } +cat stdout + +count_test_results total=10 pass=2 fail=0 xpass=0 xfail=4 skip=4 error=0 + +cat > exp << 'END' +PASS: all.test 1 0 +PASS: all.test 2 - 0 +XFAIL: all.test 3 0 # TODO +XFAIL: all.test 4 - 0 # TODO +SKIP: all.test 5 0 # SKIP +SKIP: all.test 6 - 0 # SKIP +XFAIL: all.test 7 # TODO 0 +SKIP: all.test 8 # SKIP 0 +XFAIL: all.test 9 0 # TODO 0 +SKIP: all.test 10 0 # SKIP 0 +END + +$FGREP ': all.test' stdout > got + +cat exp +cat got +diff exp got + +: diff --git a/tests/tap-more.test b/tests/tap-more.test new file mode 100755 index 000000000..72fcaac49 --- /dev/null +++ b/tests/tap-more.test @@ -0,0 +1,160 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# More on TAP support: +# - more than one TAP-generating test script in $(TESTS) +# - VPATH builds +# - use with parallel make (if supported) +# - basic use of diagnostic messages (lines beginning with "#") +# - flags for TAP driver defined through AC_SUBST in configure.ac +# - messages generated by the testsuite harness reference the +# correct test script(s) +# - "make distcheck" works + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TESTS = 1.test 2.test 3.test +EXTRA_DIST = $(TESTS) tap-driver +END + +cat > 1.test <<'END' +#! /bin/sh +echo 1..2 +echo ok 1 - mu +if test -f not-skip; then + echo "not ok 2 zardoz" +else + echo "ok 2 zardoz # SKIP" +fi +END + +cat > 2.test <<'END' +#! /bin/sh +echo 1..3 +echo "ok" +echo "not ok # TODO not implemented" +echo "ok 3" +END + +cat > 3.test < Makefile + for run_make in "$MAKE -j3" "$MAKE -j 3" "$MAKE"; do + $run_make && break + done + rm -f Makefile + ;; + esac + else + fatal_ "internal error, invalid value of '$try' for \$try" + fi + + $srcdir/configure + ls -l # For debugging. + + # Success. + $run_make check >stdout || { cat stdout; Exit 1; } + cat stdout + count_test_results total=6 pass=4 fail=0 xpass=0 xfail=1 skip=1 error=0 + grep '^PASS: 1\.test 1 - mu$' stdout + grep '^SKIP: 1\.test 2 zardoz # SKIP$' stdout + test `$FGREP -c '1.test' stdout` -eq 2 + grep '^PASS: 2\.test 1$' stdout + grep '^XFAIL: 2\.test 2 # TODO not implemented$' stdout + grep '^PASS: 2\.test 3$' stdout + test `$FGREP -c '2.test' stdout` -eq 3 + grep '^PASS: 3\.test 1 - blah blah blah$' stdout + grep '^# 3\.test: Some diagnostic$' stdout + test `$FGREP -c '3.test' stdout` -eq 2 + + # Failure. + : > not-skip + : > bail-out + $run_make check >stdout && { cat stdout; Exit 1; } + cat stdout + count_test_results total=7 pass=4 fail=1 xpass=0 xfail=1 skip=0 error=1 + grep '^PASS: 1\.test 1 - mu$' stdout + grep '^FAIL: 1\.test 2 zardoz$' stdout + test `$FGREP -c '1.test' stdout` -eq 2 + grep '^PASS: 2\.test 1$' stdout + grep '^XFAIL: 2\.test 2 # TODO not implemented$' stdout + grep '^PASS: 2\.test 3$' stdout + test `$FGREP -c '2.test' stdout` -eq 3 + grep '^PASS: 3\.test 1 - blah blah blah$' stdout + grep '^# 3\.test: Some diagnostic$' stdout + grep '^ERROR: 3\.test - Bail out! Kernel Panic$' stdout + test `$FGREP -c '3.test' stdout` -eq 3 + + cd $srcdir + +done + +$MAKE distcheck + +: diff --git a/tests/tap-more2.test b/tests/tap-more2.test new file mode 100755 index 000000000..810f9b86c --- /dev/null +++ b/tests/tap-more2.test @@ -0,0 +1,93 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# More on TAP support: +# - more LOG_COMPILER at once for TAP tests +# - binary programs in $(TESTS) +# - interactions with `check_*' variables + +required='cc native' +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_EXTENSIONS = .sh .tap + +tap_driver = $(PERL) $(srcdir)/tap-driver + +LOG_DRIVER = $(tap_driver) +SH_LOG_DRIVER = $(tap_driver) +TAP_LOG_DRIVER = $(tap_driver) + +TAP_LOG_COMPILER = cat +SH_LOG_COMPILER = $(SHELL) +LOG_COMPILER = + +EXTRA_DIST = baz.tap + +check_SCRIPTS = bar.sh +bar.sh: Makefile + echo '#!/bin/sh' > $@-t + echo 'echo 1..1' >> $@-t + echo 'echo "not ok 1 # TODO"' >> $@-t + chmod a-w $@-t && mv -f $@-t $@ +CLEANFILES = bar.sh + +check_PROGRAMS = foo-test +foo_test_SOURCES = foo.c + +TESTS = foo-test $(check_SCRIPTS) baz.tap + +EXTRA_DIST += tap-driver +END + +cat > foo.c <<'END' +#include +int main (void) +{ + printf ("1..1\n"); + printf ("ok 1\n"); + return 0; +} +END + +cat > baz.tap << 'END' +1..1 +ok 1 # SKIP +END + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +for target in check distcheck; do + $MAKE $target >stdout || { cat stdout; Exit 1; } + cat stdout + count_test_results total=3 pass=1 fail=0 xpass=0 xfail=1 skip=1 error=0 +done + +: diff --git a/tests/tap-no-merge-stdout-stderr.test b/tests/tap-no-merge-stdout-stderr.test new file mode 100755 index 000000000..c2995b29d --- /dev/null +++ b/tests/tap-no-merge-stdout-stderr.test @@ -0,0 +1,66 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - By default, TAP input is only from the stdout (and not the stderr) +# of the test command. + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_DRIVER_FLAGS = --comments +TESTS = all.test +END + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +cat > all.test <&2 +echo 'not ok 1 # TODO' >&2 +echo 'ok 1' +echo '# foo foo foo' +echo '# bar bar bar' >&2 +echo 'ok 44 # TODO' >&2 +echo 'Bail out!' >&2 +echo 'ok 2 # SKIP' +END + +chmod a+x all.test + +$MAKE check >stdout || { cat stdout; Exit 1; } +cat stdout + +count_test_results total=2 pass=1 fail=0 xpass=0 xfail=0 skip=1 error=0 +$FGREP 'foo foo foo' stdout +$FGREP 'bar bar bar' stdout && Exit 1 + +: diff --git a/tests/tap-no-spurious-summary.test b/tests/tap-no-spurious-summary.test new file mode 100755 index 000000000..4c358da08 --- /dev/null +++ b/tests/tap-no-spurious-summary.test @@ -0,0 +1,65 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - `:test-results:' directives in test scripts' output doesn't +# originate spurious results in the testsuite summary + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = all.test +END + +cat > all.test <<'END' +1..1 +:test-result: PASS +:test-result: FAIL +:test-result: XPASS +:test-result: XFAIL +:test-result: SKIP +:test-result: ERROR +:test-result: UNKNOWN +ok 1 +END + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +$MAKE check >stdout || { cat stdout; Exit 1; } +cat stdout + +count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0 + +for result in PASS FAIL XPASS XFAIL SKIP ERROR UNKNOWN; do + grep "^ *:test-result: $result$" all.log +done + +: diff --git a/tests/tap-no-spurious.test b/tests/tap-no-spurious.test new file mode 100755 index 000000000..cfe7dbd48 --- /dev/null +++ b/tests/tap-no-spurious.test @@ -0,0 +1,114 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - don't spuriously recognize lines that are "almost" TAP lines as +# real TAP lines + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = all.test +END + +echo 1..5 > all.test + +# The only recognized directives are "TODO" and "SKIP". +# So the following should count as passed tests. +cat >> all.test <> all.test <> all.test <<'END' +bailout +bailout! +bail out +bail out! +Bailout +Bailout! +Bail out + Bail out + Bail out! +#Bail out! +# Bail out! +END + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +$MAKE check >stdout || { cat stdout; Exit 1; } +cat stdout + +count_test_results total=5 pass=5 fail=0 xpass=0 xfail=0 skip=0 error=0 + +: diff --git a/tests/tap-not-ok-skip.test b/tests/tap-not-ok-skip.test new file mode 100755 index 000000000..57b8d7a97 --- /dev/null +++ b/tests/tap-not-ok-skip.test @@ -0,0 +1,57 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - a "not ok # SKIP" line should count as a failure, for consistency +# with the prove(1) utility. + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = all.test +END + +cat > all.test <<'END' +1..4 +not ok # SKIP +not ok 2 # SKIP +not ok - foo # SKIP +not ok 4 - bar # SKIP +END + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +$MAKE check >stdout && { cat stdout; Exit 1; } +cat stdout + +count_test_results total=4 pass=0 fail=4 skip=0 xpass=0 xfail=0 error=0 + +: diff --git a/tests/tap-numeric-description.test b/tests/tap-numeric-description.test new file mode 100755 index 000000000..67b3733ab --- /dev/null +++ b/tests/tap-numeric-description.test @@ -0,0 +1,86 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP result lines whose description is a number. + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = all.test +END + +# Some random numbers to be used as test names. +# These are selected so that $n != n for every n >= 1. +set 5 7 45 11 7 3 6 9 1000 1 +# $1 $2 $3 $4 $5 $6 $7 $8 $9 $10 + +cat > all.test < exp <stdout && { cat stdout; Exit 1; } +cat stdout + +count_test_results total=10 pass=2 fail=2 xpass=2 xfail=2 skip=2 error=0 + +$FGREP ': all.test' stdout > got + +cat exp +cat got +diff exp got + + +: diff --git a/tests/tap-out-of-order.test b/tests/tap-out-of-order.test new file mode 100755 index 000000000..2639e2cb4 --- /dev/null +++ b/tests/tap-out-of-order.test @@ -0,0 +1,98 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - out-of-order test results + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = a.test b.test c.test d.test +END + +cat > a.test < b.test < c.test < d.test <<'END' +1..5 +ok 1 +ok 2 +ok 15 +ok 16 +ok 17 +END + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +$MAKE check >stdout && { cat stdout; Exit 1; } +cat stdout + +count_test_results total=17 pass=8 fail=0 xpass=0 xfail=1 skip=0 error=8 + +test `$FGREP -c ': a.test' stdout` -eq 3 +test `$FGREP -c ': b.test' stdout` -eq 5 +test `$FGREP -c ': c.test' stdout` -eq 4 +test `$FGREP -c ': d.test' stdout` -eq 5 + +grep '^ERROR: a\.test 71 # OUT-OF-ORDER (expecting 3)$' stdout +grep '^ERROR: b\.test 4 foo # OUT-OF-ORDER (expecting 3)$' stdout +grep '^ERROR: b\.test 2 bar # OUT-OF-ORDER (expecting 4)$' stdout +grep '^ERROR: c\.test 4 - zardoz # OUT-OF-ORDER (expecting 3)$' stdout +grep '^ERROR: c\.test 3 # OUT-OF-ORDER (expecting 4)$' stdout +grep '^ERROR: d\.test 15 # OUT-OF-ORDER (expecting 3)$' stdout +grep '^ERROR: d\.test 16 # OUT-OF-ORDER (expecting 4)$' stdout +grep '^ERROR: d\.test 17 # OUT-OF-ORDER (expecting 5)$' stdout + +: diff --git a/tests/tap-passthrough-exit.test b/tests/tap-passthrough-exit.test new file mode 100755 index 000000000..98213a381 --- /dev/null +++ b/tests/tap-passthrough-exit.test @@ -0,0 +1,79 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - non-success exit status of a test script is reported in the +# log file +# See also related test 'tap-passthrough.test'. + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = $(SHELL) +TESTS = +END + +exit_statuses='1 2 77 99' + +for e in $exit_statuses; do + unindent > exit-$e.test <> Makefile.am +done + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +st=0 +$MAKE check || st=$? +for e in $exit_statuses; do cat exit-$e.log; done +cat test-suite.log +test $st -gt 0 || Exit 1 + +for e in $exit_statuses; do + for log in exit-$e.log test-suite.log; do + grep "^ERROR: exit-$e\\.test - exited with status $e$" $log + done +done + +st=0 +env TEST_LOG_DRIVER_FLAGS='--ignore-exit' $MAKE -e check || st=$? +for e in $exit_statuses; do cat exit-$e.log; done +cat test-suite.log +test $st -eq 0 || Exit 1 + +for e in $exit_statuses; do + grep "exit-$e\\.test - exited with status $e$" exit-$e.log +done + +: diff --git a/tests/tap-passthrough.test b/tests/tap-passthrough.test new file mode 100755 index 000000000..ab9b3e55a --- /dev/null +++ b/tests/tap-passthrough.test @@ -0,0 +1,202 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - all input (valid TAP lines, invalid TAP lines, non-TAP lines) +# are passed through in the log file +# - TAP errors are reported in the log file too +# See also related test 'tap-passthrough-exit.test'. + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = +END + +weirdchars=\''"\$@!&()[]<>#;,:.^?*/' + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +# +# Only successful tests. +# + +# The whitespace in this test might be normalized in the testsuite +# progress output, but should be copied verbatim in the log files. +cat > ok.test < tiny.test < ok.test < ko.test < bail.test < skip.test < err.test <. + +# TAP support: +# - some corner cases for TAP plan + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = all.test +END + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +cat > leading-repeated-1.test < trailing-repeated-1.test < leading-repeated-2.test < trailing-repeated-2.test <stdout && { cat stdout; Exit 1; } + cat stdout + count_test_results total=`expr $i + 2` pass=$i fail=0 \ + xpass=0 xfail=0 skip=0 error=2 + grep '^ERROR: all\.test - test plan in middle of output$' stdout + grep '^ERROR: all\.test - multiple test plans$' stdout + done +done + +: diff --git a/tests/tap-plan-corner2.test b/tests/tap-plan-corner2.test new file mode 100755 index 000000000..155cf8586 --- /dev/null +++ b/tests/tap-plan-corner2.test @@ -0,0 +1,67 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - more corner cases for TAP plan + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = all.test +END + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +# The leading blank line is meant. +cat > leading-blank.test < trailing-blank.test <stdout && { cat stdout; Exit 1; } + cat stdout + count_test_results total=3 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 + grep '^ERROR: all\.test - test plan in middle of output$' stdout +done + +: diff --git a/tests/tap-plan-errors.test b/tests/tap-plan-errors.test new file mode 100755 index 000000000..539cb8fe3 --- /dev/null +++ b/tests/tap-plan-errors.test @@ -0,0 +1,110 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: the following situations should be flagged as errors: +# - unmatched test plan (too few tests run) +# - misplaced test plan +# - multiple test plans +# - missing test plan +# Checks about unplanned tests are performed in 'tap-unplanned.test'. +# More checks about corner-cases in TAP plans are performed in +# 'tap-plan-corner.test' and 'tap-plan-corner2.test'. + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = foo.test +END + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +my_check () +{ + cat > foo.test + test -n "$err" || fatal_ "\$err not set before calling my_check" + cat foo.test # For debugging. + $MAKE check >stdout && { cat stdout; Exit 1; } + cat stdout + count_test_results "$@" + grep "^ERROR: foo\\.test - $err$" stdout + unset err +} + +err='too few tests run (expected 2, got 1)' +my_check total=2 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=1 <. + +# TAP support: +# - test scripts with the test plan at the beginning +# - test scripts with the test plan at the end +# - test scripts without a test plan + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = top.test bot.test +END + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +cat > top.test < bot.test <stdout || { cat stdout; Exit 1; } +cat stdout +count_test_results total=7 pass=5 xfail=1 skip=1 fail=0 xpass=0 error=0 + +: diff --git a/tests/tap-realtime.test b/tests/tap-realtime.test new file mode 100755 index 000000000..0f43e9e43 --- /dev/null +++ b/tests/tap-realtime.test @@ -0,0 +1,115 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - testsuite progress on console should happen mostly "in real time"; +# i.e., it's not acceptable for the driver to wait the end of the +# script to start displaying results from it. +# FIXME: this test uses expect(1) to ensure line buffering from make and +# children, and is pretty hacky and complex; is there a better way to +# accomplish the checks done here? + +parallel_tests=yes +. ./defs || Exit 1 + +cat >expect-check <<'END' +eval spawn $env(SHELL) -c ":" +expect eof +END +expect -f expect-check || { + echo "$me: failed to find a working expect program" >&2 + Exit 77 +} +rm -f expect-check + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TESTS = all.test +END + +cat > all.test <<'END' +#! /bin/sh +echo 1..3 + +# FIXME: creative quoting to placate maintainer-check +sleep="sleep "3 + +echo ok 1 - foo +$sleep +test -f ok-1 || { echo 'Bail out!'; exit 1; } + +echo ok 2 - bar +$sleep +test -f ok-2 || { echo 'Bail out!'; exit 1; } + +echo ok 3 - baz +$sleep +test -f ok-3 || { echo 'Bail out!'; exit 1; } + +: > all-is-well +END + +chmod a+x all.test + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +cat > expect-make <<'END' +eval spawn $env(MAKE) check +expect { + "PASS: all.test 1 - foo" { + open "ok-1" "w" + exp_continue + } + "PASS: all.test 2 - bar" { + open "ok-2" "w" + exp_continue + } + "PASS: all.test 3 - baz" { + open "ok-3" "w" + exp_continue + } + "Testsuite summary" { + exit 0 + } + timeout { + puts "expect timed out" + exit 1 + } + default { + puts "expect error" + exit 1 + } +} +END + +# Expect should simulate a tty as stdout, which should ensure a +# line-buffered output. +MAKE=$MAKE expect -f expect-make +test -f all-is-well + +: diff --git a/tests/tap-recheck-logs.test b/tests/tap-recheck-logs.test new file mode 100755 index 000000000..a3484bbc7 --- /dev/null +++ b/tests/tap-recheck-logs.test @@ -0,0 +1,132 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - RECHECK_LOGS + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = foo.test bar.test baz.test +baz.log: zardoz +END + +: > zardoz + +cat > foo.test <<'END' +1..2 +ok 1 +ok 2 +END + +cat > bar.test <<'END' +1..1 +not ok 1 +END + +cat > baz.test <<'END' +1..1 +Bail out! +END + +$ACLOCAL +$AUTOCONF +$AUTOMAKE -a + +./configure + +# Even the tests that are not re-run should contribute to the testsuite +# summary when obtained by "make check RECHECK_LOGS=". +grep_summary () +{ + grep '^# TOTAL: *4$' stdout + grep '^# PASS: *2$' stdout + grep '^# XPASS: *0$' stdout + grep '^# FAIL: *1$' stdout + grep '^# XFAIL: *0$' stdout + grep '^# SKIP: *0$' stdout + grep '^# ERROR: *1$' stdout +} + +$MAKE -e check && Exit 1 +test -f foo.log +test -f bar.log +test -f baz.log + +rm -f foo.log bar.log + +env RECHECK_LOGS= $MAKE -e check > stdout && { cat stdout; Exit 1; } +cat stdout +test -f foo.log +test -f bar.log +grep '^PASS: foo\.test 1$' stdout +grep '^PASS: foo\.test 2$' stdout +grep '^FAIL: bar\.test 1$' stdout +grep 'baz\.test' stdout && Exit 1 +grep_summary + +$sleep +touch foo.test +# We re-run only a successful test, but the tests that failed in the +# previous run should still be taken into account, and cause an overall +# failure. +env RECHECK_LOGS= $MAKE -e check > stdout && { cat stdout; Exit 1; } +cat stdout +grep '^PASS: foo\.test 1$' stdout +grep '^PASS: foo\.test 2$' stdout +grep 'ba[rz]\.test' stdout && Exit 1 +is_newest foo.log foo.test +grep_summary + +$sleep +touch zardoz +env RECHECK_LOGS= $MAKE -e check > stdout && { cat stdout; Exit 1; } +cat stdout +grep '^ERROR: baz\.test' stdout +$EGREP '(foo|bar)\.test' stdout && Exit 1 +is_newest baz.log zardoz +grep_summary + +# Now, explicitly retry with all test logs already updated, and ensure +# that the summary is still displayed. +env RECHECK_LOGS= $MAKE -e check > stdout && { cat stdout; Exit 1; } +cat stdout +$EGREP '(foo|bar|baz)\.test' stdout && Exit 1 +grep_summary + +# The following should re-run foo.test (and only foo.test), even if its +# log file is up-to-date. +: > older +env RECHECK_LOGS=foo.log $MAKE -e check > stdout && { cat stdout; Exit 1; } +cat stdout +grep '^PASS: foo\.test 1$' stdout +grep '^PASS: foo\.test 2$' stdout +grep 'ba[rz]\.test' stdout && Exit 1 +is_newest foo.log older +grep_summary + +: diff --git a/tests/tap-recheck.test b/tests/tap-recheck.test new file mode 100755 index 000000000..14b02ad15 --- /dev/null +++ b/tests/tap-recheck.test @@ -0,0 +1,217 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# Test the 'recheck' target for TAP test protocol. +# Keep in sync with 'test-driver-custom-multitest-recheck.test'. + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TESTS = a.test b.test c.test d.test +END + +cat > a.test << 'END' +#! /bin/sh +echo 1..2 +echo ok 1 +echo ok 2 +: > a.run +END + +cat > b.test << 'END' +#! /bin/sh +: > b.run +echo 1..2 +echo ok 1 +if test -f b.ok; then + echo ok 2 +else + echo 'Bail out!' +fi +END + +cat > c.test << 'END' +#! /bin/sh +echo 1..3 +if test -f c.pass; then + echo 'ok - c is ok :-)' +else + echo 'not ok - c is ko :-(' +fi +if test -f c.xfail; then + echo 'not ok 2 # TODO' +else + echo 'ok 2 # TODO' +fi +echo 'not ok 3 - blah blah # TODO need better diagnostic' +: > c.run +END + +cat > d.test << 'END' +#! /bin/sh +test -f ./d.count && . ./d.count +echo 1..${test_count-2} +echo ok 1 '# SKIP: who cares ...' +(. ./d.extra) || echo 'not ok 2 - d.extra failed' +: > d.run +END + +chmod a+x *.test + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +do_recheck () +{ + case $* in + --fail) on_bad_rc='&&';; + --pass) on_bad_rc='||';; + *) fatal_ "invalid usage of function 'do_recheck'";; + esac + rm -f *.run + eval "\$MAKE recheck >stdout $on_bad_rc { cat stdout; ls -l; Exit 1; }; :" + cat stdout; ls -l +} + +for vpath in : false; do + if $vpath; then + mkdir build + cd build + srcdir=.. + else + srcdir=. + fi + + $srcdir/configure + + : A "make recheck" in a clean tree should run no tests. + do_recheck --pass + cat test-suite.log + test ! -r a.run + test ! -r a.log + test ! -r b.run + test ! -r b.log + test ! -r c.run + test ! -r c.log + test ! -r d.run + test ! -r d.log + count_test_results total=0 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0 + + : Run the tests for the first time. + $MAKE check >stdout && { cat stdout; Exit 1; } + cat stdout + ls -l + # All the test scripts should have run. + test -f a.run + test -f b.run + test -f c.run + test -f d.run + count_test_results total=9 pass=3 fail=2 xpass=1 xfail=1 skip=1 error=1 + + : Let us make b.test pass. + echo OK > b.ok + do_recheck --fail + # a.test has been successful the first time, so no need to re-run it. + # Similar considerations apply to similar checks, below. + test ! -r a.run + test -f b.run + test -f c.run + test -f d.run + count_test_results total=7 pass=2 fail=2 xpass=1 xfail=1 skip=1 error=0 + + : Let us make the first part of c.test pass. + echo OK > c.pass + do_recheck --fail + test ! -r a.run + test ! -r b.run + test -f c.run + test -f d.run + count_test_results total=5 pass=1 fail=1 xpass=1 xfail=1 skip=1 error=0 + + : Let us make also the second part of c.test pass. + echo KO > c.xfail + do_recheck --fail + test ! -r a.run + test ! -r b.run + test -f c.run + test -f d.run + count_test_results total=5 pass=1 fail=1 xpass=0 xfail=2 skip=1 error=0 + + : Nothing changed, so only d.test should be run. + for i in 1 2; do + do_recheck --fail + test ! -r a.run + test ! -r b.run + test ! -r c.run + test -f d.run + count_test_results total=2 pass=0 fail=1 xpass=0 xfail=0 skip=1 error=0 + done + + : Let us make d.test run more testcases, and experience _more_ failures. + echo 'test_count=9' > d.count + unindent > d.extra <<'END' + echo 'ok # SKIP s' + echo 'not ok - f 1' + echo 'ok - p 1' + echo 'not ok - f 2' + echo 'ok # TODO xp' + echo 'not ok - f 3' + echo 'not ok - f 4' + echo 'ok - p 2' + echo 'ok' # Extra test. + echo 'Bail out!' +END + do_recheck --fail + test ! -r a.run + test ! -r b.run + test ! -r c.run + test -f d.run + count_test_results total=11 pass=2 fail=4 xpass=1 xfail=0 skip=2 error=2 + + : Let us finally make d.test pass. + echo 'test_count=1' > d.count + echo : > d.extra + do_recheck --pass + test ! -r a.run + test ! -r b.run + test ! -r c.run + test -f d.run + count_test_results total=1 pass=0 fail=0 xpass=0 xfail=0 skip=1 error=0 + + : All tests have been successful or skipped, nothing should be re-run. + do_recheck --pass + test ! -r a.run + test ! -r b.run + test ! -r c.run + test ! -r d.run + count_test_results total=0 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0 + + cd $srcdir + +done + +: diff --git a/tests/tap-signal.test b/tests/tap-signal.test new file mode 100755 index 000000000..b1ac2dfed --- /dev/null +++ b/tests/tap-signal.test @@ -0,0 +1,65 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - a test script terminated by a signal causes an hard error + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TESTS = +END + +for sig in 1 2 13 15; do + unindent > signal-$sig.test <> Makefile.am +done + +chmod a+x *.test + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +for append in '' 'TEST_LOG_DRIVER_FLAGS = --ignore-exit'; do + echo "$append" >> Makefile + $MAKE check >stdout && { cat stdout; Exit 1; } + cat stdout + count_test_results total=8 pass=4 fail=0 xpass=0 xfail=0 skip=0 error=4 + for sig in 1 2 13 15; do + grep "^ERROR: signal-$sig\\.test - terminated by signal.*" stdout + done +done + +: diff --git a/tests/tap-skip-whole-whitespace.test b/tests/tap-skip-whole-whitespace.test new file mode 100755 index 000000000..2a19d4c9e --- /dev/null +++ b/tests/tap-skip-whole-whitespace.test @@ -0,0 +1,68 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - normalization of whitespace in console testsuite progress associated +# with a SKIP directive in the TAP plan + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +sp=' ' +tab=' ' + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = foo.test bar.test baz.test +END + +cat > foo.test < bar.test < baz.test < stdout || { cat stdout; Exit 1; } +cat stdout + +count_test_results total=3 pass=0 fail=0 error=0 xpass=0 xfail=0 skip=3 + +grep "SKIP: foo\\.test - Strip leading & trailing$" stdout +grep "SKIP: bar\\.test - Preserve ${tab} middle${tab}${tab}space" stdout +grep "SKIP: baz\\.test - Strip${tab}external preserve ${tab}middle" stdout + +: diff --git a/tests/tap-skip-whole.test b/tests/tap-skip-whole.test new file mode 100755 index 000000000..bd75e904a --- /dev/null +++ b/tests/tap-skip-whole.test @@ -0,0 +1,100 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# Basic TAP test protocol support: +# - special plan format to skip all the tests in a script + +parallel_tests=yes +. ./defs || Exit 1 + +weirdchars=\''"$!&()[]<>#;^?*/@%=,.:' + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = foo.test bar.test baz.test wget.test curl.test mu.test +END + +cat > foo.test < bar.test < baz.test < wget.test < curl.test < mu.test <stdout || { cat stdout; Exit 1; } +cat stdout + +count_test_results total=6 pass=0 fail=0 xpass=0 xfail=0 skip=6 error=0 + +grep '^SKIP: foo\.test$' stdout +grep '^SKIP: bar\.test$' stdout +grep '^SKIP: baz\.test' stdout # Deliberately laxer, see above for why. +grep '^SKIP: wget\.test .* wget(1) not installed$' stdout +grep '^SKIP: curl\.test .* Can'\''t connect to gnu\.org!$' stdout +grep '^SKIP: mu\.test' stdout | $FGREP "$weirdchars" stdout +test `grep -c ': .*\.test' stdout` -eq 6 + +: diff --git a/tests/tap-summary.test b/tests/tap-summary.test new file mode 100755 index 000000000..649485e72 --- /dev/null +++ b/tests/tap-summary.test @@ -0,0 +1,363 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - testsuite summary +# - colorized testsuite summary +# TODO: splitting this into two distinct test scripts might be worthwhile + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat > configure.in < Makefile.stub << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = all.test +END + +$ACLOCAL +$AUTOCONF + +br='============================================================================' + +# The following shell variables are influential for this function: +# - expect_failure +# - use_colors +do_check () +{ + case $#,$1 in + 1,--pass) expect_failure=no;; + 1,--fail) expect_failure=yes;; + *) fatal_ "invalid usage of 'do_check'";; + esac + shift + cat > summary.exp + cat all.test + st=0 + $MAKE check > stdout || st=$? + cat stdout + if test $expect_failure = yes; then + test $st -gt 0 || Exit 1 + else + test $st -eq 0 || Exit 1 + fi + $PERL -w "$testsrcdir"/extract-testsuite-summary stdout > summary.got \ + || fatal_ "cannot extract testsuite summary" + cat summary.exp + cat summary.got + if test $use_colors = yes; then + # Use cmp, not diff, because the files might contain binary data. + compare=cmp + else + compare=diff + fi + $compare summary.exp summary.got || Exit 1 +} + +# FIXME: also enable testing of colors! +#for use_colors in "no" "yes"; do +for use_colors in "no"; do + + cp Makefile.stub Makefile.am + + if test $use_colors = yes; then + red='' + grn='' + lgn='' + blu='' + mgn='' + brg='' + std='' + echo AUTOMAKE_OPTIONS = color-tests >> Makefile.am + else + red= grn= lgn= blu= mgn= brg= std= + fi + + success_header="\ +${grn}${br}${std} +${grn}Testsuite summary for GNU AutoTAP 5.12${std} +${grn}${br}${std}" + + success_footer=${grn}${br}${std} + + failure_header="\ +${red}${br}${std} +${red}Testsuite summary for GNU AutoTAP 5.12${std} +${red}${br}${std}" + + failure_footer="\ +${red}${br}${std} +${red}See ./test-suite.log${std} +${red}Please report to bug-automake@gnu.org${std} +${red}${br}${std}" + + $AUTOMAKE --add-missing + + ./configure + + # 1 pass. + { echo 1..1 && echo ok; } > all.test + do_check --pass < all.test + do_check --pass < all.test + do_check --pass < all.test + do_check --fail < all.test + do_check --fail < all.test + do_check --fail < all.test < all.test < all.test < all.test < all.test < tap + + # Lots of non-failures (300 per kind). + (cat tap && cat tap && cat tap) > all.test + test `wc -l > all.test # Test plan. + do_check --pass < all.test + test `wc -l > all.test # Test plan. + do_check --fail < all.test + test `wc -l > all.test # Test plan. + do_check --fail <. + +# TAP support: +# - TODO and SKIP directives on the same line: the first one wins + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = all.test +END + +cat > all.test <stdout || { cat stdout; Exit 1; } +cat stdout + +count_test_results total=2 pass=0 fail=0 xpass=0 xfail=1 skip=1 error=0 + +grep '^SKIP: all\.test 1 ' stdout +grep '^XFAIL: all\.test 2 ' stdout + +: diff --git a/tests/tap-todo-skip-whitespace.test b/tests/tap-todo-skip-whitespace.test new file mode 100755 index 000000000..68fbbc1a1 --- /dev/null +++ b/tests/tap-todo-skip-whitespace.test @@ -0,0 +1,118 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - normalization of whitespace in console testsuite progress associated +# with TODO and SKIP directives + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = all.test +END + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +cat > stub.tap < stub.exp <stdout || : # Don't care about the exit status in this test. + cat stdout + count_test_results total=15 pass=0 fail=0 error=0 \ + xpass=$xpass xfail=$xfail skip=$skip + # Don't be too strict w.r.t. possible normalization of "TODO: foo" into + # "TODO : foo" (as is done by, e.g., the 'TAP::Parser' perl module). + LC_ALL=C grep '^[A-Z][A-Z]*:' stdout \ + | sed -e 's/# \(TODO\|SKIP\) *:/# \1:/' > got + cat exp + cat got + diff exp got +} + +# For "TODO" directives leading to XPASS results. +(echo $plan && sed -e 's/^/ok /' stub.tap) > all.test +sed -e 's/^/XPASS: all.test /' stub.exp > exp +my_make_check xpass + +# For "TODO" directives leading to XFAIL results. +(echo $plan && sed -e 's/^/not ok /' stub.tap) > all.test +sed -e 's/^/XFAIL: all.test /' stub.exp > exp +my_make_check xfail + +# For "SKIP" directives. +(echo $plan && sed -e 's/^/ok /' -e 's/TODO/SKIP/' stub.tap) > all.test +sed -e 's/TODO/SKIP/' -e 's/^/SKIP: all.test /' stub.exp > exp +my_make_check skip + +: diff --git a/tests/tap-todo-skip.test b/tests/tap-todo-skip.test new file mode 100755 index 000000000..c37b63158 --- /dev/null +++ b/tests/tap-todo-skip.test @@ -0,0 +1,109 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - TODO and SKIP directives are case-insensitive +# - TODO and SKIP directives can be followed optionally by a colon ":" +# and by an optional explanation. +# - our driver isn't fooled into recognizing TODO and SKIP directives +# spuriously +# - the reasons for TODO and SKIP, if present, are nicely printed in +# the testsuite progress output + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = all.test +END + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +# ----------------------------------------------------- # +# Check all possible combinations of: # +# - uppercase/lowercase # +# - with/without colon character ":" # +# - with/without explanatory message # +# in TODO and SKIP directives. # +# ----------------------------------------------------- # + +# There are 2 * 2^6 + 2 * 2^6 = 256 tests. +echo 1..256 > all.test + +# These nested loops are clearer without indentation. + +for c1 in t T; do +for c2 in o O; do +for c3 in d D; do +for c4 in o O; do +for ex in '' ':' ' foo' ': foo'; do + echo "not ok # $c1$c2$c3$c4$ex" + echo "not ok# $c1$c2$c3$c4$ex" +done; done; done; done; done >> all.test + +for c1 in s S; do +for c2 in k K; do +for c3 in i I; do +for c4 in p P; do +for ex in '' ':' ' foo' ': foo'; do + echo "ok # $c1$c2$c3$c4$ex" + echo "ok# $c1$c2$c3$c4$ex" +done; done; done; done; done >> all.test + +cat all.test # For debugging. + +$MAKE check >stdout || { cat stdout; Exit 1; } +cat stdout + +count_test_results total=256 pass=0 fail=0 xpass=0 xfail=128 skip=128 error=0 + +# -------------------------------------------------------- # +# TODO ans SKIP directives aren't recognized spuriously. # +# -------------------------------------------------------- # + +cat > all.test <<'END' +1..9 +ok TODO +ok - TODO +ok 3 TODO +ok 4 - TODO +ok SKIP +ok - SKIP +ok 7 SKIP +ok 8 - SKIP +ok 9 +END + +$MAKE check >stdout || { cat stdout; Exit 1; } +cat stdout + +count_test_results total=9 pass=9 fail=0 xpass=0 xfail=0 skip=0 error=0 + +: diff --git a/tests/tap-unplanned.test b/tests/tap-unplanned.test new file mode 100755 index 000000000..696f73bef --- /dev/null +++ b/tests/tap-unplanned.test @@ -0,0 +1,177 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - unplanned tests are properly reported as errors + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = foo.test +END + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +cat > foo.test <stdout && { cat stdout; Exit 1; } +cat stdout +count_test_results total=3 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=2 +grep '^ERROR: foo\.test - too many tests run (expected 1, got 2)$' stdout +grep '^ERROR: foo\.test 2 # UNPLANNED$' stdout + +cat > foo.test <stdout && { cat stdout; Exit 1; } +cat stdout +count_test_results total=4 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=2 +grep '^ERROR: foo\.test - too many tests run (expected 2, got 3)$' stdout +grep '^ERROR: foo\.test 3 # UNPLANNED$' stdout + +# Interaction with XFAIL_TESTS. +cat > foo.test <stdout && { cat stdout; Exit 1; } +cat stdout +count_test_results total=6 pass=0 fail=0 xpass=0 xfail=1 skip=1 error=4 +grep '^ERROR: foo\.test - too many tests run (expected 2, got 5)$' stdout +grep '^ERROR: foo\.test 3 # UNPLANNED$' stdout +grep '^ERROR: foo\.test 4 # UNPLANNED$' stdout +grep '^ERROR: foo\.test 5 # UNPLANNED$' stdout + +cat > foo.test < t <stdout && { cat stdout; Exit 1; } +cat stdout +count_test_results total=22 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=21 + +echo 'PASS: foo.test 1' > exp +sed -e '/^$/d' -e 's/.*/ERROR: foo.test & # UNPLANNED/' t >> exp +echo 'ERROR: foo.test - too many tests run (expected 1, got 21)' >> exp + +$FGREP ': foo.test' stdout > got + +cat exp +cat got +diff exp got + +# Note that, if the TAP input has a trailing plan, it is not possible +# to flag unplanned tests as such, since we do not know they're unplanned +# until the plan is reached; still, we should give at least an error +# message about the unmatched number of tests once we've got the plan. + +for x in 'ok' 'ok 3' 'not ok' 'not ok # TODO' 'ok # TODO' 'ok # SKIP'; do + unindent > foo.test <stdout && { cat stdout; Exit 1; } + cat stdout + test `$FGREP -c ': foo.test' stdout` -eq 4 + $EGREP '^PASS: foo\.test 1($| )' stdout + $EGREP '^SKIP: foo\.test 2($| )' stdout + $EGREP ': foo\.test 3($| )' stdout + grep '^ERROR: foo\.test - too many tests run (expected 2, got 3)$' stdout +done + +: diff --git a/tests/tap-whitespace-normalization.test b/tests/tap-whitespace-normalization.test new file mode 100755 index 000000000..16e68015f --- /dev/null +++ b/tests/tap-whitespace-normalization.test @@ -0,0 +1,181 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: whitespace normalization (or lack thereof) in the testsuite +# progress output on console. We keep all these checks here in a single +# script so that a potential cosmetic change in the output format won't +# force us to tweak dozens of other tests (hopefully). +# See also related test 'tap-todo-skip-whitespace.test'. + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = +END + +: > exp + +sp=' ' +tab=' ' + +spaces_a=${sp}${tab}${tab}${sp}${sp}${tab} +spaces_b=${tab}${tab}${sp}${tab}${sp}${sp}${sp} + +#----------------------------------------------------------------------- + +echo TESTS += numbers.test >> Makefile.am + +cat > numbers.test <> exp <> Makefile.am + +cat > description.test <> exp <> Makefile.am + +cat > bailout.test <> exp <> Makefile.am + +cat > bailout2.test <> exp <> Makefile.am +echo TESTS += cmnt.test >> Makefile.am + +cat > cmnt.test <> exp <stdout || : +cat stdout + +LC_ALL=C sort exp > t +mv -f t exp + +# We need the sort below to account for parallel make usage. +LC_ALL=C grep '[a-z0-9][a-z0-9]*\.test' stdout | LC_ALL=C sort > got + +cat exp +cat got +diff exp got + +: diff --git a/tests/tap-with-and-without-number.test b/tests/tap-with-and-without-number.test new file mode 100755 index 000000000..a1f7a4bf4 --- /dev/null +++ b/tests/tap-with-and-without-number.test @@ -0,0 +1,68 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - numbered and unnumbered test results can coexist +# - tests without explicit number get automatically numbered in the +# testsuite progress output on console + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = all.test +END + +cat > all.test <<'END' +1..7 +ok 1 foo +ok 2 - foo2 +ok - bar +not ok baz # TODO +not ok 5 - asd # TODO +ok 6 # SKIP +ok zardoz +END + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +$MAKE check >stdout || { cat stdout; Exit 1; } +cat stdout +count_test_results total=7 pass=4 fail=0 xpass=0 xfail=2 skip=1 error=0 + +grep '^PASS: all\.test 1 foo$' stdout +grep '^PASS: all\.test 2 - foo2$' stdout +grep '^PASS: all\.test 3 - bar$' stdout +grep '^XFAIL: all\.test 4 baz # TODO$' stdout +grep '^XFAIL: all\.test 5 - asd # TODO$' stdout +grep '^SKIP: all\.test 6 # SKIP$' stdout +grep '^PASS: all\.test 7 zardoz$' stdout + +: diff --git a/tests/tap-xfail-tests.test b/tests/tap-xfail-tests.test new file mode 100755 index 000000000..f159e8ae7 --- /dev/null +++ b/tests/tap-xfail-tests.test @@ -0,0 +1,85 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP and $(XFAIL_TESTS): test results without directives are turned from +# PASS to XPASS and from FAIL to XFAIL; other results are unchanged. + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +TESTS = the.test +XFAIL_TESTS = $(TESTS) +END + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +cat > the.test <stdout && { cat stdout; Exit 1; } +cat stdout + +count_test_results total=7 pass=0 fail=0 xpass=2 xfail=3 skip=1 error=1 + +grep '^XPASS: the\.test 1$' stdout +grep '^XFAIL: the\.test 2$' stdout +grep '^XPASS: the\.test 3 # TODO' stdout +grep '^XFAIL: the\.test 4 # TODO' stdout +grep '^SKIP: the\.test 5 # SKIP' stdout +grep '^XFAIL: the\.test 6 # SKIP' stdout +grep '^ERROR: the\.test - Bail out!' stdout + +# Check that the exit status of the testsuite is indeed 0 when we +# would expect success. + +cat > the.test <stdout || { cat stdout; Exit 1; } +cat stdout + +count_test_results total=3 pass=0 fail=0 xpass=0 xfail=2 skip=1 error=0 + +: diff --git a/tests/test-driver-custom-multitest-recheck.test b/tests/test-driver-custom-multitest-recheck.test index 137ff96b1..8bbcc7c2a 100755 --- a/tests/test-driver-custom-multitest-recheck.test +++ b/tests/test-driver-custom-multitest-recheck.test @@ -20,6 +20,7 @@ # support, but also serves as a "usability test" for our APIs. # See also related tests `test-driver-custom-multitest-recheck2.test' # and `parallel-tests-recheck-override.test'. +# Keep in sync with 'tap-recheck.test'. parallel_tests=yes . ./defs || Exit 1 -- 2.47.2