tclwebtest output

Output

The output of every single test (every .test file) will be surrounded by a start and end marker, containing the file name and the date and duration of the test run, like this:

tils@tp:~/tclwebtest$ ./tclwebtest selftest/asserts.test 

----- START: selftest/asserts.test at [31/Jul/2001:15:09:30] -----
some messages ...
----- SUCCESS: selftest/asserts.test (took 0s)               -----

The messages in between can be generated with the puts command.

Some commands emit a log message automatically - for example for each http request that is being issued the url is logged. This should produce some meaningful overview of what has happened without producing tons of output.

tclwebtest will exit with return code 1 when at least one of the tests failed that it was called with, 0 if all succeeded. If some tests fail it will report which ones as its last output line. The details can then be found inside the markers.

Redirecting output

tclwebtest output goes to stdout. Since the markers contain date information it should be safe to call tclwebtest from a cron job or something similar, and check for the return code only.

Debug messages

When writing tests one might want to add some debugging output command into the test to check if the test does more or less what is expected or to find out why a test fails when it shouldn't. This can be done with the

debug
    message

statement of tclwebtest, which is a simple wrapper around puts. The message will go to stdout and will only be printed when tclwebtest is run with the -d option (soon).

Stack trace

When an assertion fails an error will be thrown and catched internally, and the resulting errorInfo will be parsed for the containing line number. That line number will then be used to display the line on which the test failed and a few of the preceding lines. For example:

tils@tp:~/tclwebtest$ ./tclwebtest playtests/some.test 

----- START: playtests/some.test at [31/Jul/2001:21:03:23] -----
--- do_request for http://localhost/
http status: >>200<<

"[llength [cookies all]] > 0" failed. 
in "playtests/some.test" line 4:
do_request "http://localhost/"

# do we get some cookies at all?
assert { [llength [cookies all]] > 0 }

-----  FAILED: playtests/some.test (took 0s)               -----

1 of 1 tests FAILED: playtests/some.test

The last line before the end marker is the one that threw the assertion_failed. If the writer of the test wants to be verbose then she can place comments like the one above before commands that might throw an assertion, as a message to the person that might investigate future failures of the test.

Note: I have no idea if the method for parsing the stacktrace is reliable - it has not been extensively tested. If the parsing fails a normal stacktrace will be put out. In that case the whole output become much less meaningful, because it may be hard to find the offending line in the test (especially in large test files).

Simple Variables

Whenever an assertion fails with a condition that contains tcl variables, tclwebtest will print their values to provide a meaningful error message. For example:

tils@tp:~/tclwebtest$ ./tclwebtest playtests/fail.test 

----- START: playtests/fail.test at [31/Jul/2001:21:05:58] -----

"$a == $b" failed. $a: apple; $b: orange
in "playtests/fail.test" line 4:
# a very silly test
set a "apple"
set b "orange"
assert { $a == $b }

-----  FAILED: playtests/fail.test (took 0s)               -----

1 of 1 tests FAILED: playtests/fail.test

This will only be done on simple variables - commands can not be evaluated. The value will be truncated to the first few characters to prevent large values from producing confusing output.

Tilmann Singer

Last modified: Sun Mar 2 10:50:36 IST 2003