Reference Material

Test Selection and Execution

Tests are defined in ATS input using two commands, test and its little brother, testif. However, not every test that gets defined is necessarily going to be executed. The user can define logical conditions (filters) that a test must satisfy to be chosen for execution, and the hardware available may cause others to be skipped.

In order to make it easier to structure suites of tests, there is an elaborate set of facilities involving filters, command-line options, and arguments to test statements, as well as facilities for grouping and ordering your test executions.

ATS Execution and Command-line Options

In specifying the names of input files, you can give the filename or omit the filename extension. ATS will attempt to find the file using its name, then with a .ats extension, and then with a .py extension.

Unix or Mac

To start ATS on a Unix system or Mac, execute this line in a terminal window:

ats [options] [input files]

Note that the --exec option is frequently used to define a default executable, but any given test can specify any executable as the program to be tested.

Before executing ATS, it may be desirable to have defined the environment variables MACHINE_TYPE and / or SYS_TYPE; and there may be others for testing particular executables. Please consult with the owner of your local ATS installation, and the owners of any custom ATS drivers you may be using.

Windows

Execution on windows can be done in the same way from a command window, but can be made more convenient by defining a .bat file, such as:

C:\python27\python c:\python27\ats $*

These instructions need improvement as the first Windows users determine the right way to do this.

Command-line Options

What follows are the the most important command-line options available in any ATS installation.

Note

The exact set of command-line options depends on the machine you are using and / or upon any custom driver you are using for testing a particular program. To see the complete list for a given ATS installation, enter ats --help.

This will also show you abbreviations for some of the options.

--allInteractive

Run every test in interactive mode.

--cutoff cutofftime

This invokes a special mode in which no test is allowed to run longer than cutofftime, regardless of its actual timelimit option. Jobs that reach this threshold are treated as failures in the sense that any jobs depending upon them are not run; but they are given status HALTED rather than TIMEDOUT. The forms for giving the time are the same as for --timelimit.

--debug

Debug mode; prints more information in the log and on the shell window.

--exec EXEC

Give the path to the code to be tested. The path is tilde- and dollar-expanded.

This option sets the environment variable ATSROOT, if not already set, to the directory in which the executable resides. Most of the time this option is used, and the executable so named is referred to in this documentation as the specified executable.

However, tests with different executables can also be specified, by using the executable=’/path/to/my/code’ as one of the test options. The purpose of ATSROOT is to allow you to specify related tools for your code that are located in the same directory as the executable. In specifying a test, you can use this variable in the script or executable using either $ATSROOT or %(ATSROOT).

Note that you don’t have to have one main code to be tested. You can specify a different executable for each test, or group of tests.

--filter FILTER

Add a filter; may be repeated. Be sure to use quotes if the filter contains spaces and remember that the shell will remove one level of quotes.

--glue FILTER

Has the effect of executing glue(FILTER) before execution of the tests. May be repeated. Be sure to use quotes if the filter contains spaces and remember that the shell will remove one level of quotes. The glue function is used to set persistent test option defaults.

--help

Show the list of options and exit. There may be more options than are shown in this document, such as batch or node control options.

--info

Print information about ATS, such as version, path to the executable, and some parameter values.

--keep

Keep the output files from the tests that succeed. Normally the output from tests that fail, or which must be checked, is kept.

--logs LOGDIR

Sets the name of the log directory. The default log directory is arch.time.logs, where arch will be an architecture-dependent name, and time will be digits of the form yymmddhhmmss. All logs and the continuation file are placed in this directory. The log itself is named ats.log.

--level LEVEL

Set the maximum level of test to run. Level is simply a built-in easy-to-use filter.

--skip

Skip actual execution of the tests, but show filtering results and missing test files, and show additional details about the input.

--nobatch

Do not run any “batch” tests..

--npMax value

Value is an integer, the maximum number of tests to run at once (on a node, if multinode machine). Some machines allow you to set this higher than the actual number of nodes, at your own risk.

--okInvalid

Run tests even if there is an invalid test. Examples are tests specifying missing scripts or executables.

--oneFailure

Stop if a test fails.

--serial

Run only one job at a time.

--timelimit TIMELIMIT

Set the default timelimit test option. TIMELIMIT may be given as an integer number of seconds or a string specification such as ‘2m’, or ‘3h30m20s’. A similar notation can be used for filtering by time limit, such as -f ‘timelimit < “30m”’.

--verbose

Verbose mode. Both starts and finishes of tests are noted on the terminal, plus other reports. Test failures are reported regardless.

--version

Show program’s version number and exit.

Basic Operations

The goal of ATS is to execute a series of test problems. It does this by reading input files written in the Python language, with some predefined ATS functions added. In particular, ATS supplies a function named test. Each execution of the test statement defines a particular program to execute, including its command line and a variety of options used by ATS to know how to run it or to decide not to run it.

After running the tests, the ats prints a summary of which tests have passed (that is, returned with a normal exit status) and which have failed.

The second basic statement is the source statement, which causes a file to be read containing additional commands. An introspection procedure, described below, is also available to allow scripts meant as problem input to contain definitions of how they are to be run when run by ATS.

Retrying Failed Tests

If any tests fail or are not completed, a “continuation” file is written and a message issued in the summary section giving the name of the file. The continuation file is named continue.ats and it is inside the log directory.

You can rerun the exact same ATS command, adding the path to the continuation file as an extra command-line argument.

Note

You must run the exact same command with this added argument at the end of the command line.

Doing this will redo those families of tests that had a failed member. This process may be repeated until all tests pass. In your log, tests that had passed before well be marked “Previously passed” and batch jobs will be “skipped”. The continuation file is pretty self-explanatory and you can edit it with thought.

Note that if a descendent of a test failed, the test will be rerun because the error might have been in files produced by the parent test, even though it appeared to pass.

The intention of this facility is to let you fix your code without having to rerun all your tests. For correctness, you should rerun everything once you believe you have corrected all errors.

Results Facility

Each run creates an atsr.py file in the log directory. This file, if run under Python, creates one variable named “state”, which is an object that is a dictionary whose values can be read and written using either dictionary or attribute notation. This type is called an AttributeDict.

The object state has attributes corresponding to the major features of the manager object, including a machine and testlist, which is a list of AttributeDicts, each encapsulating the major properties of each test.

Two methods in the manager object control this facility, which may be used by custom drivers.

onSave(saver)

Registers a function saver(results, manager), which will be called when the data for the state is collected. It may modify the AttributeDict results in any way it likes, usually by adding to it. Calling results.clear() would be a way of minimizing the use of resources devoted to this file.

onSave is available in the test environment also, for use in input files. Note that the call does not cause the save of the file at the time it is executed.

Three other manager methods can be called from custom drivers.

getResults()

Returns the AttributeDict containing the state. The manager’s machine and, if set, batchmachine, are given a change to contribute fields to the end result, and finally any onSave-registered routines are called in the order they were registered.

saveResults(filename='atsr.py')

Save the state to a file using given file name; if not absolute, put it in the log directory.

printResults(file=sys.stdout)

Do the actual job of writing the state file. Here file should be an open file handle. You would only use this function if you wanted to add something to the file other than the state variable.

Normally saveResults creates the file and asks printResults to call getResults and print the returned state into the file, preceded by a header that imports the symbols in the ats module so that the code will execute correctly.

Interactive inspection of the resulting file is most easily accomplished with an interactive Python session, such as:

cd <logdirectory>
python -i atsr.py
   print "Number of tests = ", len(state.testlist)
   print "Machine name", state.machine.name
   print "Number timed out", \
         len([t for t in state.testlist if t.status == TIMEDOUT])

Note that ATS statuses will compare equal if they compare to another status or the name or the abbreviation. So in the last line above, TIMEDOUT, “TIME”, or “TIMEDOUT” would all work.

To compare different files you can rename state as you read it:

d= {}
execfile("atsr.py", d)
state1 = d['state']

You can change the name of the file to be used by setting manager.saveResultsName in your input file. If not an absolute path, the file will be created in the logs directory.

Controlling Input

File Sourcing

source(*paths, **vocabulary)

Process one or more paths as if each was the name of an input file given on the command line. (This function is the same as manager.source)

The current stuck options are saved upon entry, cleared before beginning processing, and then restored on completion. See stick below for further details.

Path names are expanded both for tilde and environment-variable names using the dollar sign.

The vocabulary items can be any number of keyword = value pairs.

Vocabulary words are added to the environment in which input files are compiled by Python. The scope of this environment is just within the input of the paths given to this source command. To add a vocabulary value to all subsequent source commands, use the define command, described next.

The vocabulary word introspection can be used to change the commenting convention used for ATS’ introspection facility. Details are given below.

define(keyword=value, ...)

adds one or more keywords to the vocabulary used by the source command to parse input. This is the same function as manager.define.

undefine(keyword, ...)

removes one or more keywords from the vocabulary used by the source command to parse input. This is the same function as manager.undefine.

showDefine(*keywords, **options)

logs the current definition of one or more keywords in the vocabulary used by the source command. If no argument is given, all the definitions are shown. This function is used to help debug your vocabulary setup. The options may include echo and logging, and are passed on to the call to log. The defaults are both True. This is the same function as manager.showDefine.

A file may be ‘sourced’ because it was given on the command line or because a source function was executed with it as an argument. (Note: In what follows it is is assumed that a line that starts #ATS: is a comment to your application; however, it is possible to change the commenting convention to suit your input convention, using the second argument to source.

Examining and prioritizing tests

After collection of the tests the user may wish to examine or alter the tests before they are executed. This is done by registering one or more routines to be called (in the order in which they were registered) by using onCollected. See also onPrioritized, below.

onCollected(routine)

The routine registered is called when the input is complete. It is given the manager object as its single argument. The routine thus has access to the manager.testlist.

The routine may make use of the routine that ATS itself is about to use to divide the tests into interactive and batch tests:

interactiveTests, batchTests = manager.sortTests()

You can effect what happens next by changing statuses (such as setting the status to BATCH or FILTERED or CREATED (i.e., interactive)) or change totalPriority (see below).

You also have a chance at this point to use each test’s directory attribute to prepare the file system, or to build data structures for later use in a postprocessor.

Use this facility with caution. Do not attempt to change tests that would not have executed at all into ones that will. If you change a label it must be unique when you are done. Do not alter serial or group numbers.

After the onCollected actions, the scheduler prioritizes the interactive tests. The totalPriority attribute of each test is set to the sum of the test’s own value plus the sum of the priorities of each test that must wait for this one to complete. (Such conditions are created by dependencies or wait or group commands.)

The user may wish to examine or alter the priorities of the tests tests before they are executed. This is done by registering one or more routines to be called (in the order in which they were registered) by using onPrioritized.

onPrioritized(routine)

The routine should take a single argument, interactiveTests. The intent is for the user to examine or alter the totalPriority attribute of a test. Altering priority attributes will not work. Altering anything else about the test is probably ill-advised.

In summary, there are two ways to change the totalPriority attribute: in an onCollected routine, which will contribute the new value to its predecessors, or in an onPrioritized routine, where you are setting the final absolute value.

Using Introspection

When a file is sourced, ATS looks to see if the file contains any lines that begin with the five characters #ATS:. If so, the set of such lines with the leading #ATS: removed will be executed as Python code. The remainder of the file will be ignored. This procedure is called introspection.

Note that Python’s indentation rules apply, so there should not be any spaces after the #ATS: except on lines that should be indented.

For example, continuation of lines is allowed in the normal Python manner:

#ATS:test('myfile.py',
#ATS:     'my command line args',
#ATS:     np = 4)

Picture the first five characters as defining the left edge of the lines to be executed.

During this procedure, the symbol SELF will be defined to be the name of the file being sourced. Thus a line such as:

#ATS:test(SELF, 'command line options', np=4, w=2)

will cause the file to be tested with the given command line, using the options np = 4 and w = 2 as context for filtering.

A file may contain many such lines, in order to exercise the same test with a variety of parameters. Also note that not all the #ATS: lines need to be ATS commands; they can be any Python code. They can also include log commands, source other files, etc.

Changing the introspection convention

If a value for the vocabulary word “introspection” is given, it should be a python function which, when given a line, returns None or the value of the line as introspection. The default is a function that returns None unless the line begins with #ATS:, in which case it returns the line less that prefix.

By prescribing your own value for introspection, you can allow the introspection process to work on source files with a different commenting convention than “#”.

In particular, to change the default function used for introspection, just use define after you declare it. For example:

def asteriskinterpolation(line):
    "Any line that starts with *ATS: is magic"
    if line.startswith("*ATS:"):
        return line[5:]
    else:
        return None
define(interpolation=asteriskinterpolation)

Grouping

If you have a test that creates some files for postprocessing, you can group that test with the related ones.

You begin with:

group(independent=False, report=False, **kw)

and after defining some tests, finish with:

endgroup()

A group is also ended by another group statement, or the end of the current input file. The arguments to the group call become default options for each test defined inside the group. They can be overridden by options in the test and testif statements within the group.

Only the first test result will be included in the final reports unless some member of the group fails, or you change the report argument to True. The output files of the entire group will be kept if anything fails; otherwise the usual keep options will prevail.

The independent test option determines if a test will block any other test (other than ones in its group) that uses the same directory. By default, then, a group will lock-out any non-independent test or group from running in the directory or directories its tests use. This is not different than the default behavior of ATS, but is a convenience for making sure that the members of the group will not be interleaved with other, non-independent tests that use the same directories, if you have glued or tacked or stuck independent to be be True.

These two arguments are used as test options for all tests in the group, but for any particular test can be overridden by an explicit option in the test statement itself.

Note that grouping does not make each test depend on the preceding tests in the group. Two members of the group may execute together. It also does not make the failure of one test skip another. To achieve dependency, use the ‘testif’ facility.

Wait

It is certainly possible to make two tests that appear to be independent but which cannot in fact run simultaneously. ATS prevents many cases of this due to its reluctance to run two tests in the same directory at the same time. If that fails to solve the problem, and the group or the testif statements are not sufficient, you can try the wait statement:

wait()

All the tests defined so far in this source file will be finished before proceeding to any tests defined later in this source file. Tests defined in other files that are sourced after the ‘wait’ must also wait for all the tests before the wait in this source file.

wait() may be a useful way to express massive dependencies without using excessive testif calls. However, if used excessively, wait may cripple ATS’s ability to run tests simultaneously.

You can debug your wait structure with this command:

ats yoursource --skip

This will show a list at the end of the log file, under “ATS RESULTS”, showing the serial numbers being waited for by each test.

When all tests are completed, ATS issues a final report and runs any postprocessors that have been registered using the onExit facility described later.

Example

Suppose we have this test file “waitforit.ats”:

glue(executable = "/bin/ls")
test(label='first')
test(label='second')
wait()
test(label='third')

Then the third test will not execute until the first two are done – but this says nothing about the order in which the first two will execute.

Suppose now we add a source of another file, so we have:

glue(executable = "/bin/ls")
test(label='first')         #1
test(label='second')        #2
wait()
source('waitfor1.ats')
test(label='third')         #6

with the file being sourced containing:

test(label='waitfor1 first')   #3
test(label='waitfor1 second')  #4
wait()
test(label='waitfor1 third')   #5

We have thus defined six tests in all. The output of the debugging process is:

Interactive tests:
#1 INIT ls(first) ready
   []
#2 INIT ls(second) ready
   []
#3 INIT ls(waitfor1 first) ready
   [1, 2]
#4 INIT ls(waitfor1 second) ready
   [1, 2]
#5 INIT ls(waitfor1 third) ready
   [1, 2, 3, 4]
#6 INIT ls(third) ready
   [1, 2]

The parts in square brackets are lists of the tests this one must wait for. (The list will include any tests of which this one is a dependent.) So we see for example that #6, the last test in the main file, waits for the first two tests, because a wait() occurs after #2, but it is not affected by the wait statement in the sourced file. In that file the first two tests are waiting for the first two, and the third waits for the first four.

Executing Tests

ATS attempts to execute as many tests as it can at the same time in order to keep the computational resources it has been given busy, subject to respecting the test options priority and independent, and the group and wait statements. The following sections describe this process.

Scheduling

After the ATS has read all the input and knows what tests are to be run, it examines the collection and combines the information generated by the group, and wait commands with the test dependencies to figure out which tests must execute before others. It can then combine the priorities of tests to determine a preferred order of execution – which however will be subject to processor availability.

This work is done by a scheduler object. A standard scheduler is provided, and is an attribute on the machine object. A user could potentially modify it by inheritance from its defining class, schedulers.StandardScheduler.

Each test has a priority. By default the scheduling priority (totalPriority) is the number of processors required by the test plus the priorities of any tests which cannot execute until this one is finished. In this way those tests with a lot of dependents are started early.

A test may specifiy its priority as an option “priority=n” where n is a nonzero integer. A test whose priority is zero or less will not be run. Thus, a long-running 1-processor job without dependents might profit from being given a priority, say 3, so that it starts earlier. Note that an np = 0 job requires 1 processor.

As tests are selected to be started, the highest-priority job that will fit on an available machine is chosen. You can examine the tests in postprocessing if you want to understand what influenced the scheduling:

• Test option priority,

• Test attribute totalPriority,

• Test attribute group,

• Test option independent (described below)

• Test attribute runOrder, an integer indicating the order of test launch.

Note

Important: by default two tests will not be run in the same directory at the same time.

This is a modestly conservative scheme to avoid common resource conflicts when testing one file with different parameters.

If you know a test does not have such a problem, you can give it the option independent = True. Note that the group command makes the default value of independent False for all members of the group, overriding anything except an actual option in the test statement. Thus if you do not want this behavior for the group you must use independent = True as an argument in your group command.

The standard scheduler sorts the groups by the highest priority test in the group. In effect, every member of a group behaves as if it has the priority of the highest-priority test in the group. This ensures a large prejudice towards running members of a group once it has started, until they are all complete.

Progress Reports

When a test starts this fact is shown on the terminal output. You can use the command option --verbose to cause test completions and other additional events to be reported as well. All the information is always in the log. Additional output is generated by the --debug option.

Every minute ATS issues a report on its progress to the terminal only.

Output Files

The standard output and standard error of a test are written into files in the directory where the logs are written. These files are (usually) removed when the test concludes successfully; for a group, this occurs when all members of the group have succeeded.

The name and label of the test script or executable, along with the test’s serial number, are used to create the file names.

The –keep option prevents the removal of these output files even when the tests are successful. They are also kept if the test has the option keep=True or check=True.

See also

Postprocessors set using the onExit facility can access the magic output of a test as test.outputats.

Interrupting a Run

A control-C interrupt will terminate the program and all the tests it is running. Any test started but still not finished will be reported in RUNNING status.

Creating and Selecting Tests

Creating Tests

test(*args, **options)

This notation means that you can give positional, unnamed arguments, followed by keyword=value arguments.

• If you give just one positional argument, it is called “script”.

• If you give two, they are “script” and “clas”.

• If you do not give one or both positionally, they are given in the options, with their default values being blank strings.

It is an error to give more than two positional arguments.

Positional arguments are allowed for backwards compatibility – it is preferable to name everything.

In the test function call:

• script is a file name, which may be be relative to the directory containing the input file or absolute. Note that ATSROOT can be used in such names to designate either a preset environment value or the directory of the specified executable. The script if given will be used as the first argument on the test’s command line, and will supply a default name for the test.

• clas is a string giving the command-line arguments to be passed to the execution. Before doing so, python string interpolation is used with the options dictionary. This means, for example, that:

  test(clas = "-in %(input)s -parallelism %d", np=4, input='foo')
  

  will result in:

  clas = "-in foo -parallelism 4"
  

You might want to do this if, for example, this expression for clas was constant over many tests except for these variations of input and np. Then you could stick or glue this value for clas and not have to repeat it over and over.

Options can be any keyword = value pairs declaring the properties of this particular test; these are used in filtering and also serve as documentation for the test’s properties.

test returns an test object whose attribute ‘status’ is one of the following attributes of the ats module: CREATED, RUNNING, HALTED, PASSED, FAILED, TIMED, FILTERED, SKIPPED, BATCHED, INVALID.

Warning

Testing the truth value of a test object, such as using it in an if clause, causes the test to be marked FAILED. See testif below.

The test object will execute in the directory test.directory. This value can be set in the test options, but if it is not (which is usually the case) it is set to the directory in which the script resides, if the script is given. Otherwise it is set to the directory in which the test statement was read.

Note that if executable is 1, the script isn’t really a script, so directory is set to the directory in which the test statement was read.

testif(othertest, *args, **options)

This is the same as the test statement except that this test will only be run if othertest is eligible to run, has been run, and has been successful.

For example:

t = test('foo.py', 'dumpat=25')
testif(t, 'foo.py', 'restartat=25', label='restart test')

Explanation: This works because the test call returned a test object, t.

Expecting Failure

Sometimes you want to make sure a test will fail. To do this use the tilde (~) operator on the test:

~test(....)

The test will count as passed if its status ends up FAILED.

You can also set the expectedResult attribute of the test directly to something other than PASSED:

t = test(....)
t.expectedResult = TIMEDOUT

It is pointless to have a dependent of a test that is not expected to PASS. It will be SKIPPED.

Test Options

Each test can define arbitrary keyword = value pairs. With the exception of a few special options described below, the keyword names are arbitrary. Most options do not affect the running of the test, just the decision about whether or not to run it.

There are five lifetimes of option specification:

• defaults (often with command-line options to change the value),

• permanent (see glue and unglue),

• current and descendent files (see tack and untack)

• per sourced file (see stick and unstick), and

• per test (using the options portion of the test command).

Reserved option names

While you are free to use any desired scheme for options and filters, do not use the following names except for the purposes described. These are listed roughly in the order of their frequency of use by the end user.

label

label can be set to a string that will be appended to the name of the test to identify the test more fully. Thus, two different runs of the same script can be distinguished. label by default is the test’s serial number, the number that distinguishes the order in which the test was defined. labels are adjusted after all tests have been read to add distinguishing characters, so that no two tests have the same label.

name

This is the test name, as is printed out in the summary. If a script is given, it is that file name less the extension. Otherwise it defaults to the base name of the executable.

np

The option ‘np’ is reserved for specifying the number of processors to be used to run the program if the machine is a parallel processor. np = 0, the default, means a scalar run. np = 1 will be treated as a serial run on serial computers. np can be used in filters, e.g. np < 32.

executable

This option sets the path to the program to be run for this test. The default value of this option is usually set by the –exec command line option.

The executable program will be considered to have passed or failed depending on its exit status.

The executable may contain options after the path; it may also be given as a list of strings, the first component being the path and the rest options. If the path contains an internal space, you must use the list form.

Deprecated since version If: executable is 1, the first positional argument to the test function is the name of the executable program. It is preferable to use executable = /path/to/executable.

batch

This option is used to run a test in batch by setting it equal to 1 or True. Note that the filter batch (which you can set with the –filter batch command-line option) will restrict submissions to only batch jobs and the remaining non-batch jobs are skipped.

check

If check is not zero, this test is marked to be checked by hand rather than marked as passed, if it finishes normally. Such jobs are reported separately in the summary.

keep

If true, the test’s output files are kept even if it passed.

independent

If independent is True, the user is certifying that there is no obstacle to this test executing at the same time as any other test. Otherwise, by default tests are assumed to conflict with others in the same directory, because they might write files there with the same names as those read or written by other tests. If two tests conflict, they are never run at the same time. Judicious use of independent = True will increase ATS’s throughput. We suggest that while a stick(independent=True) may be appropriate, in some test files, to glue this definition may be reckless.

priority

By default the priority of a test is np + the sum of the priorities of and dependent jobs. The priority option lets you override this by giving an integer value. A value of zero means the test will be skipped.

env

By default the environment passed to the test will be the value of the ATS environment os.environ. To modify this dictionary, give the option env=D, with a value D that is a dictionary of the additions or changes to environment variables that you desire. If None, or not given, the default is used.

record

If a test is given option record=False, it is not reported as a separate test unless it fails in some way.

timelimit

Specifying a timelimit denotes maximum execution time for the test. For example, timelimit=”30m” will kill the test after 30 minutes and give it TIMEDOUT status.

SYSTEMS

SYSTEMS defaults to a list of one value. That value is the value of the “name” attribute of the machine object ATS has discovered. A filter:

  s in SYSTEMS

where s is this same value, is always used. Thus, by specifying SYSTEMS as
an option, the test will run only on the machines(s) named in SYSTEMS.

magic

magic controls the treatment of certain lines of test output. The default value is #ATS:.

If a test prints any lines beginning with the characters #ATS:, those lines will appear verbatim in the output, but also will be printed, less the #ATS: prefix, in the summary messages that appear when the test finishes.

If magic is set to None or a blank string, the entire parsing of the output file is skipped.

hideOutput

If true, do not print magic output lines in the log.

Extra Arguments On The Executable

If you want to always execute a given application with some fixed arguments in addition to others that vary, you may give them as part of the executable option to a test or on the command line. For example:

my_application = "/foo/bar -a -b"
test(clas="-d", executable=my_application)

will result in the execute line /foo/bar -a -b -d.

Be careful about quoting levels. For example, to make a test that did the equivalent of:

python -c "print '3+4'"

you must use an extra quotation level:

my_application = "python -c"
test(executable=my_application, clas = "\"print '3+4'\"")

Filters

A filter is a string that can be evaluated to a logical result. Filters can be defined with the command line option -f or –filter, or using the function filter. Helper functions can be defined using filterdefs.

Each test declares options: these are keyword = value pairs. To decide whether or not to execute a test, each filter is evaluated using Python’s eval function, in an environment consisting of these symbols:

• The options set by the test (including current ‘stuck’, ‘tacked’, and ‘glued’ option values described below)

• Symbols created parsing of text added by calls to filterdefs.

• The ats environment, consisting of these objects, which are each described in this document:

  manager, test, testif, source, log, filter, filterdefs, stick, unstick,
  tack, untack, glue, unglue,
  getGlue, getTack, getStick, sys, os, AtsError, AtsTest, abspath,
  is_valid_file, is_valid_executable, statuses,
  CREATED, RUNNING, INVALID, PASSED, HALTED,
  FAILED, BATCHED, SKIPPED, FILTERED,
  SYS_TYPE, MACHINE_TYPE, MACHINE_DIR, BATCH_TYPE,
  onExit, onSave, getResults.
  

• SELF is equal to the test object and some of its attributes may be interesting for filtering (name, label, basename).

If the filter returns true when evaluated, the test will be run. Otherwise, or if the filter gets a NameError when evaluated, the test will not be run.

Thus, a test run with:

test('mytest.py', x = 7)

would pass the filter ‘x==7’ but not pass the filter ‘x==5’ nor the filter ‘y==7’ (because the symbol y is not defined by the test).

Additional ATS Vocabulary

ATS input is written in a expanded dialect of Python. That dialect contains the following facilities.

Debugging and logging

debug([value = None])

debug() can be called in your input; it will return the current debug level: zero if –debug was not specified, or one if it was.

You can give debug an argument to set a new value, such as debug(2), and issue conditional code depending on the value which is returned by debug().

log(*items[, echo=False, logging = True])

The log written by ATS, and the terminal (in the form of stderr), can also be written to from user input. The log function adds a line to the log, using the enumerated items as if in print statement, unless logging is False. If echo is True, it prints to standard error.

With no items log prints a blank line.

For example:

log("I want to eat", 5, "donuts")

prints:

I want to eat 5 donuts

terminal(*items)

This is a version of log that writes only to the terminal.

Other methods and attributes in the log object are:

log.indent()

Increase the current indentation.

log.dedent()

Decrease the current indentation.

log.reset()

Reset indentation.

logging

A switch that controls logging to file

echo

A switch that controls logging to stderr.

Shortly after it gets organized, log sets the defaults for logging and echo. To be SURE you write something to stderr, use echo=True. And if you change logging or echo, or the indentation level, put things back as you found them, please.

It is not possible to log a partial line.

Manipulating Test Options

The following facilities provide for setting more-or-less persistent default values for test options. Each type listed will override the ones above it while it is still in scope.

1. A default value for most options is built in to ATS.

2. Command-line options override the default. Command-line options are not available for every test option, just the most important ones.

3. glued: Values set with a glue call. Such values apply until overridden by another glue call.

4. tacked: Values set with a tack call. These values apply until processing of the current file is finished, including in files sourced by this one.

5. stuck: Values set with a stick call. These values apply only in the file in which the call appears.

6. group: Values set with a group call. Such values can be overridden by an explicit value in the test. Group values last until the next group or endgroup, or the end of the source file.

7. explicit: Options given in a test or testif call always apply to that test.

Great care should be used with glued and tacked options, because they are not visible locally in files that are later sourced “from above”, and a person working on one of these files may not realize they are inheriting a value already that will take effect unless they override it. This will also cause the file to behave differently if used stand-alone as opposed to sourced from another file. Use the least scope that will get the job done for you.

Putting tests in groups has other consequences you should be aware of. See in particular directory blocking.

Here are the functions for controlling test option defaults:

stick(**keys)

Add the keyword = value pairs to the current dictionary of stuck test options. Stuck options persist until the end of the current file but do not apply in files sourced from this one.

A stuck option overrides a tacked or glued option, and is in turn overridden by an explicit option to test or testif.

tack(**keys)

Add the keyword = value pairs to the current dictionary of tacked test options. Tacked options persist until end of the current file and do apply in files sourced from this one.

A tacked option overrides a glued option, and is in turn overridden by a stuck value or by an explicit option to test or testif.

glue(**keys)

Add the keyword = value pairs to the current dictionary of glued test options.

Glued options apply to all subsequent test definitions. A glued option can be overridden by a stuck or tacked option, which in turn can be overridden by a value given in a test or testif statement.

Think of glued options as permanent changes to the default value of an option. One use might be to be sure every test has a value for some option name so that a filter can be constructed.

Notice the language here carefully. In the following example, the value which will be used in the test for the option color is “blue”:

stick(color = "blue")
glue(color = "red")
test("myscript", clas = "%(color)s")

The stuck option overrides the glued one of the same name.

Items can be removed from these dictionaries with:

unstick(*names)

Remove each name from the list of stuck options. If no list is given, remove all the stuck options.

untack(*names)

Remove each name from the list of tacked options. If no list is given, remove all the tacked options.

unglue(*names)

Remove each name from the list of glued options. If no list is given, remove all the glued options.

Filters are constructed with:

filter(*filters)

Add each string argument as a filter. With no arguments, delete all existing filters. Note that if you attempt to filter using the name of an option for which you have not set a default using the facilities above, then any test in which the option is not specifically set will be not be executed.

Each --filter command-line option is simply a call to this function.

The command-line option –skip allows you to test your filters without executing any tests.

To assist you in constructing filters we have:

getOptions()

Return a dictionary of the options as they would be seen by a test defined at the location of this call. Intended to aide debugging of options.

filterdefs(text=None)

Add result of parsing text to the filter environment. Usually used to add functions to use in filters. If text is None, clear the environment.

Despite the power available here, we recommend you don’t get too cute about it. The main thing is for it to be clear what is happening.

Customization

The Andyroid Tutorial contains ideas on various sorts of customization. These include defining your own postprocessor, main program, and application-specific input language extensions.

Using Levels

To use levels, make a master.ats file with stick commands separating the tests, such as this example input:

stick(level=10)
test("test1.py")
test("test2.py")

stick(level=20)
test("test3.py")
test("test4.py")
t5 = test("test5.py")

stick(level=30)
test("test6.p7")

# this test sets a level explicitly, that overrides the "stick".
testif(t5, "test7.py", level=10)

The currently “stuck” value is set in every test that does not explicitly set level. Thus test3, for example, has level 20, as if the level=20 were given in the test statement.

Executing ats on this file with the option –level 30 will execute all these tests. Executing ats with –level 15 will execute only test1 and test2; test7 depends on test5, which has level 20, so it will not be run even though it has level 10.

The Test Class

When a test is created by the test or testif command, a test object representing it is added to manager.testlist. This object is an instance of a class named AtsTest. Some users may wish to use the following details for debugging or postprocessors or customization.

The class AtsTest is available to users as ats.AtsTest.

AtsTest(*args, **options):

stuck, glued, tacked

These are the current dictionaries for determining test options.

test_number

The counter showing the number of tests defined so far.

serialNumber

The unique serial number of this test.

name

Set from an option to the test creation, or as the name of the script, or the name of the executable, plus the label. Eventually each test’s name is made unique.

label

Set from an option to the test creation, incorporated in the name if given.

options

The options for this test, after resolution using defaults, stuck, tacked, and glued.

depends_on

If not None, the test instance this one depends upon.

dependents

A list of any direct dependents of this test.

exited

Has the job been run and exited?

output

A list of lines of magic output, newlines and magic removed

notes

List of notes from the run; user feel free to append to this list.

..attribute:: level

Test level set from resolved options. Same as options.level.

np

Number of processors required. Same as options.np.

batchDic

A dictionary that may contain various things for a batch job.

clas

A string containing the command line arguments after option interpolation.

executable

An Executable object specifying the executable’s full path.

directory

The full path to the directory in which the test is executed.

groupNumber

The number of the group to which this test belongs, if positive.

groupSerialNumber

The number of the test within its group definition.

outname

The path to the standard output file for the test.

errname

The path to the standard error file for the test.

message

Explains the current value of status.

runOrder

A number indicating the order in which the interactive tests were run.

shortoutname

An abbreviated form of outname used for labeling.

timelimit

An object of class Duration – timelimit.value is the limit in seconds. Duration objects can be compared to integer numbers of seconds correctly.

waitUntil

A list of serial numbers of tests this one must wait for.

set(status, message)

Set the object’s status and message.

elapsedTime()

Returns a string, the formatted elapsed time of the run.

stick, unstick, glue, unglue, etc.

Class methods stick, unstick, glue, unglue, etc. are equivalent to the ones accessible in the vocabulary or ats module.

There are other methods that are not intended for end users.

Test Statuses

There are eleven status values that a test can have. This value is stored in the test’s attribute status. Collectively this set of a statuses is in the list ats.statuses and each of them individually is in module ats.

Each status has a four-character abbreviation, shown in parentheses. The status can also be accessed under this name in the ats module. For example, PASS and PASSED are the same object. You can correctly compare two statuses using “is” or “is not”, == or !=, or compare a status to a string representing its name or abbreviation, as in PASSED == "PASS".

The statuses are:

INVALID (INVD)

The test was not properly stated. For example, it referred to a script file that did not exist. See the log file for the error.

CREATED (INIT)

The test was created but not (yet) run.

PASSED (PASS)

The test was run and succeeded.

FAILED (FAIL)

The test was run and failed.

EXPECTED (EXPT)

The test ran and failed in an expected way.

TIMEDOUT (TIME)

The test ran longer than its timelimit and was killed.

SKIPPED (SKIP)

The test was created successfully but skipped for some reason. The reason is in the test object’s attribute message.

FILTERED (FILT)

The test was created successfully but filtered out for some reason. The reason is in the test object’s attribute message.

BATCHED (BACH)

The test was deemed eligible for batch processing, and has been shipped off to the batch system. ATS does not know its fate.

RUNNING (EXEC)

The test is running, or was running when an error or keyboard interrupt occurred.

HALTED (HALT)

The test was stopped after running successfully for one minute. This status is only possible if the --cutoff command-line option is used.

Postprocessing

After ATS has finished executing tests, but before it exits, it calls any Python routines that have been registered with it by calling:

manager.onExit(routine)

The routine should have the signature

def routine (manager):
...

The routine can do anything it wants. In particular, manager.testlist is available. Here’s an example of a trivial postprocessor in an input file:

def routine(manager):
   passedTests = [test for test in manager.testlist \
           if test.status is manager.PASSED]
   print [test.name for test in passedTests]
manager.onExit(routine)
source ("set1.ats")
source ("set2.ats")

The postprocessing file is designed to make it possible to run postprocessing functions of this kind using the state variable as the manager argument, rather than doing it as an onExit routine.

Test Suite Strategies

One of the problems with excessive choice is the paralzying effect of choice. There are a lot ways to do things with ATS. So here we describe a basic strategy to use until you have enough experience to form your own opinion.

We strongly urge that you read the Andyroid Tutorial as well.

This scheme assumes your code sources are destributed over a set of directories with a common parent called Home, with a subdirectory Test.

In each subdirectory with code that has a separate test (such as a unit test, or a test that emphasizes that coding) put a file with extension “ats”. This file contains a series of source statements that get further input or are test inputs containing introspective test statements).

::

test(clas = “-in myinput”, np = 1) source(“mysubdir/moretests.ats”)

Separate these inputs into levels with stick-level statements such as:

stick(level = 10)
...some tests...
stick(level=20)
...longer-running tests...
stick(level=30)
...still more...

You choose how many different levels you like. We recommend choosing well-spaced numbers in case you later change your mind and want to insert levels between the ones you start out with. Note that any test can still specify a level on its own that would override the stuck level.

As you go up your directory tree toward Home, put files that source the ones below it, until finally you have a tree leading to a file, say “testsuite.ats”, residing in your Home/Test directory.

Then you can make a series of small drivers. For example, your shortest test suite my be driven by this file:

glue("level <= 10")
source("testsuite.ats")

Running ats with this file as its input will result in only tests with level 10 or less being executed.

When the team that maintains a certain area wants to add a test, they add it to the closest member of the test-file tree relative to the source code they work with. They put it in the file at the appropriate level. This scheme leads to only rare source-code control conflicts, and ones that are usually a trivial merge; this avoids the conflicts generated by having a central test file.

Teams should be encourage to use introspection so that other members, less informed about how to test a certain area, can nevertheless exercise a good suite of tests using ATS, while allowing the experts to still use the input file directly with the code.

If there is one principle program being tested, it makes sense to use the -e option for it, and only explicitly specify an executable when it is different.

::

mycode = ‘/full/path/to/my/code’ test(executable=mycode, script=’foo.py’)

The extended example in Examples/Andyroid gives you many more ideas about how to use ATS.

Porting and Custom Machines

ATS decides on which machine characteristics to use by examining the value of the environment variable MACHINE_TYPE; or, if it is not defined, the value of the environment variable SYS_TYPE; or as default the value of Python’s sys.platform variable.

The reason for this three-level structure is to allow you to distinguish machine architectures when you have machines of the same basic type but with varied environments such as current OS level, parallel processing directives, or attached hardware. For an ordinary user on a personal computer, there is no reason to do anything special.

Most of the interaction between ATS and the platform takes place in a machine module, defined by default in the sources in file Lib/machines.py. Different behaviors are obtained by inheriting from this module, or one derived from it, and overriding various methods. We then connect our new machine module to a value for MACHINE_TYPE with a comment in our module file, and install that module in a directory in the Python distribution.

Porting ATS to a new platform is just one of the things you can do with the technique we describe in this section; you can also do things like doing something special when a job finishes, inventing your own scheduling algorithm, etc. You’ll need a decent knowledge of Python to do it, but you don’t need to be an expert.

If you invent a new value for MACHINE_TYPE, you can change the way ATS launches and finishes jobs and keeps track of resources, amongst other things. You can add command-line options and react to the user’s use of them. Your options will even appear when the user executes with --help.

To do this, you write a new Python source file, usually having a module name equal to your value for MACHINE_TYPE. This file must define a new child of machines.Machine, and you must have a comment:

#ATS:name module class npMax

This line or lines defines the relationship between a MACHINE_TYPE and this module’s machine class and provides the maximum number of jobs you wish to execute at once (or it may mean the maximum number of processors one job can use in a parallel programming environment):

• name is the name to match with MACHINE_TYPE.

• module is the name of the module file, or SELF.

• class is the name of the class in that module to use as a Machine.

• npMax is a limit on np; if this number is negative it is a suggested default only.

• machine.scheduler is created by the standard __init__ method of the machine. If you want to create your own scheduler you can replace this attribute. See Customizing the Scheduler below.

The file Lib/machines.py is well documented and it is usually not a large problem to get things working.

Once you have your module file ready, you write a setup.py file to go with it:

from distutils.core import setup
myMachines =[myMachine.py]   # list your machine module files
setup(name="myAtsAddon",
     author = "you",
     version = "1.0",
     description = "All About My Machine",
     data_files = [('atsMachines', myMachines)],
     scripts = ['mycustomdriver'],  #if you have one
)

and then execute python setup.py install. Set the environment variable MACHINE_TYPE and run ATS. It will report the machine module it has discovered.

In this setup.py file, the unchangeable word is atsMachines. This is the name of a directory below your Python installation root where the machine files are found by ATS. The scripts line can be omitted if you do not want to install your own driver.

Adding Test Options Via Machine

In a customized machine, the examineOptions routine can add entries to a dictionary, options.testDefaults. These will be default option values for each test. For example, here is how you would add an option nt that could be specified on the command line in the machine file:

def addOptions(parser):
    parser.add_option('--nt', dest='nt', default=1, type='int',
        help='Set default number of threads per test.')

def examineOptions(options):
   options.testDefaults['nt'] = options['nt']

Of course, the machine would also have to examine and use properly the value of each test’s option nt; but it would always have one, and hence it could be used in filters.

Customizing the Scheduler

The scheduler class StandardScheduler is defined in module schedulers. It handles issues such as priorities, and enforcing rules for the group() and wait() commands, and the independent option.

Customizing the scheduler is possible but difficult. It should in particular supply a method testlist() that returns the list of tests that are not yet completed. Inheritance is strongly suggested, so that you only change what you need to change. You’ll probably want to change the machine too so that it creates the correct scheduler, but it feasible to create and assign a new machine attribute scheduler at any point up to and including the call to machine.load.

The important thing is to maintain correct separation between the scheduler and the machine objects. The scheduler must ask the machine for such things as canRunNow that are within the purview of the machine, and ask it about whether jobs have finished. The machine contains an attribute running, a list of the jobs currently running. The periodicReport in the scheduler does the basic report once a minute; a machine can call this and then add more.

The ats Module

The ats module can be imported in custom drivers and postprocessors. Resources available in it are all imported from internal modules. These are documented further in the Appendix.

log, terminal

See the discussion of the log. terminal is simply a version of log that only writes to the terminal, not the log.

times

Is a module containing useful time-handling routines

configuration

Is the module that has information about the machine and command-line options.

manager

Is the manager object. It has in particular testlist, and the routines discussed above. It is defined in the management module.

testEnvironment

Is the vocabulary dictionary.

AtsTest

Is the test class.

debug(value=None)

Is the debug function

exception AtsError

Is the class of exceptions thrown by ATS.

statuses, CREATED, INVALID, PASSED, FAILED, HALTED, SKIPPED,

BATCHED, RUNNING, FILTERED, TIMEDOUT, SYS_TYPE, MACHINE_TYPE

Discussed previously, these are available via the ats module as attributes.

Using A Batch Facility

General Information

When running ATS, if a batch facility exists, both the interactive jobs and batch jobs will run. You have to use the facilities of that batch facility to find out what happened to those tests, because ATS will likely finish and exit long before those jobs are done.

Unfortunately, the world doesn’t have a standard batch facility. So here is an example of using the MSUB batch system at the Livermore Computing Center. Much of what follows would apply to any batch system.

To add a different batch system one must customize a batch machine to be installed in your ATS. For advice on how to do this, please contact us.

The basics are simple: if a test has a batch = 1 option, it is a batch test. Each of the batch tests are individually submitted to the batch system. The --allInteractive flag is available to execute such tests without using the batch system. Otherwise, they are simply skipped if no batch system is found.

For the LC system in particular,

• A testName.bat file is craated for the test.

• The test information is written to a “batchContinue.log”. This file will be a concatenation of all the batch tests and will provide information about the tests.

Running Entirely In Batch

Submitting a lot of single batch jobs may overwhelm some batch systems. In that case it may be preferable to submit just one big batch job. One batch job is created to run all the tests (both batch and interactive).

The ATS option --allInteractive is neccesary in the ATS command to prevent the tests from being submitted seperately as batch.

An example of a batch script using MSUB at LC:

#!/bin/csh

#MSUB -N tmpAts0.157456004499.job
#MSUB -j oe
#MSUB -o tmpAts0.157456004499.job.out
#MSUB -q pbatch
#MSUB -l nodes=4:ppn=16
#MSUB -l ttc=64
#MSUB -l walltime=200
#MSUB -V                    # exports all environment var
#MSUB -A myBank             # bank to use

setenv SYS_TYPE chaos_4_x86_64_ib

date
cd /my/work/directory/; atsb --allInteractive --numNodes=4  -useSrunStep Test/full.ats
date

The command-line options --numNodes=4 --useSrunStep are not a part of standard ATS. In this case, the ATS machine type chaos_4_x86_64_ib has been defined in a custom machine file, and custom machine files can add command-line options.

More Examples

Introspection

mytestA.py:
  #ATS:test(SELF, batch=1, np=2, ...)
  ...mytestA problem...

mytestB.py:
  #ATS:stick(batch=1)
  #ATS:test(SELF, ...)
  ...mytestB problem...

myAts.ats:
  tack(batch=1)
  source('mytestC.py')
  source('mytestD.py')
  source('mytestE.py')

In myytestA.py, a 2-processor batch job is created by introspection.

In mytestB.py, the test created through introspection will be run in batch, unless it happened to explicitly contain the option batch = 0, because the stick call makes batch = 1 the default in this file.

Running myAts.ats, the tack makes batch = 1 apply also in the three files that get read. If this were a stick, it wouldn’t apply inside those other files.

Test Control

Suppose the file mytest.py contains a test script. The script throws an exception if it gets an error. It has a command line argument delta. Suppose mytest.py reads:

#ATS:log('mytest.py tests sanity of my group leader.')
#ATS:test(SELF, 'delta=0.5')
#ATS:test(SELF, 'delta=0.6', sanitycheck = 1)
#ATS:test(SELF, 'delta=0.7', np=4, sanitycheck = 1)
import physics
...command line processing to get delta's value...
...test problem....
...throws an exception if it fails...

If we run:

ats --exec myapplication mytest.py

then it is equivalent to running 3 tests:

myapplication mytest.py delta=0.5
myapplication mytest.py delta=0.6
myapplication mytest.py delta=0.7

The last one is run on 4 processors if the machine supports it.

Consider the command line:

ats --exec myapplication -f 'sanitycheck == 0' mytest.py

None of the tests are run; the first because sanitycheck is not one of its options, the other two because it is but the value is not zero. We could make sanitycheck have a default value of zero for all tests in mytest.py by adding this line to the top of mytest.py:

#ATS:stick(sanitycheck=0)

With this line added we would run only the first test.

Using the filter sanitycheck==1 would run the last two tests but skip the first. Using the filter ‘not np’ would run only the first two jobs, since they have by default np == 0.

Suppose mytest.ats reads:

source('mytestA.py')
source('mytestB.py')

and mytestA.py reads:

#ATS:stick(batch=1)
#ATS:test(SELF,delta=0.1)
...mytestA problem....

and mytestB.py reads:

#ATS:test(SELF)
...mytestB problem....

If we run:

ats -e myapplication --nobatch mytest.ats

then only myTestB.py is executed, and execution of mytestA.py is skipped, since ats is not set for batch tests to run. Note --exec can be abbreviated as -e.

If we run:

ats -e myapplication mytest.ats

then mytestA.py is submitted to batch and mytestB.py is run interactively. If there is no batch system, mytestA.py is skipped.

In practice a batch facility, if present, would add further options for controlling itself, such as options to set accounts or priorities or timelimits. The maintainers of such batch facilities will provide the documentation for them.

Finally,

ats --allInteractive -e myapplication mytest.ats

will test both myTestA and myTestB.

Resources For Learning ATS

The Examples directory in the distribution contains the sources that accompany the Andyroid Tutorial, including some sample customizations.

The Test directory contains more examples, although care must be taken in reading them as some of these are designed to fail.

At your particular location you may find other directories that define machines and batch systems for your local computer center.

Quick Recipes

• To run only the batch tests:

  ats --filter 'batch == 1'  mytest.ats
  

• To run only the interactive tests:

  ats --nobatch  mytest.ats
  

• To run all tests as interactive tests:

  ats --allInteractive mytest.ats
  

• To check your input add –skip; add –debug for even more information.

• To keep the output files even if the test succeeds, add –keep