Reference Material¶
Test Selection and Execution¶
Tests are defined in ATS input using two commands, test
and its little
brother, testif
. However, not every test that gets defined is necessarily
going to be executed. The user can define logical conditions (filters) that
a test must satisfy to be chosen for execution, and the hardware available may
cause others to be skipped.
In order to make it easier to structure suites of tests, there is an elaborate
set of facilities involving filters, command-line options, and arguments to
test
statements, as well as facilities for grouping and ordering your test
executions.
ATS Execution and Command-line Options¶
In specifying the names of input files, you can give the filename or omit
the filename extension. ATS will attempt to find the file
using its name, then with a .ats
extension, and then with a .py
extension.
Unix or Mac¶
To start ATS on a Unix system or Mac, execute this line in a terminal window:
ats [options] [input files]
Note that the --exec
option is frequently used to define a default
executable, but any given test can specify any executable as the program to be
tested.
Before executing ATS, it may be desirable to have defined the environment variables MACHINE_TYPE and / or SYS_TYPE; and there may be others for testing particular executables. Please consult with the owner of your local ATS installation, and the owners of any custom ATS drivers you may be using.
Windows¶
Execution on windows can be done in the same way from a command window, but
can be made more convenient by defining a .bat
file, such as:
C:\python27\python c:\python27\ats $*
These instructions need improvement as the first Windows users determine the right way to do this.
Command-line Options¶
What follows are the the most important command-line options available in any ATS installation.
Note
The exact set of command-line options depends on the machine you are using
and / or upon any custom driver you are using for testing a particular
program. To see the complete list for a given ATS installation, enter
ats --help
.
This will also show you abbreviations for some of the options.
- --allInteractive
Run every test in interactive mode.
- --cutoff cutofftime
Over-rides the timelimit for all jobs no matter where the timelimit is set. Jobs that fail once reaching the cutoff will TIMEOUT. The forms for giving the time are the same as for
--timelimit
. Note: Jobs that TIMEOUT are marked as FAIL when using Flux.- --debug
Debug mode; prints more information in the log and on the shell window.
- --exec EXEC
Give the path to the code to be tested. The path is tilde- and dollar-expanded.
This option sets the environment variable ATSROOT, if not already set, to the directory in which the executable resides. Most of the time this option is used, and the executable so named is referred to in this documentation as the specified executable.
However, tests with different executables can also be specified, by using the executable=’/path/to/my/code’ as one of the test options. The purpose of ATSROOT is to allow you to specify related tools for your code that are located in the same directory as the executable. In specifying a test, you can use this variable in the script or executable using either $ATSROOT or %(ATSROOT).
Note that you don’t have to have one main code to be tested. You can specify a different executable for each test, or group of tests.
- --filter FILTER
Add a filter; may be repeated. Be sure to use quotes if the filter contains spaces and remember that the shell will remove one level of quotes.
- --glue FILTER
Has the effect of executing glue(FILTER) before execution of the tests. May be repeated. Be sure to use quotes if the filter contains spaces and remember that the shell will remove one level of quotes. The glue function is used to set persistent test option defaults.
- --help
Show the list of options and exit. There may be more options than are shown in this document, such as batch or node control options.
- --info
Print information about ATS, such as version, path to the executable, and some parameter values.
- --keep
Keep the output files from the tests that succeed. Normally the output from tests that fail, or which must be checked, is kept.
- --logs LOGDIR
Sets the name of the log directory. The default log directory is arch.time.logs, where arch will be an architecture-dependent name, and time will be digits of the form yymmddhhmmss. All logs and the continuation file are placed in this directory. The log itself is named ats.log.
- --level LEVEL
Set the maximum level of test to run. Level is simply a built-in easy-to-use filter.
- --skip
Skip actual execution of the tests, but show filtering results and missing test files, and show additional details about the input.
- --nobatch
Do not run any “batch” tests..
- --nosrun
Run the code without srun. This option can also be used on BlueOS to run ALL test on a login node as it circumvents the login node check. If the tests need to be run on a working node, then the tests themselves will need to get an allocation.
- --npMax value
Value is an integer, the maximum number of tests to run at once (on a node, if multinode machine). Some machines allow you to set this higher than the actual number of nodes, at your own risk.
- --okInvalid
Run tests even if there is an invalid test. Examples are tests specifying missing scripts or executables.
- --oneFailure
Stop if a test fails.
- --removeStartNote
Removes the messages printed at the start of a test running.
- --removeEndNote
Removes the message printed at the end of a test running. Will still get results printed with pass/fail, just no “stop”.
- --serial
Run only one job at a time.
- --timelimit TIMELIMIT
Set the default
timelimit
test option. TIMELIMIT may be given as an integer number of seconds or a string specification such as ‘2m’, or ‘3h30m20s’. A similar notation can be used for filtering by time limit, such as -f ‘timelimit < “30m”’. Note: Jobs that TIMEOUT are marked as FAIL when using Flux.- --verbose
Verbose mode. Both starts and finishes of tests are noted on the terminal, plus other reports. Test failures are reported regardless.
- --version
Show program’s version number and exit.
Basic Operations¶
The goal of ATS is to execute a series of test problems. It does this by
reading input files written in the Python language, with some predefined ATS
functions added. In particular, ATS supplies a function named test
. Each
execution of the test
statement defines a particular program to execute,
including its command line and a variety of options used by ATS to know
how to run it or to decide not to run it.
After running the tests, the ats prints a summary of which tests have passed (that is, returned with a normal exit status) and which have failed.
The second basic statement is the source
statement, which causes a
file to be read containing additional commands. An introspection
procedure, described below, is also available to allow scripts meant as
problem input to contain definitions of how they are to be run when run by
ATS.
Retrying Failed Tests¶
If any tests fail or are not completed, a “continuation” file is written and a message issued in the summary section giving the name of the file. The continuation file is named continue.ats and it is inside the log directory.
You can rerun the exact same ATS command, adding the path to the continuation file as an extra command-line argument.
Note
You must run the exact same command with this added argument at the end of the command line.
Doing this will redo those families of tests that had a failed member. This process may be repeated until all tests pass. In your log, tests that had passed before well be marked “Previously passed” and batch jobs will be “skipped”. The continuation file is pretty self-explanatory and you can edit it with thought.
Note that if a descendent of a test failed, the test will be rerun because the error might have been in files produced by the parent test, even though it appeared to pass.
The intention of this facility is to let you fix your code without having to rerun all your tests. For correctness, you should rerun everything once you believe you have corrected all errors.
Results Facility¶
Each run creates an atsr.py
file in the log directory. This file, if
run under Python, creates one variable named “state”, which is an
object that is a dictionary whose values can be read and written using
either dictionary or attribute notation. This type is called an
AttributeDict.
The object state has attributes corresponding to the major features of
the manager object, including a machine
and testlist
, which is
a list of AttributeDicts, each encapsulating the major properties of
each test.
Two methods in the manager object control this facility, which may be used by custom drivers.
- onSave(saver)¶
Registers a
function saver(results, manager)
, which will be called when the data for the state is collected. It may modify the AttributeDictresults
in any way it likes, usually by adding to it. Callingresults.clear()
would be a way of minimizing the use of resources devoted to this file.onSave is available in the test environment also, for use in input files. Note that the call does not cause the save of the file at the time it is executed.
Three other manager methods can be called from custom drivers.
- getResults()¶
Returns the AttributeDict containing the state. The manager’s
machine
and, if set,batchmachine
, are given a change to contribute fields to the end result, and finally any onSave-registered routines are called in the order they were registered.
- saveResults(filename='atsr.py')¶
Save the state to a file using given file name; if not absolute, put it in the log directory.
- printResults(file=sys.stdout)¶
Do the actual job of writing the state file. Here file should be an open file handle. You would only use this function if you wanted to add something to the file other than the
state
variable.
Normally saveResults
creates the file and asks printResults
to
call getResults
and print the returned state into the file, preceded
by a header that imports the symbols in the ats
module so that the code
will execute correctly.
Interactive inspection of the resulting file is most easily accomplished with an interactive Python session, such as:
cd <logdirectory>
python -i atsr.py
print "Number of tests = ", len(state.testlist)
print "Machine name", state.machine.name
print "Number timed out", \
len([t for t in state.testlist if t.status == TIMEDOUT])
Note that ATS statuses will compare equal if they compare to another status or the name or the abbreviation. So in the last line above, TIMEDOUT, “TIME”, or “TIMEDOUT” would all work.
To compare different files you can rename state as you read it:
d= {}
execfile("atsr.py", d)
state1 = d['state']
You can change the name of the file to be used by setting manager.saveResultsName in your input file. If not an absolute path, the file will be created in the logs directory.
Controlling Input¶
File Sourcing¶
- source(*paths, **vocabulary)¶
Process one or more paths as if each was the name of an input file given on the command line. (This function is the same as manager.source)
The current stuck options are saved upon entry, cleared before beginning processing, and then restored on completion. See stick below for further details.
Path names are expanded both for tilde and environment-variable names using the dollar sign.
The vocabulary items can be any number of keyword = value pairs.
Vocabulary words are added to the environment in which input files are compiled by Python. The scope of this environment is just within the input of the paths given to this source command. To add a vocabulary value to all subsequent source commands, use the define command, described next.
The vocabulary word introspection can be used to change the commenting convention used for ATS’ introspection facility. Details are given below.
- define(keyword=value, ...)¶
adds one or more keywords to the vocabulary used by the source command to parse input. This is the same function as manager.define.
- undefine(keyword, ...)¶
removes one or more keywords from the vocabulary used by the source command to parse input. This is the same function as manager.undefine.
- showDefine(*keywords, **options)¶
logs the current definition of one or more keywords in the vocabulary used by the source command. If no argument is given, all the definitions are shown. This function is used to help debug your vocabulary setup. The options may include echo and logging, and are passed on to the call to log. The defaults are both True. This is the same function as manager.showDefine.
A file may be ‘sourced’ because it was given on the command line or
because a source
function was executed with it as an argument. (Note: In
what follows it is is assumed that a line that starts #ATS: is a comment to
your application; however, it is possible to change the commenting convention to
suit your input convention, using the second argument to source
.
Examining and prioritizing tests¶
After collection of the tests the user may wish to examine or alter the
tests before they are executed. This is done by registering one or more
routines to be called (in the order in which they were registered) by using
onCollected
. See also onPrioritized
, below.
- onCollected(routine)¶
The routine registered is called when the input is complete. It is given the
manager object as its single argument. The routine thus has access to the
manager.testlist
.
The routine may make use of the routine that ATS itself is about to use to divide the tests into interactive and batch tests:
interactiveTests, batchTests = manager.sortTests()
You can effect what happens next by changing statuses (such as setting the
status to BATCH or FILTERED or CREATED (i.e., interactive)) or change
totalPriority
(see below).
You also have a chance at this point to use each test’s directory
attribute
to prepare the file system, or to build data structures for later use in a postprocessor.
Use this facility with caution. Do not attempt to change tests that would not have executed at all into ones that will. If you change a label it must be unique when you are done. Do not alter serial or group numbers.
After the onCollected
actions, the scheduler prioritizes the interactive
tests. The totalPriority
attribute of each test is set to the sum of the
test’s own value plus the sum of the priorities of each test that must wait
for this one to complete. (Such conditions are created by dependencies or
wait
or group
commands.)
The user may wish to examine or alter the priorities of the tests
tests before they are executed. This is done by registering one or more
routines to be called (in the order in which they were registered) by using
onPrioritized
.
- onPrioritized(routine)¶
The routine should take a single argument, interactiveTests. The intent is for
the user to examine or alter the totalPriority
attribute of a test.
Altering priority
attributes will not work. Altering anything else about
the test is probably ill-advised.
In summary, there are two ways to change the totalPriority
attribute:
in an onCollected
routine, which will contribute the new value to its
predecessors, or in an onPrioritized
routine, where you are setting the
final absolute value.
Using Introspection¶
When a file is sourced, ATS looks to see if the file contains any lines that begin with the five characters #ATS:. If so, the set of such lines with the leading #ATS: removed will be executed as Python code. The remainder of the file will be ignored. This procedure is called introspection.
Note that Python’s indentation rules apply, so there should not be any spaces after the #ATS: except on lines that should be indented.
For example, continuation of lines is allowed in the normal Python manner:
#ATS:test('myfile.py',
#ATS: 'my command line args',
#ATS: np = 4)
Picture the first five characters as defining the left edge of the lines to be executed.
During this procedure, the symbol SELF will be defined to be the name of the file being sourced. Thus a line such as:
#ATS:test(SELF, 'command line options', np=4, w=2)
will cause the file to be tested with the given command line, using the options np = 4 and w = 2 as context for filtering.
A file may contain many such lines, in order to exercise the same test with a variety of parameters. Also note that not all the #ATS: lines need to be ATS commands; they can be any Python code. They can also include log commands, source other files, etc.
Changing the introspection convention¶
If a value for the vocabulary word “introspection” is given, it should be a python function which, when given a line, returns None or the value of the line as introspection. The default is a function that returns None unless the line begins with #ATS:, in which case it returns the line less that prefix.
By prescribing your own value for introspection, you can allow the introspection process to work on source files with a different commenting convention than “#”.
In particular, to change the default function used for introspection, just use define after you declare it. For example:
def asteriskinterpolation(line):
"Any line that starts with *ATS: is magic"
if line.startswith("*ATS:"):
return line[5:]
else:
return None
define(interpolation=asteriskinterpolation)
Grouping¶
If you have a test that creates some files for postprocessing, you can group that test with the related ones.
You begin with:
- group(independent=False, report=False, **kw)¶
and after defining some tests, finish with:
- endgroup()¶
A group is also ended by another group statement, or the end of the current input file.
The arguments to the group
call become default options for each test defined
inside the group. They can be overridden by options in the test
and testif
statements within the group.
Only the first test result will be included in the final reports unless some member of the group fails, or you change the report argument to True. The output files of the entire group will be kept if anything fails; otherwise the usual keep options will prevail.
The independent
test option determines if a test will block any other test (other than
ones in its group) that uses the same directory. By default, then, a group
will lock-out any non-independent test or group from running in the directory or
directories its tests use. This is not different than the default behavior of ATS,
but is a convenience for making sure that the members of the group will not be
interleaved with other, non-independent tests that use the same directories, if you
have glued or tacked or stuck independent to be be True.
These two arguments are used as test options for all tests in the group, but for any particular test can be overridden by an explicit option in the test statement itself.
Note that grouping does not make each test depend on the preceding tests in the group. Two members of the group may execute together. It also does not make the failure of one test skip another. To achieve dependency, use the ‘testif’ facility.
Wait¶
It is certainly possible to make two tests that appear to be independent but which
cannot in fact run simultaneously. ATS prevents many cases of this due to its reluctance
to run two tests in the same directory at the same time. If that fails to solve the
problem, and the group
or the testif
statements are not sufficient, you can try
the wait
statement:
- wait()¶
All the tests defined so far in this source file will be finished before proceeding to any tests defined later in this source file. Tests defined in other files that are sourced after the ‘wait’ must also wait for all the tests before the wait in this source file.
wait() may be a useful way to express massive dependencies without using excessive testif calls. However, if used excessively, wait may cripple ATS’s ability to run tests simultaneously.
You can debug your wait structure with this command:
ats yoursource --skip
This will show a list at the end of the log file, under “ATS RESULTS”, showing the serial numbers being waited for by each test.
When all tests are completed, ATS issues a final report and runs any postprocessors that have been registered using the onExit facility described later.
Example¶
Suppose we have this test file “waitforit.ats”:
glue(executable = "/bin/ls")
test(label='first')
test(label='second')
wait()
test(label='third')
Then the third test will not execute until the first two are done – but this says nothing about the order in which the first two will execute.
Suppose now we add a source of another file, so we have:
glue(executable = "/bin/ls")
test(label='first') #1
test(label='second') #2
wait()
source('waitfor1.ats')
test(label='third') #6
with the file being sourced containing:
test(label='waitfor1 first') #3
test(label='waitfor1 second') #4
wait()
test(label='waitfor1 third') #5
We have thus defined six tests in all. The output of the debugging process is:
Interactive tests:
#1 INIT ls(first) ready
[]
#2 INIT ls(second) ready
[]
#3 INIT ls(waitfor1 first) ready
[1, 2]
#4 INIT ls(waitfor1 second) ready
[1, 2]
#5 INIT ls(waitfor1 third) ready
[1, 2, 3, 4]
#6 INIT ls(third) ready
[1, 2]
The parts in square brackets are lists of the tests this one must wait for.
(The list will include any tests of which this one is a dependent.)
So we see for example that #6
, the last test in the main file, waits for the
first two tests, because a wait()
occurs after #2
, but it is
not affected by the wait statement in the sourced file. In that file
the first two tests are waiting for the first two, and the third waits for
the first four.
Executing Tests¶
ATS attempts to execute as many tests as it can at the same time in order to keep
the computational resources it has been given busy, subject to respecting the
test options priority
and independent
, and the group
and wait
statements.
The following sections describe this process.
Scheduling¶
After the ATS has read all the input and knows what tests are to be run, it examines the collection and combines the information generated by the group, and wait commands with the test dependencies to figure out which tests must execute before others. It can then combine the priorities of tests to determine a preferred order of execution – which however will be subject to processor availability.
This work is done by a scheduler object. A standard scheduler is provided, and is an
attribute on the machine
object. A user could potentially modify it by inheritance from
its defining class, schedulers.StandardScheduler
.
Each test has a priority. By default the scheduling priority (totalPriority
)
is the number of processors required by the test plus the priorities of any tests which
cannot execute until this one is finished. In this way those tests with a lot of dependents
are started early.
A test may specifiy its priority as an option “priority=n” where n is a nonzero integer. A test whose priority is zero or less will not be run. Thus, a long-running 1-processor job without dependents might profit from being given a priority, say 3, so that it starts earlier. Note that an np = 0 job requires 1 processor.
As tests are selected to be started, the highest-priority job that will fit on an available machine is chosen. You can examine the tests in postprocessing if you want to understand what influenced the scheduling:
Test option priority,
Test attribute totalPriority,
Test attribute group,
Test option independent (described below)
Test attribute runOrder, an integer indicating the order of test launch.
Note
Important: by default two tests will not be run in the same directory at the same time.
This is a modestly conservative scheme to avoid common resource conflicts when testing one file with different parameters.
If you know a test does not have such a problem, you can give it the option
independent = True
. Note that the group
command makes the default value of
independent
False for all members of the group, overriding anything except an actual
option in the test statement. Thus if you do not want this behavior for the group
you must use independent = True as an argument in your group command.
The standard scheduler sorts the groups by the highest priority test in the group. In effect, every member of a group behaves as if it has the priority of the highest-priority test in the group. This ensures a large prejudice towards running members of a group once it has started, until they are all complete.
Progress Reports¶
When a test starts this fact is shown on the terminal output. You can use the command
option --verbose
to cause test completions and other additional events to be reported
as well. All the information is always in the log. Additional output is generated by
the --debug
option.
Every minute ATS issues a report on its progress to the terminal only.
Output Files¶
The standard output and standard error of a test are written into files in the directory where the logs are written. These files are (usually) removed when the test concludes successfully; for a group, this occurs when all members of the group have succeeded.
The name and label of the test script or executable, along with the test’s serial number, are used to create the file names.
The –keep option prevents the removal of these output files even when the tests are successful. They are also kept if the test has the option keep=True or check=True.
See also
Postprocessors set using the onExit facility can access the magic output of a test as test.outputats.
Interrupting a Run¶
A control-C interrupt will terminate the program and all the tests it is running. Any test started but still not finished will be reported in RUNNING status.
Creating and Selecting Tests¶
Creating Tests¶
- test(*args, **options)¶
This notation means that you can give positional, unnamed arguments, followed by keyword=value arguments.
If you give just one positional argument, it is called “script”.
If you give two, they are “script” and “clas”.
If you do not give one or both positionally, they are given in the options, with their default values being blank strings.
It is an error to give more than two positional arguments.
Positional arguments are allowed for backwards compatibility – it is preferable to name everything.
In the test function call:
script is a file name, which may be be relative to the directory containing the input file or absolute. Note that ATSROOT can be used in such names to designate either a preset environment value or the directory of the specified executable. The script if given will be used as the first argument on the test’s command line, and will supply a default name for the test.
clas is a string giving the command-line arguments to be passed to the execution. Before doing so, python string interpolation is used with the options dictionary. This means, for example, that:
test(clas = "-in %(input)s -parallelism %d", np=4, input='foo')will result in:
clas = "-in foo -parallelism 4"
You might want to do this if, for example, this expression for clas was constant over many tests except for these variations of input and np. Then you could stick or glue this value for clas and not have to repeat it over and over.
Options can be any keyword = value pairs declaring the properties of this particular test; these are used in filtering and also serve as documentation for the test’s properties.
test returns an test object whose attribute ‘status’ is one of the following attributes of the ats module: CREATED, RUNNING, HALTED, PASSED, FAILED, TIMED, FILTERED, SKIPPED, BATCHED, INVALID.
Warning
Testing the truth value of a test object, such as using it in an if clause, causes the test to be marked FAILED. See testif below.
The test object will execute in the directory test.directory. This value can be set in the test options, but if it is not (which is usually the case) it is set to the directory in which the script resides, if the script is given. Otherwise it is set to the directory in which the test statement was read.
Note that if executable is 1, the script isn’t really a script, so directory is set to the directory in which the test statement was read.
- testif(othertest, *args, **options)¶
This is the same as the test statement except that this test will only be run if
othertest
is eligible to run, has been run, and has been successful.
For example:
t = test('foo.py', 'dumpat=25')
testif(t, 'foo.py', 'restartat=25', label='restart test')
Explanation: This works because the test call returned a test
object, t
.
Expecting Failure¶
Sometimes you want to make sure a test will fail. To do this use the tilde (~) operator on the test:
~test(....)
The test will count as passed if its status ends up FAILED.
You can also set the expectedResult
attribute of the test directly to something
other than PASSED:
t = test(....)
t.expectedResult = TIMEDOUT
It is pointless to have a dependent of a test that is not expected to PASS. It will be SKIPPED.
Test Options¶
Each test can define arbitrary keyword = value pairs. With the exception of a few special options described below, the keyword names are arbitrary. Most options do not affect the running of the test, just the decision about whether or not to run it.
There are five lifetimes of option specification:
defaults (often with command-line options to change the value),
permanent (see glue and unglue),
current and descendent files (see tack and untack)
per sourced file (see stick and unstick), and
per test (using the options portion of the test command).
Reserved option names¶
While you are free to use any desired scheme for options and filters, do not use the following names except for the purposes described. These are listed roughly in the order of their frequency of use by the end user.
- label
label can be set to a string that will be appended to the name of the test to identify the test more fully. Thus, two different runs of the same script can be distinguished. label by default is the test’s serial number, the number that distinguishes the order in which the test was defined. labels are adjusted after all tests have been read to add distinguishing characters, so that no two tests have the same label.
- name
This is the test name, as is printed out in the summary. If a script is given, it is that file name less the extension. Otherwise it defaults to the base name of the executable.
- np
The option ‘np’ is reserved for specifying the number of processors to be used to run the program if the machine is a parallel processor. np = 0, the default, means a scalar run. np = 1 will be treated as a serial run on serial computers. np can be used in filters, e.g. np < 32.
- executable
This option sets the path to the program to be run for this test. The default value of this option is usually set by the –exec command line option.
The executable program will be considered to have passed or failed depending on its exit status.
The executable may contain options after the path; it may also be given as a list of strings, the first component being the path and the rest options. If the path contains an internal space, you must use the list form.
Deprecated since version If: executable is 1, the first positional argument to the test function is the name of the executable program. It is preferable to use executable = /path/to/executable.
- batch
This option is used to run a test in batch by setting it equal to 1 or True. Note that the filter batch (which you can set with the –filter batch command-line option) will restrict submissions to only batch jobs and the remaining non-batch jobs are skipped.
- check
If check is not zero, this test is marked to be checked by hand rather than marked as passed, if it finishes normally. Such jobs are reported separately in the summary.
- keep
If true, the test’s output files are kept even if it passed.
- independent
If independent is True, the user is certifying that there is no obstacle to this test executing at the same time as any other test. Otherwise, by default tests are assumed to conflict with others in the same directory, because they might write files there with the same names as those read or written by other tests. If two tests conflict, they are never run at the same time. Judicious use of independent = True will increase ATS’s throughput. We suggest that while a stick(independent=True) may be appropriate, in some test files, to glue this definition may be reckless.
- priority
By default the priority of a test is np + the sum of the priorities of and dependent jobs. The priority option lets you override this by giving an integer value. A value of zero means the test will be skipped.
- env
By default the environment passed to the test will be the value of the ATS environment
os.environ
. To modify this dictionary, give the option env=D, with a value D that is a dictionary of the additions or changes to environment variables that you desire. If None, or not given, the default is used.- record
If a test is given option record=False, it is not reported as a separate test unless it fails in some way.
- timelimit
Specifying a timelimit denotes maximum execution time for the test. For example, timelimit=”30m” will kill the test after 30 minutes and give it TIMEDOUT status.
- SYSTEMS
SYSTEMS defaults to a list of one value. That value is the value of the “name” attribute of the machine object ATS has discovered. A filter:
s in SYSTEMS where s is this same value, is always used. Thus, by specifying SYSTEMS as an option, the test will run only on the machines(s) named in SYSTEMS.
- magic
magic controls the treatment of certain lines of test output. The default value is
#ATS:
.If a test prints any lines beginning with the characters #ATS:, those lines will appear verbatim in the output, but also will be printed, less the #ATS: prefix, in the summary messages that appear when the test finishes.
If magic is set to None or a blank string, the entire parsing of the output file is skipped.
- hideOutput
If true, do not print magic output lines in the log.
- same_node
ONLY WORKS ON FLUX (ATS can run Flux under slurm). Specify a string identifier for the tests that you want to be run on the same node. Useful for tests that depend on some data output by another test that might not be accessable from other nodes. NOTE: Using this option will limit -N 1 and -n to max on one node, if more than that were requested. Ex: same_node=’abc’
Extra Arguments On The Executable¶
If you want to always execute a given application with some fixed arguments in addition to others that vary, you may give them as part of the executable option to a test or on the command line. For example:
my_application = "/foo/bar -a -b"
test(clas="-d", executable=my_application)
will result in the execute line /foo/bar -a -b -d
.
Be careful about quoting levels. For example, to make a test that did the equivalent of:
python -c "print '3+4'"
you must use an extra quotation level:
my_application = "python -c"
test(executable=my_application, clas = "\"print '3+4'\"")
Filters¶
A filter is a string that can be evaluated to a logical result. Filters can be defined with the command line option -f or –filter, or using the function filter. Helper functions can be defined using filterdefs.
Each test declares options: these are keyword = value pairs. To decide whether or not to execute a test, each filter is evaluated using Python’s eval function, in an environment consisting of these symbols:
The options set by the test (including current ‘stuck’, ‘tacked’, and ‘glued’ option values described below)
Symbols created parsing of text added by calls to filterdefs.
The ats environment, consisting of these objects, which are each described in this document:
manager, test, testif, source, log, filter, filterdefs, stick, unstick, tack, untack, glue, unglue, getGlue, getTack, getStick, sys, os, AtsError, AtsTest, abspath, is_valid_file, is_valid_executable, statuses, CREATED, RUNNING, INVALID, PASSED, HALTED, FAILED, BATCHED, SKIPPED, FILTERED, SYS_TYPE, MACHINE_TYPE, MACHINE_DIR, BATCH_TYPE, onExit, onSave, getResults.SELF is equal to the test object and some of its attributes may be interesting for filtering (name, label, basename).
If the filter returns true when evaluated, the test will be run. Otherwise, or if the filter gets a NameError when evaluated, the test will not be run.
Thus, a test run with:
test('mytest.py', x = 7)
would pass the filter ‘x==7’ but not pass the filter ‘x==5’ nor the filter ‘y==7’ (because the symbol y is not defined by the test).
Additional ATS Vocabulary¶
ATS input is written in a expanded dialect of Python. That dialect contains the following facilities.
Debugging and logging¶
- debug([value = None])¶
debug() can be called in your input; it will return the current debug level: zero if –debug was not specified, or one if it was.
You can give debug an argument to set a new value, such as debug(2), and issue conditional code depending on the value which is returned by debug().
- log(*items[, echo=False, logging = True])¶
The log written by ATS, and the terminal (in the form of stderr), can also be written to from user input. The log function adds a line to the log, using the enumerated items as if in print statement, unless logging is False. If echo is True, it prints to standard error.
With no items log prints a blank line.
For example:
log("I want to eat", 5, "donuts")
prints:
I want to eat 5 donuts
- terminal(*items)¶
This is a version of
log
that writes only to the terminal.
Other methods and attributes in the log object are:
- log.indent()¶
Increase the current indentation.
- log.dedent()¶
Decrease the current indentation.
- log.reset()¶
Reset indentation.
- logging¶
A switch that controls logging to file
- echo¶
A switch that controls logging to stderr.
Shortly after it gets organized, log sets the defaults for logging and echo. To be SURE you write something to stderr, use echo=True. And if you change logging or echo, or the indentation level, put things back as you found them, please.
It is not possible to log a partial line.
Manipulating Test Options¶
The following facilities provide for setting more-or-less persistent default values for test options. Each type listed will override the ones above it while it is still in scope.
A default value for most options is built in to ATS.
Command-line options override the default. Command-line options are not available for every test option, just the most important ones.
glued: Values set with a glue call. Such values apply until overridden by another glue call.
tacked: Values set with a tack call. These values apply until processing of the current file is finished, including in files sourced by this one.
stuck: Values set with a stick call. These values apply only in the file in which the call appears.
group: Values set with a group call. Such values can be overridden by an explicit value in the test. Group values last until the next
group
orendgroup
, or the end of the source file.explicit: Options given in a test or testif call always apply to that test.
Great care should be used with glued and tacked options, because they are not visible locally in files that are later sourced “from above”, and a person working on one of these files may not realize they are inheriting a value already that will take effect unless they override it. This will also cause the file to behave differently if used stand-alone as opposed to sourced from another file. Use the least scope that will get the job done for you.
Putting tests in groups has other consequences you should be aware of. See in particular directory blocking.
Here are the functions for controlling test option defaults:
- stick(**keys)¶
Add the keyword = value pairs to the current dictionary of stuck test options. Stuck options persist until the end of the current file but do not apply in files sourced from this one.
A stuck option overrides a tacked or glued option, and is in turn overridden by an explicit option to
test
ortestif
.
- tack(**keys)¶
Add the keyword = value pairs to the current dictionary of tacked test options. Tacked options persist until end of the current file and do apply in files sourced from this one.
A tacked option overrides a glued option, and is in turn overridden by a stuck value or by an explicit option to
test
ortestif
.
- glue(**keys)¶
Add the keyword = value pairs to the current dictionary of glued test options.
Glued options apply to all subsequent test definitions. A glued option can be overridden by a stuck or tacked option, which in turn can be overridden by a value given in a test or testif statement.
Think of glued options as permanent changes to the default value of an option. One use might be to be sure every test has a value for some option name so that a filter can be constructed.
Notice the language here carefully. In the following example, the value which will be used in the test for the option color is “blue”:
stick(color = "blue")
glue(color = "red")
test("myscript", clas = "%(color)s")
The stuck option overrides the glued one of the same name.
Items can be removed from these dictionaries with:
- unstick(*names)¶
Remove each name from the list of stuck options. If no list is given, remove all the stuck options.
- untack(*names)¶
Remove each name from the list of tacked options. If no list is given, remove all the tacked options.
- unglue(*names)¶
Remove each name from the list of glued options. If no list is given, remove all the glued options.
Filters are constructed with:
- filter(*filters)¶
Add each string argument as a filter. With no arguments, delete all existing filters. Note that if you attempt to filter using the name of an option for which you have not set a default using the facilities above, then any test in which the option is not specifically set will be not be executed.
Each
--filter
command-line option is simply a call to this function.The command-line option –skip allows you to test your filters without executing any tests.
To assist you in constructing filters we have:
- getOptions()¶
Return a dictionary of the options as they would be seen by a test defined at the location of this call. Intended to aide debugging of options.
- filterdefs(text=None)¶
Add result of parsing text to the filter environment. Usually used to add functions to use in filters. If text is None, clear the environment.
Despite the power available here, we recommend you don’t get too cute about it. The main thing is for it to be clear what is happening.
Customization¶
The Andyroid Tutorial contains ideas on various sorts of customization. These include defining your own postprocessor, main program, and application-specific input language extensions.
Using Levels¶
To use levels, make a master.ats file with stick commands separating the tests, such as this example input:
stick(level=10)
test("test1.py")
test("test2.py")
stick(level=20)
test("test3.py")
test("test4.py")
t5 = test("test5.py")
stick(level=30)
test("test6.p7")
# this test sets a level explicitly, that overrides the "stick".
testif(t5, "test7.py", level=10)
The currently “stuck” value is set in every test that does not explicitly set level. Thus test3, for example, has level 20, as if the level=20 were given in the test statement.
Executing ats on this file with the option –level 30 will execute all these tests. Executing ats with –level 15 will execute only test1 and test2; test7 depends on test5, which has level 20, so it will not be run even though it has level 10.
The Test Class¶
When a test is created by the test or testif command, a test object representing
it is added to manager.testlist. This object is an instance of a class named
AtsTest
. Some users may wish to use the following details for debugging
or postprocessors or customization.
The class AtsTest
is available to users as ats.AtsTest
.
- AtsTest(*args, **options):
- stuck, glued, tacked
These are the current dictionaries for determining test options.
- test_number¶
The counter showing the number of tests defined so far.
- serialNumber¶
The unique serial number of this test.
- name¶
Set from an option to the test creation, or as the name of the script, or the name of the executable, plus the label. Eventually each test’s name is made unique.
- label¶
Set from an option to the test creation, incorporated in the name if given.
- options¶
The options for this test, after resolution using defaults, stuck, tacked, and glued.
- depends_on¶
If not None, the test instance this one depends upon.
- dependents¶
A list of any direct dependents of this test.
- exited¶
Has the job been run and exited?
- output¶
A list of lines of magic output, newlines and magic removed
- notes¶
List of notes from the run; user feel free to append to this list.
..attribute:: level
Test level set from resolved options. Same as
options.level
.- np¶
Number of processors required. Same as
options.np
.
- batchDic¶
A dictionary that may contain various things for a batch job.
- clas¶
A string containing the command line arguments after option interpolation.
- executable¶
An Executable object specifying the executable’s full path.
- directory¶
The full path to the directory in which the test is executed.
- groupNumber¶
The number of the group to which this test belongs, if positive.
- groupSerialNumber¶
The number of the test within its group definition.
- outname¶
The path to the standard output file for the test.
- errname¶
The path to the standard error file for the test.
- message¶
Explains the current value of
status
.
- runOrder¶
A number indicating the order in which the interactive tests were run.
- shortoutname¶
An abbreviated form of
outname
used for labeling.
- timelimit¶
An object of class Duration –
timelimit.value
is the limit in seconds. Duration objects can be compared to integer numbers of seconds correctly.
- waitUntil¶
A list of serial numbers of tests this one must wait for.
- nosrun¶
Boolean value, runs the code without srun when
True
. This can also be used to circumvent the login node check so that a test can be run on a login node. When used it will be up to the test to get an allocation if needed.
- set(status, message)¶
Set the object’s status and message.
- elapsedTime()¶
Returns a string, the formatted elapsed time of the run.
- stick, unstick, glue, unglue, etc.
Class methods stick, unstick, glue, unglue, etc. are equivalent to the ones accessible in the vocabulary or ats module.
There are other methods that are not intended for end users.
Test Statuses¶
There are eleven status values that a test can have. This value is stored in the
test’s attribute status
. Collectively this set of a statuses is in the
list ats.statuses
and each of them individually is in module ats
.
Each status has a four-character abbreviation, shown in parentheses. The status can also be
accessed under this name in the ats module. For example, PASS and PASSED are the same
object. You can correctly compare two statuses using “is” or “is not”, ==
or !=
,
or compare a status to a string representing its name or abbreviation, as in
PASSED == "PASS"
.
The statuses are:
- INVALID (INVD)
The test was not properly stated. For example, it referred to a script file that did not exist. See the log file for the error.
- CREATED (INIT)
The test was created but not (yet) run.
- PASSED (PASS)
The test was run and succeeded.
- FAILED (FAIL)
The test was run and failed.
- EXPECTED (EXPT)
The test ran and failed in an expected way.
- TIMEDOUT (TIME)
The test ran longer than its timelimit and was killed.
- SKIPPED (SKIP)
The test was created successfully but skipped for some reason. The reason is in the test object’s attribute
message
.- FILTERED (FILT)
The test was created successfully but filtered out for some reason. The reason is in the test object’s attribute
message
.- BATCHED (BACH)
The test was deemed eligible for batch processing, and has been shipped off to the batch system. ATS does not know its fate.
- RUNNING (EXEC)
The test is running, or was running when an error or keyboard interrupt occurred.
- HALTED (HALT)
The test was stopped after running successfully for one minute. This status is only possible if the
--cutoff
command-line option is used.
Postprocessing¶
After ATS has finished executing tests, but before it exits, it calls any Python routines that have been registered with it by calling:
manager.onExit(routine)
The routine should have the signature
def routine (manager):
...
The routine can do anything it wants. In particular, manager.testlist is available. Here’s an example of a trivial postprocessor in an input file:
def routine(manager):
passedTests = [test for test in manager.testlist \
if test.status is manager.PASSED]
print [test.name for test in passedTests]
manager.onExit(routine)
source ("set1.ats")
source ("set2.ats")
The postprocessing file is designed to make it possible to run postprocessing functions
of this kind using the state
variable as the manager
argument, rather than
doing it as an onExit
routine.
Test Suite Strategies¶
One of the problems with excessive choice is the paralzying effect of choice. There are a lot ways to do things with ATS. So here we describe a basic strategy to use until you have enough experience to form your own opinion.
We strongly urge that you read the Andyroid Tutorial as well.
This scheme assumes your code sources are destributed over a set of directories with a common parent called Home, with a subdirectory Test.
In each subdirectory with code that has a separate test (such as a unit test, or a test that emphasizes that coding) put a file with extension “ats”. This file contains a series of source statements that get further input or are test inputs containing introspective test statements).
- ::
test(clas = “-in myinput”, np = 1) source(“mysubdir/moretests.ats”)
Separate these inputs into levels with stick-level statements such as:
stick(level = 10)
...some tests...
stick(level=20)
...longer-running tests...
stick(level=30)
...still more...
You choose how many different levels you like. We recommend choosing well-spaced numbers in case you later change your mind and want to insert levels between the ones you start out with. Note that any test can still specify a level on its own that would override the stuck level.
As you go up your directory tree toward Home, put files that source the ones below it, until finally you have a tree leading to a file, say “testsuite.ats”, residing in your Home/Test directory.
Then you can make a series of small drivers. For example, your shortest test suite my be driven by this file:
glue("level <= 10")
source("testsuite.ats")
Running ats with this file as its input will result in only tests with level 10 or less being executed.
When the team that maintains a certain area wants to add a test, they add it to the closest member of the test-file tree relative to the source code they work with. They put it in the file at the appropriate level. This scheme leads to only rare source-code control conflicts, and ones that are usually a trivial merge; this avoids the conflicts generated by having a central test file.
Teams should be encourage to use introspection so that other members, less informed about how to test a certain area, can nevertheless exercise a good suite of tests using ATS, while allowing the experts to still use the input file directly with the code.
If there is one principle program being tested, it makes sense to use the -e option for it, and only explicitly specify an executable when it is different.
- ::
mycode = ‘/full/path/to/my/code’ test(executable=mycode, script=’foo.py’)
The extended example in Examples/Andyroid gives you many more ideas about how to use ATS.
Porting and Custom Machines¶
ATS decides on which machine characteristics to use by examining the value of
the environment variable MACHINE_TYPE; or, if it is not defined, the value
of the environment variable SYS_TYPE; or as default the value of Python’s
sys.platform
variable.
The reason for this three-level structure is to allow you to distinguish machine architectures when you have machines of the same basic type but with varied environments such as current OS level, parallel processing directives, or attached hardware. For an ordinary user on a personal computer, there is no reason to do anything special.
Most of the interaction between ATS and the platform takes place in
a machine module, defined by default in the sources in file Lib/machines.py
.
Different behaviors are obtained by inheriting from this module, or one
derived from it, and overriding various methods. We then connect our new
machine module to a value for MACHINE_TYPE with a comment in our module file,
and install that module in a directory in the Python distribution.
Porting ATS to a new platform is just one of the things you can do with the technique we describe in this section; you can also do things like doing something special when a job finishes, inventing your own scheduling algorithm, etc. You’ll need a decent knowledge of Python to do it, but you don’t need to be an expert.
If you invent a new value for MACHINE_TYPE, you can change the way ATS launches
and finishes jobs and keeps track of resources, amongst other things. You can
add command-line options and react to the user’s use of them. Your options will
even appear when the user executes with --help
.
To do this, you write a new Python source file, usually having a module name
equal to your value for MACHINE_TYPE. This file must define a new child of
machines.Machine
, and you must have a comment:
#ATS:name module class npMax
This line or lines defines the relationship between a MACHINE_TYPE and this module’s machine class and provides the maximum number of jobs you wish to execute at once (or it may mean the maximum number of processors one job can use in a parallel programming environment):
name is the name to match with MACHINE_TYPE.
module is the name of the module file, or SELF.
class is the name of the class in that module to use as a Machine.
npMax is a limit on np; if this number is negative it is a suggested default only.
machine.scheduler
is created by the standard__init__
method of the machine. If you want to create your own scheduler you can replace this attribute. See Customizing the Scheduler below.
The file Lib/machines.py
is well documented and it is usually not a large
problem to get things working.
Once you have your module file ready, you write a setup.py file to go with it:
from distutils.core import setup
myMachines =[myMachine.py] # list your machine module files
setup(name="myAtsAddon",
author = "you",
version = "1.0",
description = "All About My Machine",
data_files = [('atsMachines', myMachines)],
scripts = ['mycustomdriver'], #if you have one
)
and then execute python setup.py install
. Set the environment variable
MACHINE_TYPE and run ATS. It will report the machine module it has discovered.
In this setup.py
file, the unchangeable word is atsMachines
. This
is the name of a directory below your Python installation root where the
machine files are found by ATS. The scripts line can be omitted if you
do not want to install your own driver.
Installing Machines as Plugins¶
As an alternative to the data_files
approach you can register custom machines
with ATS using setup tools’ entry points plugin mechanism. This may be convenient
if you are building more customized ATS wrappers that are themselves packages, but
can also be used on standalone plugin packages. In this install method, ATS will look for
these in the group "ats.machines"
. This entry point name space is required for ATS
to find your machine plugin. The example below shows how to set this up with a
pyproject.toml
build system using setuptools:
...
[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"
...
[project.entry-points."ats.machines"]
custom_slurm_proc_sched = "mywrapper.atsMachines.myAtsSlurmProcessorScheduled:MyAtsSlurmProcessorScheduled"
This shows adding a custom_slurm_proc_sched
machine that’s defined in the parent
wrapper mywrapper
’s atsMachines submodule, where the MyAtsSlurmProcessorScheduled
class is defined in myAtsSlurmProcessorScheduled.py
. This makes the machine name
custom_slurm_proc_sched
available to ats to use to instatiate a new machine. With
this case you can either write whole new machines from scratch, or inherit from one
of the default machines to change it’s behavior. For more details see the setuptools
documentations, which also includes more how-to’s for setup.py
and setup.cfg
based
packages.
Note one major difference with this method currently: the default machine config is not
read from the #ATS:name module class npMax
comment. The name, module and class gets
read in from the plugin info, but the npMax
field is not set. It defaults to -1
in this case; the current convention is to use env vars to override it inside the machines,
so be sure and set those accordingly when configuring your custom machines.
Adding Test Options Via Machine¶
In a customized machine, the examineOptions
routine can add entries to
a dictionary, options.testDefaults
. These will be default option values for each
test. For example, here is how you would add an option nt
that could
be specified on the command line in the machine file:
def addOptions(parser):
parser.add_option('--nt', dest='nt', default=1, type='int',
help='Set default number of threads per test.')
def examineOptions(options):
options.testDefaults['nt'] = options['nt']
Of course, the machine would also have to examine and use properly the value of
each test’s option nt
; but it would always have one, and hence it could be
used in filters.
Customizing the Scheduler¶
The scheduler class StandardScheduler is defined in module schedulers
. It
handles issues such as priorities, and enforcing rules for the group()
and wait()
commands, and the independent
option.
Customizing the scheduler is possible but difficult. It should in particular
supply a method testlist() that returns the list of tests that are not yet
completed. Inheritance is strongly suggested, so that you only change what
you need to change. You’ll probably want to change the machine too so that
it creates the correct scheduler, but it feasible to create and assign a new
machine attribute scheduler
at any point up to and including the call to
machine.load
.
The important thing is to maintain correct separation between the scheduler
and the machine objects. The scheduler must ask the machine for such things
as canRunNow
that are within the purview of the machine, and ask it
about whether jobs have finished. The machine contains an attribute running
,
a list of the jobs currently running. The periodicReport
in the
scheduler does the basic report once a minute; a machine can call this and
then add more.
The ats Module¶
The ats
module can be imported in custom drivers and postprocessors.
Resources available in it are all imported from internal modules.
These are documented further in the Appendix.
- log, terminal
See the discussion of the log.
terminal
is simply a version oflog
that only writes to the terminal, not the log.
- times¶
Is a module containing useful time-handling routines
- configuration¶
Is the module that has information about the machine and command-line options.
- manager¶
Is the manager object. It has in particular
testlist
, and the routines discussed above. It is defined in themanagement
module.
- testEnvironment¶
Is the vocabulary dictionary.
- AtsTest¶
Is the test class.
- debug(value=None)¶
Is the debug function
- exception AtsError¶
Is the class of exceptions thrown by ATS.
- statuses, CREATED, INVALID, PASSED, FAILED, HALTED, SKIPPED,
- BATCHED, RUNNING, FILTERED, TIMEDOUT, SYS_TYPE, MACHINE_TYPE
Discussed previously, these are available via the ats module as attributes.
Using A Batch Facility¶
General Information¶
When running ATS, if a batch facility exists, both the interactive jobs and batch jobs will run. You have to use the facilities of that batch facility to find out what happened to those tests, because ATS will likely finish and exit long before those jobs are done.
Unfortunately, the world doesn’t have a standard batch facility. So here is an example of using the MSUB batch system at the Livermore Computing Center. Much of what follows would apply to any batch system.
To add a different batch system one must customize a batch machine to be installed in your ATS. For advice on how to do this, please contact us.
The basics are simple: if a test has a batch = 1
option, it is a batch test.
Each of the batch tests are individually submitted to the batch system.
The --allInteractive
flag is available to execute such tests without using
the batch system. Otherwise, they are simply skipped if no batch system is
found.
For the LC system in particular,
A testName.bat file is craated for the test.
The test information is written to a “batchContinue.log”. This file will be a concatenation of all the batch tests and will provide information about the tests.
Running Entirely In Batch¶
Submitting a lot of single batch jobs may overwhelm some batch systems. In that case it may be preferable to submit just one big batch job. One batch job is created to run all the tests (both batch and interactive).
The ATS option --allInteractive
is neccesary in the ATS command to prevent
the tests from being submitted seperately as batch.
An example of a batch script using MSUB at LC:
#!/bin/csh
#MSUB -N tmpAts0.157456004499.job
#MSUB -j oe
#MSUB -o tmpAts0.157456004499.job.out
#MSUB -q pbatch
#MSUB -l nodes=4:ppn=16
#MSUB -l ttc=64
#MSUB -l walltime=200
#MSUB -V # exports all environment var
#MSUB -A myBank # bank to use
setenv SYS_TYPE chaos_4_x86_64_ib
date
cd /my/work/directory/; atsb --allInteractive --numNodes=4 -useSrunStep Test/full.ats
date
The command-line options --numNodes=4 --useSrunStep
are not a part of
standard ATS. In this case, the ATS machine type chaos_4_x86_64_ib
has been
defined in a custom machine file, and custom machine files can add command-line options.
More Examples¶
Introspection¶
mytestA.py:
#ATS:test(SELF, batch=1, np=2, ...)
...mytestA problem...
mytestB.py:
#ATS:stick(batch=1)
#ATS:test(SELF, ...)
...mytestB problem...
myAts.ats:
tack(batch=1)
source('mytestC.py')
source('mytestD.py')
source('mytestE.py')
In myytestA.py
, a 2-processor batch job is created by introspection.
In mytestB.py
, the test created through introspection will be run in batch,
unless it happened to explicitly contain the option batch = 0, because the
stick
call makes batch = 1 the default in this file.
Running myAts.ats
, the tack
makes batch = 1 apply also in the three
files that get read. If this were a stick
, it wouldn’t apply inside those
other files.
Test Control¶
Suppose the file mytest.py contains a test script. The script throws an exception if it gets an error. It has a command line argument delta. Suppose mytest.py reads:
#ATS:log('mytest.py tests sanity of my group leader.')
#ATS:test(SELF, 'delta=0.5')
#ATS:test(SELF, 'delta=0.6', sanitycheck = 1)
#ATS:test(SELF, 'delta=0.7', np=4, sanitycheck = 1)
import physics
...command line processing to get delta's value...
...test problem....
...throws an exception if it fails...
If we run:
ats --exec myapplication mytest.py
then it is equivalent to running 3 tests:
myapplication mytest.py delta=0.5
myapplication mytest.py delta=0.6
myapplication mytest.py delta=0.7
The last one is run on 4 processors if the machine supports it.
Consider the command line:
ats --exec myapplication -f 'sanitycheck == 0' mytest.py
None of the tests are run; the first because sanitycheck is not one of its options, the other two because it is but the value is not zero. We could make sanitycheck have a default value of zero for all tests in mytest.py by adding this line to the top of mytest.py:
#ATS:stick(sanitycheck=0)
With this line added we would run only the first test.
Using the filter sanitycheck==1 would run the last two tests but skip the first. Using the filter ‘not np’ would run only the first two jobs, since they have by default np == 0.
Suppose mytest.ats reads:
source('mytestA.py')
source('mytestB.py')
and mytestA.py reads:
#ATS:stick(batch=1)
#ATS:test(SELF,delta=0.1)
...mytestA problem....
and mytestB.py reads:
#ATS:test(SELF)
...mytestB problem....
If we run:
ats -e myapplication --nobatch mytest.ats
then only myTestB.py is executed, and execution of mytestA.py is skipped,
since ats is not set for batch tests to run. Note --exec
can be abbreviated
as -e
.
If we run:
ats -e myapplication mytest.ats
then mytestA.py is submitted to batch and mytestB.py is run interactively. If there is no batch system, mytestA.py is skipped.
In practice a batch facility, if present, would add further options for controlling itself, such as options to set accounts or priorities or timelimits. The maintainers of such batch facilities will provide the documentation for them.
Finally,
ats --allInteractive -e myapplication mytest.ats
will test both myTestA and myTestB.
Resources For Learning ATS¶
The Examples directory in the distribution contains the sources that accompany the Andyroid Tutorial, including some sample customizations.
The Test directory contains more examples, although care must be taken in reading them as some of these are designed to fail.
At your particular location you may find other directories that define machines and batch systems for your local computer center.
Quick Recipes¶
To run only the batch tests:
ats --filter 'batch == 1' mytest.ats
To run only the interactive tests:
ats --nobatch mytest.ats
To run all tests as interactive tests:
ats --allInteractive mytest.ats
To check your input add –skip; add –debug for even more information.
To keep the output files even if the test succeeds, add –keep