Jip J. Dekker
fad1b07018
git-subtree-dir: software/minizinc git-subtree-split: 4f10c82056ffcb1041d7ffef29d77a7eef92cf76
161 lines
4.5 KiB
Markdown
161 lines
4.5 KiB
Markdown
MiniZinc Testing
|
|
================
|
|
|
|
## Setup
|
|
|
|
Requires Python 3 (on Windows 3.8 is required). Make sure you're in the `tests/` directory.
|
|
|
|
```sh
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
## Running Test Suite
|
|
|
|
To run the full test suite:
|
|
|
|
```sh
|
|
pytest
|
|
```
|
|
|
|
An HTML report will be generated at `output/report.html`.
|
|
|
|
Use the `--solvers` option to specify a subset of solvers to use in the tests. Tests which use a solver not in this list will be skipped.
|
|
```sh
|
|
pytest --solvers gecode,chuffed
|
|
```
|
|
|
|
Use the `--driver` option to specify the directory containing the minizinc executable. Otherwise, default directories and PATH will be used to locate it.
|
|
```sh
|
|
pytest --driver=../build
|
|
```
|
|
|
|
## Multiple test suites
|
|
|
|
To facilitate running the test suite with different minizinc options, `specs/suites.yml` contains configurations for running tests, or a subset of tests using different options.
|
|
|
|
```yaml
|
|
my-test-suite: !Suite
|
|
includes: ['*'] # Globs for included .mzn files
|
|
solvers: [gecode, chuffed] # The allowed solvers (if the test case itself specifies a different solver it will be skipped)
|
|
strict: false # Allow tests to pass if they check against another solver (default true)
|
|
options:
|
|
-O3: true # Default options to pass to minizinc (merged and overwritten by individual test cases)
|
|
```
|
|
|
|
For example, to run the `optimize-2` and `no-mip-domains` configurations only:
|
|
|
|
```sh
|
|
pytest --suite optimize-2 --suite no-mip-domains
|
|
```
|
|
|
|
## Creating/editing test cases with the web interface
|
|
|
|
A web interface can be used to create and edit test cases graphically. Python 3.8.1 or newer and Flask is required.
|
|
|
|
```sh
|
|
pip install flask
|
|
```
|
|
|
|
Start the web interface with
|
|
|
|
```sh
|
|
cd tests
|
|
python -m minizinc_testing.create
|
|
```
|
|
|
|
Then open `http://localhost:5000`.
|
|
|
|
The web interface detects `.mzn` files in the `tests/spec` directory. To add a new test, simply create a `.mzn` containing the model, and then open the file in the web interface. Test cases can then be generated accordingly.
|
|
|
|
## Creating/editing test cases manually
|
|
|
|
The test cases are defined using `YAML` inside the minizinc `.mzn` files. This YAML definition must be inside a block comment like the following:
|
|
|
|
```c
|
|
/***
|
|
!Test
|
|
expected: !Result
|
|
solution:
|
|
x: 1
|
|
***/
|
|
```
|
|
|
|
Multiple cases can be specified for one `.mzn` file:
|
|
|
|
```c
|
|
/***
|
|
--- !Test
|
|
...
|
|
--- !Test
|
|
...
|
|
--- !Test
|
|
...
|
|
***/
|
|
```
|
|
|
|
### YAML format
|
|
|
|
The format of the test case spec is as follows:
|
|
|
|
```yaml
|
|
!Test
|
|
solvers: [gecode, cbc, chuffed] # List of solvers to use (omit if all solvers should be tested)
|
|
check_against: [gecode, cbc, chuffed] # List of solvers used to check results (omit if no checking is needed)
|
|
extra_files: [datafile.dzn] # Data files to use if any
|
|
options: # Options passed to minizinc-python's solve(), usually all_solutions if present
|
|
all_solutions: true
|
|
timeout: !Duration 10s
|
|
expected: # The obtained result must match one of these
|
|
- !Result
|
|
status: SATISFIED # Result status
|
|
solution: !Solution
|
|
s: 1
|
|
t: !!set {1, 2, 3} # The set containing 1, 2 and 3
|
|
u: !Range 1..10 # The range 1 to 10 (inclusive)
|
|
v: [1, 2, 3] # The array with 1, 2, 3
|
|
x: !Unordered [3, 2, 1] # Ignore the order of elements in this array
|
|
_output_item: !Trim |
|
|
trimmed output item
|
|
gets leading/trailing
|
|
whitespace ignored
|
|
- !Error
|
|
type: MiniZincError # Name of the error type
|
|
message: Exact error message # Exact error message string (avoid using this as it's generally not portable)
|
|
regex: .*type-inst must be par set.* # Regex the start of the string must match (run with M and S flags)
|
|
```
|
|
|
|
For a test to pass, at least one expected result must be a subset of the obtained result. That is, the obtained result can have more attributes, but not less, and corresponding attributes must match.
|
|
|
|
If a solution is produced that does not match any given expected output, the result is checked using another solver. If this check passes then the test passes with a warning.
|
|
|
|
### Multiple solutions
|
|
|
|
When setting `all_solutions: true` and the order of the returned solutions often does not matter, use `!SolutionSet` for the list of solutions:
|
|
|
|
```yaml
|
|
!Result
|
|
status: ALL_SOLUTIONS
|
|
solution: !SolutionSet
|
|
- !Solution
|
|
x: 1
|
|
- !Solution
|
|
x: 2
|
|
```
|
|
|
|
### Testing FlatZinc output
|
|
|
|
Use `type: compile` on a test to enable only flattening.
|
|
Then `!FlatZinc filename.fzn` to give files with expected results.
|
|
|
|
```yaml
|
|
!Test
|
|
solvers: [gecode]
|
|
type: compile
|
|
expected: !FlatZinc expected.fzn
|
|
```
|
|
|
|
## TODO
|
|
|
|
- Tool for generating test cases
|
|
- Tweak YAML parsing so not so many !Tags are required
|
|
- Better documentation of how the framework operates |