ucs-test

From Univention Wiki

Revision as of 09:58, 30 January 2015 by Sgohmann (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search


ucs-test is used to check UCS installations for correct working and configuration.

Structure

Sections

The tests are categorized in sections, which are represented as sub-directories like /usr/share/ucs-test/00_section/. Every directory must begin with a two digit number followed by a name, separated by an underscore character. Every section should be packaged in a separate (binary) Debian package to allow picking only relevant tests.

Single tests

For every test a single executable file exists, which implements the test.

  • The name of the file must begin with a two or three digit number, followed by a short description.
  • No file extension (.py, .sh, ...) must be used (as this gets interpreted as a separator by Jenkins).
  • The file should be marked as executable (svn ps svn:executable 1 ...).
  • In the hash-bang-line #!/usr/share/ucs-test/runner should be used instead of using /bin/bash or /usr/bin/python directly.

Is interpreter asserts several things:

  • The current working directory is changed to the directory containing the script.
  • Meta data information from the test case file are used to check the required system role, check for required packages and versions. The test is only run if those pre-conditions are met.
  • The (integer) return value of the test script is converted to a human readable message.

Framework

Next to the test files some shell and python libraries are provided below /usr/share/ucs-test/lib/, which can be re-used by several tests.

/usr/sbin/ucs-test is a program to run several tests at once. It collects their output and creates some summary information.

Packages

ucs-test-framework
This package contains the minimum required files like runner and ucs-test.
ucs-test-libs
This package contains the optional shared libraries used by several other packages.
ucs-test-*
Each package contains the tests of the different sections.

Libraries

The ucs-test-libs package provides several libraries for common tasks, such as creating and modifying UDM objects, handling UCR variables, or common functions to simplify writing robust tests. Shell libraries should be included as

. "$TESTLIBPATH/name.sh" || exit 137

while Python libraries should be imported as regular modules like

import univention.testing.name

Currently the following libraries are shipped:

base.sh
Output helpers, software checks, version checks, test helpers
computer.sh
Simplify handling UDM computers
container.sh
Simplify handling UDM container
group.sh
Simplify handling UDM groups
ldap.sh
Directly interact with LDAP
master.sh
Execute commands remotely on domaincntroller_master
random.sh
Provide function to generate random names for different requirements
shares.sh
Simplify handling UDM shares
ucr.sh
Wrapper around UCR to restore values on exit
user.sh
Simplify handling UDM users

Working with LDAP objects

To allow the creation of LDAP objects on different system roles than DC Master and DC Backup, ucs-test sets during the installation the following UCR variables:

tests/domainadmin/account
Domain administrator LDAP bind DN to be used in test cases
tests/domainadmin/pwd
Domain administrator password to be used in test cases
tests/domainadmin/pwdfile
File to read the domain administrator password

By default these variables are set to the the DN of Administrator and to the password univention.

To use these variables in a UDM call, udm-test can be used, for example:

#!/usr/share/ucs-test/runner bash
## desc: Create a container
## roles: domaincontroller_slave

. "$TESTLIBPATH/base.sh" || exit 137
. "$TESTLIBPATH/random.sh" || exit 137

container_name="$(random_chars 20)"
udm-test container/cn create --set name="$container_name"

wait_for_replication

univention-ldapsearch "cn=$container_name"

The function wait_for_replication checks if the replication is finished. To wait also for the service restart, for example if a new printer was added, the function wait_for_replication_and_postrun can be used.

Usage

Tests can be run in two ways. Either directly using their fully qualified name (/usr/share/ucs-test/00_section/00_test), or using ucs-test to run several tests together. ucs-test supports several options to select a subset of all installed tests, which are described in its manual page (man ucs-test).

Creating new tests

Minimal test

In its simplest form a test contains only some commands, which return 0 on success. Any other return value is interpreted as a failure.

#!/usr/share/ucs-test/runner bash
## desc: Test if /etc/shadow exists
## exposure: safe
[ -f '/etc/shadow' ]
# vim: set filetype=sh tabstop=4 :
  • In the 1st line /usr/share/ucs-test/runner is used instead of /bin/bash. This wrapper prints some more information in addition to the return value of the tests. The language used for the implementation of the test is passed as the sole parameter to this wrapper. Next to bash, python and python2.6 any other interpreter might be used as well as long as the fully qualified path to the interpreter is given.
  • In the 2nd line a description is given using the meta data syntax. This description is included in the output next to the return value. The meta data block must be located at the beginning of the file right after the hash-bang line without any intermediate lines. All lines must pre prefix by ## .
  • The 3rd line declares that the test is safe to be run on any system, even systems already in production.
  • The 4th line implements the tests.

Meta data

The meta data information consists of consecutively lines, which directly follow the hash-bang line. The block ends at the first line not prefixed by ## . Without that prefix the format follows YAML. This allows other programs to parse that information as well and also allows custom data (severity, author, ...) to be added. ucs-test uses the following entries:

desc

A text used as a description for this test, which is displayed together with the test result when the test is run. The text can span multiple lines, which allows adding detailed information. How much of the description is displayed depends on the chosen output format.

## desc: A short text
## desc: |
##  A shot text for displaying.
##  A longer text, which describes the test in more detail.

bugs

A list if bug numbers from the Univention Bugzilla. Depending on the chosen output format (for example html), reports will include links to the referenced bugs.

## bugs: [23527]
## bugs:
##  - 23527

otrs

A list of ticket numbers from the Univention OTRS. Depending on the chosen output format (for example html), reports will include links to the referenced tickets.

## otrs: [2012031912061xxx]

versions

A mapping from UCS versions number (major.minor-patchlevel) to the strings found, fixed und skip. This allows to differentiate the return values and to distinguish between tests, which are for known not-yet-fixed bugs, and tests, which should be fixed but fail again.

found
the bug is relevant starting from that version.
fixed
the bug was fixed in this version and should not trigger.
skip
the test should be skipped since that version.
## versions:
##  2.0-0: found
##  2.1-0: fixed
##  3.0-0: skip

tags

A list of tags, which can be used to group tests. This can be used to either run only a specific subset of all installed tests or prevent some tagged tests from running. The name for tags can be chosen freely, but the following words have special meaning:

SKIP, WIP
The test will be skipped by default, because — for example — the implementation is still work in progress.
basic
Marks the tests as a basic test, which should be run on all systems to assert a sound installation.
## tags: [WIP]
## tags:
##  - my-tag

roles

A list of UCS role names. The test is only run on systems, which match the role name explicitly listed here. The default is to allow the test to run on all roles.

## roles:
##  - domaincontroller_master
##  - domaincontroller_backup
##  - domaincontroller_slave
##  - memberserver
##  - basesystem

roles-not

Inverse list for UCS role names. The test is skipped on systems, which match the role name listed here. Overlaps with roles will abort the test.

## roles-not: [basesystem]

join

Requires or prohibits the system to be joined. Otherwise the test is not run. By default the join status is irrelevant and the test is run on all systems.

## join: true

components

A mapping of component names to a boolean value, which requires the named component to be either installed or to be deactivated. By default no components are required and the test is run on all systems.

## components:
##  tcs: true
##  dvs: false

packages

A list of Debian package dependencies, which must be fulfilled. Otherwise the test is not run on such a system. Each entry follows the syntax used by Debian dependencies.

## packages:
##  - dpkg-dev (>= 1.15)
##  - apache2 | apache2-mpm-prefork

Normally dependencies should be specified using the Debian debian/control file. But in some cases tests need special software, which is not wanted on normal systems. This mechanism allows the user to still install the full ucs-test-* section package without forcing him to install several other packages only relevant for corner case tests. By using this mechanism of ucs-test, tests still missing some dependent packages are skipped.

exposure

A string consisting of one of the words safe, careful or dangerous, optionally followed by the md5sum over the file without the meta data block separated by a blank. This is used to classify tests in different categories:

safe
tests of read data and never modify anything on the system. These tests can be run on production systems without danger of modifying that system or losing services.
careful
test do modify the system (create files, restart services, change configurations), but revert all changes back to the original state at the end of the test, regardless of if the test succeeds or fails.
dangerous
the test has side effects, which might change this and other systems in a way unfit for production systems. This includes — for example — creating users and groups in LDAP, re-configuring essential services like LDAP.

The default is dangerous.

As an extra precaution against unwanted modification an optional MD5 checksum can be supplied. The test is skipped in case the specified checksum does not match the actual checksum.

## exposure: careful
## exposure: safe 25f9fc4bf6e01167a0cc6c84309572b7

The MD5 checksum can be computed for example by running

sed -e '/^## /d' "/usr/share/ucs-test/tests/00_section/00_test" | md5sum

Return value

In its simplest case a test should return 0 for success and 1 on failure. For backward compatibility to previous versions of ucs-test several other special values can be returned as well to provide more detailed information. More information is available in the source code or internally from UCS-Test until UCS-2.4.

Quality

ucs-test should be runnable on all systems, ranging from development machines to systems in production. Because of this requirement test developers should be extremely careful before modifying system configuration or data. To protect production systems from accidental destruction by running tests, ucs-test requires tests to follow the following criteria:

  • a test should just test one thing, so a failed tests has a clear conclusion.
  • a test should return stable results, so running the same test multiple times returns the same result each time.
  • a test should not modify the system. If anything is changed, it must be changed back regardless if the test succeeds or fails.

To enforce this ucs-test by default refrains from running tests with an #exposure level other then safe.

Personal tools