Testing Instrument Functionality

Overview

When developing new instrument classes, or adding functionality to existing instruments, it is important to also add automated checks for the correctness of the new functionality. Such tests serve two distinct purposes:

• Ensures that the protocol for each instrument is being followed correctly, even with changes in the underlying InstrumentKit behavior.
• Ensures that the API seen by external users is kept stable and consistent.

The former is especially important for instrument control, as the developers of InstrumentKit will not, in general, have access to each instrument that is supported— we rely on automated testing to ensure that future changes do not cause invalid or undesired operation.

For InstrumentKit, we rely heavily on pytest, a mature and flexible unit-testing framework for Python. When run from the command line via pytest, or when run by Travis CI, pytest will automatically execute functions and methods whose names start with test in packages, modules and classes whose names start with test or Test, depending. (Please see the pytest documentation for full details, as this is not intended to be a guide to pytest so much as a guide to how we use it in IK.) Because of this, we keep all test cases in the instruments.tests package, under a subpackage named for the particular manufacturer, such as instruments.tests.test_srs. The tests for each instrument should be contained within its own file. Please see current tests as an example. If the number of tests for a given instrument is numerous, please consider making modules within a manufacturer test subpackage for each particular device.

Below, we discuss two distinct kinds of unit tests: those that check that InstrumentKit functionality such as Property Factories work correctly for new instruments, and those that check that existing instruments produce correct protocols.

Mock Instruments

TODO

Expected Protocols

As an example of asserting correctness of implemented protocols, let’s consider a simple test case for instruments.srs.SRSDG645`:

def test_srsdg645_output_level():
    """
    SRSDG645: Checks getting/setting unitful ouput level.
    """
    with expected_protocol(ik.srs.SRSDG645,
            [
                "LAMP? 1",
                "LAMP 1,4.0",
            ], [
                "3.2"
            ],
            sep="\n"
    ) as ddg:
        unit_eq(ddg.output['AB'].level_amplitude, pq.Quantity(3.2, "V"))
        ddg.output['AB'].level_amplitude = 4.0

Here, we see that the test has a name beginning with test_, has a simple docstring that will be printed in reports on failing tests, and then has a call to expected_protocol(). The latter consists of specifying an instrument class, here given as ik.srs.DG645, then a list of expected outputs and playback to check property accessors.

Note that expected_protocol() acts as a context manager, such that it will, at the end of the indented block, assert the correct operation of the contents of that block. In this example, the second argument to expected_protocol() specifies that the instrument class should have sent out two strings, "LAMP? 1" and LAMP 1,4.0, during the block, and should act correctly when given an answer of "3.2" back from the instrument. The third parameter, sep specifies what will be appended to the end of each lines in the previous parameters. This lets you specify the termination character that will be used in the communication without having to write it out each and every time.

Protocol Assertion Functions

instruments.tests.expected_protocol(ins_class, host_to_ins, ins_to_host, sep='\n', repeat=1)

Given an instrument class, expected output from the host and expected input from the instrument, asserts that the protocol in a context block proceeds according to that expectation.

For an example of how to write tests using this context manager, see the make_name_test function below.

Parameters:

• ins_class (Instrument) – Instrument class to use for the protocol assertion.
• host_to_ins (str or list; if list, each line is concatenated with the separator given by sep.) – Data to be sent by the host to the instrument; this is checked against the actual data sent by the instrument class during the execution of this context manager.
• ins_to_host (str or list; if list, each line is concatenated with the separator given by sep.) – Data to be sent by the instrument; this is played back during the execution of this context manager, and should be used to assert correct behaviour within the context.
• sep (str) – Character to be inserted after each string in both host_to_ins and ins_to_host parameters. This is typically the termination character you would like to have inserted.
• repeat (int) – The number of times the host_to_ins and ins_to_host data sets should be duplicated. Typically the default value of 1 is sufficient, but increasing this is useful when testing multiple calls in the same test that should have the same command transactions.