Design Testing Framework

Overview

The Nautobot Design Builder testing framework streamlines and standardizes the process of testing within your designs by leveraging YAML files to create data objects and verify their consistency. This ensures no regressions in your design builds as both your designs and the Nautobot Design Builder evolve.

The Nautobot Design Builder testing framework provides:

• Data Creation: Tools to easily create and configure test data, such as devices, prefixes, and any object.
• Data Validation: Various "check" methods to verify that the data and application state meet the expected conditions.
• Modular Design: The ability to use inheritance between designs, via the depends_on capabilities.

Quick Start

To get started with the Nautobot Design Builder testing framework, create a simple YAML file to define a device and set up a test to verify its creation.

Create a file named ensure_device.yaml in your tests/testdata/ folder with the following content:

designs:
  - devices:
      - name: "Test Device"
        device_type: "WS-C9300-12X-S"
        device_role: "Test Role"
        site: "Test Site"
checks:
  - model_exists:
      model: dcim.device
      query: {name: "Test Device"}

In your test file, such as test_builder.py or unittest/test_builder.py, use the following code to verify the device creation:

from nautobot_design_builder.testing import BuilderTestCase

class TestGeneralDesigns(BuilderTestCase):
    """Designs that should work with all versions of Nautobot."""

    data_dir = os.path.join(os.path.dirname(__file__), "testdata")

This should be enough to get you started and potentially infer how the system works. Essentially, you create designs and then check that the results are as expected.

Note: While you can set the testdata folder to be anything, we will refer to it as tests/testdata/ throughout the documentation.

Unittest Setup

Most of the setup was provided in the Quick Start. For completeness, here is how you would set up your unittest. In your test file, such as test_builder.py or unittest/test_builder.py, use the following code to verify the device creation:

from nautobot_design_builder.testing import BuilderTestCase

class TestGeneralDesigns(BuilderTestCase):
    """Designs that should work with all versions of Nautobot."""

    data_dir = os.path.join(os.path.dirname(__file__), "testdata")

The data_dir folder can be any valid folder. The code provided essentially says "from the folder called testdata relative to my file". You can have multiple classes inherit from BuilderTestCase and leverage different folders, such as TestBranchDesigns(BuilderTestCase) and TestColoDesigns(BuilderTestCase) pointing to different folders.

Note: Unittest and CI setup is beyond the scope of this documentation.

Create Data

To create a design, you create a list of designs. While often there will be a single design, there is support for running through multiple designs.

Let's say you have the design called base_design.yaml.j2 that looks like:

manufacturers:
  - name: "cisco"
  - name: "arista"

device_types:
  - manufacturer__name: "cisco"
    model: "WS-C9300-12X-S"
    u_height: 1

That would become the file base_design.yaml in your tests/testdata folder:

designs:
  - manufacturers:
      - name: "cisco"
      - name: "arista"

    device_types:
      - manufacturer__name: "cisco"
        model: "WS-C9300-12X-S"
        u_height: 1

You would tier the design under an element in the designs: key. You can have multiple designs, such as:

designs:
  - manufacturers:
      - name: "cisco"
      - name: "arista"

    device_types:
      - manufacturer__name: "cisco"
        model: "WS-C9300-12X-S"
        u_height: 1

  - manufacturers: # Though simple, this is the second design
      - name: "palo alto"

This is how you declare multiple designs, as you re-initiate manufacturers in the second design.

Data Reuse

The depends_on feature allows you to create complex designs by building upon existing ones. This is useful when you have a base design that multiple other designs need to extend or modify. By using the depends_on key, you can ensure that the base design is always applied before the dependent design, maintaining consistency and reducing redundant code.

To use depends_on, reference the base design in the dependent design's YAML file.

Base Design: Create a file named base_design.yaml in your tests/testdata/ folder with the following content:

abstract: true
designs:
  - manufacturers:
      - name: "cisco"
      - name: "arista"

    device_types:
      - manufacturer__name: "cisco"
        model: "WS-C9300-12X-S"
        u_height: 1

Dependent Design: Create another file named branch_design.yaml in the same folder with the following content:

depends_on: base_design.yaml

designs:
  - devices:
      - name: "Test Device"
        device_type: "WS-C9300-12X-S"
        device_role: "Test Role"
        site: "Test Site"

In this example, branch_design.yaml depends on base_design.yaml. When branch_design.yaml is processed, it will first apply the configurations from base_design.yaml and then apply its own configurations. This ensures that the device type "WS-C9300-12X-S" defined in the base design is available when creating the "Test Device" in the extended design.

Note: If you don't need to run this design by itself, you can use the abstract key as shown in the example so that it doesn't run.

Validating Data with Checks

The "check" methods in the Nautobot Design Builder testing framework validate that the data and application state meet the expected conditions. These checks help ensure that your designs are correctly implemented and that no regressions occur as your designs or the Nautobot Design Builder evolve.

Currently, we support the following checks:

• connected
• equal
• model_exists
• model_not_exist
• in
• not_in

We will go into more detail on them in the next sections.

There are two main types of responses you can get from your check, a model or a value:

• Model: This response type indicates that the check is verifying the existence or properties of a specific model instance. For example, the model_exists check ensures that a particular instance of a model, such as a device, exists in the database.
• Value: This response type indicates that the check is verifying a specific value or set of values within a model's field. For example, the equal check ensures that a field of a model equals a specific value, such as verifying that a device's name is "Test Device".

Note: Model without using the count argument will return a list, so your value will often be a list of data.

By using these check systems, you can automate the validation process, reduce manual testing efforts, and maintain a high level of confidence in your design implementations.

Value Response Type

The value response type is straightforward. This is simply the data you define, with no data manipulation or query run to get the data. In general, this will be used in checks when you are comparing data from a model.

checks:
  - equal:
      - model: "nautobot.dcim.models.Device"
        query: {name: "test device"}
        attribute: "rack.name"
      - value: ["rack-1"]

This is saying that the model and attribute or rack.name will return exactly the value of ["rack-1"].

Model Response Type

The model response type involves using the model, query, attribute, and count keys to define the checks.

• model: Specifies the model to query. This should be a string representing the full dotted path to the model class, e.g., "nautobot.dcim.models.Device".
• query: (Optional) A dictionary that defines the conditions for the query, specifically a valid dictionary that would be accepted on a queryset to the Django model defined. This dictionary is used to filter the model instances based on the specified conditions and is valid, e.g., {"name": "test device"}.
• attribute: (Optional) Specifies the attribute of the model to check. This should be a string representing the attribute path, e.g., "rack.name". If not provided, the check returns the full model instances as a list.
• count: (Optional) This is a boolean that defaults to not being set. Use count: true and be careful of how YAML will interpret values; you would be wise to omit the key if not setting it to true.

As mentioned, the model will return a list in all cases except when using count. It will either be a list of attributes or a list of models.

Example with Attribute

In this example, the check verifies that the rack.name attribute of a device with the name "test device" is equal to ["rack-1"]:

checks:
  - equal:
      - model: "nautobot.dcim.models.Device"
        query: {name: "test device"}
        attribute: "rack.name"
      - value: ["rack-1"]

Example without Attribute

In this example, the check verifies that a device with the name "test device" exists:

checks:
  - model_exists:
      model: "nautobot.dcim.models.Device"
      query: {name: "test device"}

Example with Count

In this example, we are verifying that there are 2 model instances created.

checks:
  - equal:
      - model: "nautobot.extras.models.Secret"
        count: true
      - value: 2

Check Types

The checks generally compare two values, what is returned in the first item in the list compared to what is returned in the second item. However, you can also test if an object exists or does not exist with the model_exists and model_not_exists checks.

Note: In addition to checks, there are pre_checks, which have all of the same rules as checks, but they happen before the design is run. This is helpful if you want to ensure something does not exist before your design and does exist after your design, as one example.

The connected Check

The connected check verifies that two interface objects are connected with a cable object. For example, to check if two devices are connected:

checks:
  - connected:
      - model: "nautobot.dcim.models.Interface"
        query: {device__name: "Device 1", name: "GigabitEthernet1"}
      - model: "nautobot.dcim.models.Interface"
        query: {device__name: "Device 2", name: "GigabitEthernet1"}

This check type is a bit different from the other pattern, so pay close attention to how it is used.

The equal Check

The equal check verifies that a field of a model equals a specific value. For example, to check if a device's name is "Test Device":

In this example, we test that the model itself is the same.

checks:
  - equal:
      - model: "nautobot.dcim.models.DeviceType"
        query: {model: "model1"}
        attribute: "manufacturer"
      - model: "nautobot.dcim.models.Manufacturer"
        query: {name: "Manufacturer1"}

In this example, we test that the value is the same as returned, and we use the count value to get the number of elements.

checks:
  - equal:
      - model: "nautobot.extras.models.Secret"
        count: true
      - value: 2

The model_exists Check

The model_exists check verifies that an instance of a model exists. For example, to check if a prefix "10.0.0.0/24" exists:

checks:
  - model_exists:
      model: "nautobot.ipam.models.Prefix"
      query: {prefix: "10.0.0.0/24"}

This is often good enough to ensure your design is working, so keep that in mind as you build your tests to not over-complicate them.

Note: There is a single dictionary, not a list of dictionaries in model_exists.

The model_not_exist Check

The model_not_exist check verifies that an instance of a model does not exist. For example, to check if a device named "WS-C9300-12X-S" does not exist:

checks:
  - model_not_exist:
      model: "nautobot.dcim.models.DeviceType"
      query: {model: "WS-C9300-12X-S"}

Note: There is a single dictionary, not a list of dictionaries in model_not_exist.

The in Check

The in check verifies that a field of a model is in a specific list of values. For example, to check if the content types for Status are in the Content Types:

checks:
  - in:
      - model: "django.contrib.contenttypes.models.ContentType"
        query: {app_label: "dcim", model: "cable"}
      - model: "nautobot.extras.models.Status"
        query: {name: "Active"}
        attribute: "content_types"

The not_in Check

The not_in check verifies that a field of a model is not in a specific list of values. For example, to check if the content types for Status are not in the Content Types:

checks:
  - not_in:
      - model: "django.contrib.contenttypes.models.ContentType"
        query: {app_label: "dcim", model: "cable"}
      - model: "nautobot.extras.models.Status"
        query: {name: "Active"}
        attribute: "content_types"

Extensions

Extensions need to be explicitly defined as they are not automatically loaded. This allows you to customize and extend the functionality of the Nautobot Design Builder testing framework to fit your specific needs.

To define an extension, include it in the extensions key in your YAML file. Here is an example:

extensions:
  - "nautobot_design_builder.contrib.ext.NextPrefixExtension"
designs:
  - tenants:
      - name: "Nautobot Airports"
    roles:
      - name: "Video"
        content_types:
          - "!get:app_label": "ipam"
            "!get:model": "prefix"
      - name: "Servers"
        content_types:
          - "!get:app_label": "ipam"
            "!get:model": "prefix"
  - prefixes:
      - prefix: "10.0.0.0/23"
        status__name: "Active"
        tenant__name: "Nautobot Airports"
        role__name: "Servers"
      - prefix: "10.0.2.0/23"
        status__name: "Active"
        tenant__name: "Nautobot Airports"
        role__name: "Video"
      - "!next_prefix":
          prefix:
            - "10.0.0.0/23"
            - "10.0.2.0/23"
          length: 24
        status__name: "Active"

In this example, the NextPrefixExtension is explicitly defined in the extensions key. This extension will be loaded and used to process the !next_prefix directive in the designs section.

Skipping tests

Occasionally, you will need to skip a test, if for example, there is a known reason a test is failing but will be released anyway. You can add the skip key to the test file.

skip: true
designs:
  - tenants:
      - name: "Nautobot Airports"
checks:
  - model_exists:
      model: "nautobot.tenancy.models.Prefix"
      query: {name: "Nautobot Airports"}

Running Single Tests

Occasionally, you want to run a single test. In unittest, they have the concept of a label to run a single test. Here is an example of using the Python unittest with a label:

python -m nautobot.core.cli test nautobot_design_builder.tests.test_builder.TestGeneralDesigns --keepdb --buffer

This is generally easier to track, as the label is all real folders, files, and classes. However, the YAML file gets ingested, and the test name is auto-created. The name of the test will be the word test_ and the root of the name (e.g., without the .yaml extension on it), such as test_prefixes_for_location for the file prefixes_for_location.yaml. Here is an example of running a single test with a label:

python -m nautobot.core.cli test nautobot_design_builder.tests.test_builder.TestGeneralDesigns.test_prefixes_for_location --keepdb --buffer

This is helpful if you do not want to run all of your tests while testing a single issue or working on the root issue first.

Large Example

You can get many great examples from the test's folder, let's review a single larger example:

---
depends_on: "base_test.yaml"
designs:
  - roles:
      - "name": "EVPN Leaf"
        content_types:
          - "!get:app_label": "dcim"
            "!get:model": "device"
      - "name": "EVPN Spine"
        content_types:
          - "!get:app_label": "dcim"
            "!get:model": "device"

    devices:
      # Create Spine Switches
      - "!create_or_update:name": "spine1"
        "status__name": "Active"
        "location__name": "Site"
        "role__name": "EVPN Spine"
        "device_type__model": "model name"
        "interfaces":
          - "!create_or_update:name": "Ethernet9/3"
            "type": "100gbase-x-qsfp28"
            "status__name": "Active"
            "!ref": "spine1_to_leaf1"
          - "!create_or_update:name": "Ethernet25/3"
            "type": "100gbase-x-qsfp28"
            "status__name": "Active"
            "!ref": "spine1_to_leaf2"
      - "!create_or_update:name": "spine2"
        "status__name": "Active"
        "location__name": "Site"
        "role__name": "EVPN Spine"
        "device_type__model": "model name"
        "interfaces":
          - "!create_or_update:name": "Ethernet9/3"
            "type": "100gbase-x-qsfp28"
            "status__name": "Active"
            "!ref": "spine2_to_leaf1"
          - "!create_or_update:name": "Ethernet25/3"
            "type": "100gbase-x-qsfp28"
            "status__name": "Active"
            "!ref": "spine2_to_leaf2"
      - "!create_or_update:name": "spine3"
        "status__name": "Active"
        "location__name": "Site"
        "role__name": "EVPN Spine"
        "device_type__model": "model name"
        "interfaces":
          - "!create_or_update:name": "Ethernet9/3"
            "type": "100gbase-x-qsfp28"
            "status__name": "Active"
            "!ref": "spine3_to_leaf1"
          - "!create_or_update:name": "Ethernet25/3"
            "type": "100gbase-x-qsfp28"
            "status__name": "Active"
            "!ref": "spine3_to_leaf2"
      - "!create_or_update:name": "leaf1"
        "status__name": "Active"
        "location__name": "Site"
        "role__name": "EVPN Leaf"
        "device_type__model": "model name"
        "interfaces":
          - "!create_or_update:name": "Ethernet33/1"
            "type": "100gbase-x-qsfp28"
            "!ref": "leaf1_to_spine1"
            "status__name": "Active"
          - "!create_or_update:name": "Ethernet34/1"
            "type": "100gbase-x-qsfp28"
            "!ref": "leaf1_to_spine2"
            "status__name": "Active"
          - "!create_or_update:name": "Ethernet35/1"
            "type": "100gbase-x-qsfp28"
            "!ref": "leaf1_to_spine3"
            "status__name": "Active"
      - "!create_or_update:name": "leaf2"
        "status__name": "Active"
        "location__name": "Site"
        "role__name": "EVPN Leaf"
        "device_type__model": "model name"
        "interfaces":
          - "!create_or_update:name": "Ethernet33/1"
            "type": "100gbase-x-qsfp28"
            "!ref": "leaf2_to_spine1"
            "status__name": "Active"
          - "!create_or_update:name": "Ethernet34/1"
            "type": "100gbase-x-qsfp28"
            "!ref": "leaf2_to_spine2"
            "status__name": "Active"
          - "!create_or_update:name": "Ethernet35/1"
            "type": "100gbase-x-qsfp28"
            "!ref": "leaf2_to_spine3"
            "status__name": "Active"

    cables:
      - "!create_or_update:termination_a_id": "!ref:spine1_to_leaf1.id"
        "!create_or_update:termination_b_id": "!ref:leaf1_to_spine1.id"
        "termination_a": "!ref:spine1_to_leaf1"
        "termination_b": "!ref:leaf1_to_spine1"
        "status__name": "Planned"
      - "!create_or_update:termination_a_id": "!ref:spine2_to_leaf1.id"
        "!create_or_update:termination_b_id": "!ref:leaf1_to_spine2.id"
        "termination_a": "!ref:spine2_to_leaf1"
        "termination_b": "!ref:leaf1_to_spine2"
        "status__name": "Planned"
      - "!create_or_update:termination_a_id": "!ref:spine3_to_leaf1.id"
        "!create_or_update:termination_b_id": "!ref:leaf1_to_spine3.id"
        "termination_a": "!ref:spine3_to_leaf1"
        "termination_b": "!ref:leaf1_to_spine3"
        "status__name": "Planned"
      - "!create_or_update:termination_a_id": "!ref:spine1_to_leaf2.id"
        "!create_or_update:termination_b_id": "!ref:leaf2_to_spine1.id"
        "termination_a": "!ref:spine1_to_leaf2"
        "termination_b": "!ref:leaf2_to_spine1"
        "status__name": "Planned"
      - "!create_or_update:termination_a_id": "!ref:spine2_to_leaf2.id"
        "!create_or_update:termination_b_id": "!ref:leaf2_to_spine2.id"
        "termination_a": "!ref:spine2_to_leaf2"
        "termination_b": "!ref:leaf2_to_spine2"
        "status__name": "Planned"
      - "!create_or_update:termination_a_id": "!ref:spine3_to_leaf2.id"
        "!create_or_update:termination_b_id": "!ref:leaf2_to_spine3.id"
        "termination_a": "!ref:spine3_to_leaf2"
        "termination_b": "!ref:leaf2_to_spine3"
        "status__name": "Planned"
checks:
  - model_exists:
      - model: "nautobot.dcim.models.Device"
        query: {name: "spine1"}
  # Spine 1 to Leaf 1
  - connected:
      - model: "nautobot.dcim.models.Interface"
        query: {device__name: "spine1", name: "Ethernet9/3"}
      - model: "nautobot.dcim.models.Interface"
        query: {device__name: "leaf1", name: "Ethernet33/1"}

  # Spine 1 to Leaf 2
  - connected:
      - model: "nautobot.dcim.models.Interface"
        query: {device__name: "spine1", name: "Ethernet25/3"}
      - model: "nautobot.dcim.models.Interface"
        query: {device__name: "leaf2", name: "Ethernet33/1"}

  # Spine 2 to Leaf 1
  - connected:
      - model: "nautobot.dcim.models.Interface"
        query: {device__name: "spine2", name: "Ethernet9/3"}
      - model: "nautobot.dcim.models.Interface"
        query: {device__name: "leaf1", name: "Ethernet34/1"}

  # Spine 2 to Leaf 2
  - connected:
      - model: "nautobot.dcim.models.Interface"
        query: {device__name: "spine2", name: "Ethernet25/3"}
      - model: "nautobot.dcim.models.Interface"
        query: {device__name: "leaf2", name: "Ethernet34/1"}

  # Spine 3 to Leaf 1
  - connected:
      - model: "nautobot.dcim.models.Interface"
        query: {device__name: "spine3", name: "Ethernet9/3"}
      - model: "nautobot.dcim.models.Interface"
        query: {device__name: "leaf1", name: "Ethernet35/1"}

  # Spine 3 to Leaf 2
  - connected:
      - model: "nautobot.dcim.models.Interface"
        query: {device__name: "spine3", name: "Ethernet25/3"}
      - model: "nautobot.dcim.models.Interface"
        query: {device__name: "leaf2", name: "Ethernet35/1"}