Getting Started with Designs¶
If you've ever watched an episode of "How it's made", you probably already have a good idea how Design Builder works. Take the analogy of building a car assembly line in a plant, to support building many cars. There are various car parts made by a press, such as the hood, doors, bumper, etc. For a given car, you can only have a 40 inch hood but not the 50 inch hood that would go on the truck. The assembly line is created so that many of the things needed for the car are fixed, such as a specific kind of steering wheel, but the worker building the car can substitute variable components requested by the customer. This could include the car's color, sound system, and choice of sun roof. This ensures that the worker can only assemble the car with valid components and the outcome is guaranteed to be a proper car.
In the same way, the Design Builder app works to combine standardized design templates (the assembly line in this analogy) with a design context (the user supplied and computed information) to produce a design that can be used to create or update data within Nautobot, essentially creating the ability to expand the data massively from a few simple inputs. The combination of template, context and the job to build the objects are all collectively referred to as a design.
A special form of Nautobot Job, known as a Design Job, is the main entrypoint into a design's implementation. The design job is responsible for defining the inputs required from a user and any input validation that must take place prior to a job's implementation. Once a user has provided the necessary inputs, a design context is created from those inputs. The design context provides any variables and values needed in the design templates for rendering. The design context can provide computed values, perform database lookups within Nautobot, or perform provisioning tasks. Once the design context has been created, the design builder will render the design template using the design context as a Jinja render context.
The general flow of data through the design builder system can be visualized with the following picture:
For the remainder of this tutorial we will focus solely on the Design Job, Design Context and Design Template portions of that diagram.
Design Components¶
Designs can be loaded either from local files or from a git repository. Either way, the structure of the actual designs and all the associated files is the same. Since, fundamentally, all designs are Nautobot Jobs, everything must be in a top level jobs python package (meaning the directory must contain the file __init__.py) and all design classes must be either defined in this jobs module or be imported to it. The following directory layout is from the demo designs repository:
jobs
├── __init__.py
├── core_site
│ ├── __init__.py
│ ├── context
│ │ ├── __init__.py
│ │ └── context.yaml
│ └── designs
│ └── 0001_design.yaml.j2
├── edge_site
│ ├── __init__.py
│ ├── context
│ │ ├── __init__.py
│ │ └── context.yaml
│ └── designs
│ └── 0001_design.yaml.j2
└── initial_data
├── __init__.py
├── context
│ ├── __init__.py
└── designs
└── 0001_design.yaml.j2
The jobs directory contains everything that is needed for implementing a design. In the above case, the directory contains three designs:
- Initial Design
- Core Site
- Edge Site
Within the jobs directory, the naming of modules and files is not important. However, it is recommended to use intuitive names to help understand each file's relationship with others. For instance, above there is are design contexts specified as both Python modules as well as YAML files and each design has exactly one design template. The relationship of context YAML files and context Python modules will be discussed later.
Designs are just specialized Nautobot jobs. Any design must inherit from DesignJob, and just like any other job, design jobs must be registered using register_jobs. An example design follows:
from nautobot.apps.jobs import register_jobs
from nautobot_design_builder.design_job import DesignJob
class BasicDesign(DesignJob):
"""A basic design for design builder."""
class Meta:
"""Metadata describing this design job."""
name = "Basic Design"
commit_default = False
design_file = "designs/0001_design.yaml.j2"
register_jobs(BasicDesign)
DesignJob¶
Primary Purpose:
- Define the Job
- Provide the user inputs
- Define the Design Context and Design Templates
As previously stated, the entry point for all designs is the DesignJob class. New designs should include this class in their ancestry. Design Jobs are an extension of Nautobot Jobs with several additional metadata attributes. Here is the initial data job from our sample design:
"""Initial data required for core sites."""
from nautobot.apps.jobs import register_jobs, IntegerVar
from nautobot_design_builder.design_job import DesignJob
from .context import InitialDesignContext
class InitialDesign(DesignJob):
"""Initialize the database with default values needed by the core site designs."""
has_sensitive_variables = False
has_sensitive_variables = False
routers_per_site = IntegerVar(min_value=1, max_value=6, default=2)
class Meta:
"""Metadata needed to implement the backbone site design."""
name = "Initial Data"
commit_default = False
design_file = "designs/0001_design.yaml.j2"
context_class = InitialDesignContext
version = "1.0.0"
description = "Establish the devices and site information for four sites: IAD5, LGA1, LAX11, SEA11."
docs = """This design creates the following objects in the source of truth to establish the initia network environment in four sites: IAD5, LGA1, LAX11, SEA11.
These sites belong to the America region (and different subregions), and use Juniper PTX10016 devices.
The user input data is:
- Number of routers per site (integer)
- The description for us-west-1 region (string)
"""
name = "Demo Designs"
register_jobs(InitialDesign)
This particular design job does not collect any input from the user, it will use InitialDesignContext for its render context and it will consume the templates/initial_design.yaml.j2 file for its design. When this job is run, the Design Builder will create an instance of InitialDesignContext, read templates/initial_design.yaml.j2 and then render the template with Jinja using the design context as a render context.
Here is another, more interesting design:
"""Design to create a core backbone site."""
from nautobot.apps.jobs import register_jobs, ObjectVar, StringVar, IPNetworkVar
from nautobot.dcim.models import Location
from nautobot_design_builder.design_job import DesignJob
from .context import CoreSiteContext
class CoreSiteDesign(DesignJob):
"""Create a core backbone site."""
region = ObjectVar(
label="Region",
description="Region for the new backbone site",
model=Location,
)
site_name = StringVar(regex=r"\w{3}\d+")
site_prefix = IPNetworkVar(min_prefix_length=16, max_prefix_length=22)
has_sensitive_variables = False
class Meta:
"""Metadata needed to implement the backbone site design."""
name = "Backbone Site Design"
commit_default = False
design_file = "designs/0001_design.yaml.j2"
context_class = CoreSiteContext
nautobot_version = ">=2"
name = "Demo Designs"
register_jobs(CoreSiteDesign)
In this case, we have a design that will create a site, populate it with two racks, each rack will have a core router and each router will be populated with routing engines and switch fabric cards. The design job specifies that the user needs to supply a location for the new site, a site name and an IP prefix. These inputs will be combined in the design context to be used for building out a new site.
Class Metadata Attributes¶
The design jobs above include standard metadata (name and commit_default for instance) attributes needed by Nautobot. Additionally, there are attributes specific to the design job. The design_file is a required require attribute specifying the design template. An optional context_class attribute can be given to specify the design context Python class. A more detailed description of these attributes follows in the next section.
design_file¶
Design file specifies the Jinja template that should be used to produce the input for the design builder. The builder will resolve the file's location relative to the location of the design job class.
design_files¶
Design files specifies a list of Jinja template that should be used to produce the input for the design builder. The builder will resolve the files' locations relative to the location of the design job class. Exactly one of design_file or design_files must be present in the design's Metadata. If design_files is used for a list of design templates, each one is evaluated in order. The same context and builder are used for all files. Since a single builder instance is used, references can be created in one design file and then accessed in a later design file.
context_class¶
The value of the context_class metadata attribute should be any Python class that inherits from the nautobot_design_builder.Context base class. Design builder will create an instance of this class and use it for the Jinja rendering environment in the first stage of implementation.
report¶
This attribute is optional. A report is a Jinja template that is rendered once the design has been implemented. Like design_file the design builder will look for this template relative to the filename that defines the design job. This is helpful to generate a custom view of the data that was built during the design build.
version¶
It's an optional string attribute that is used to define the versioning reference of a design job. This will enable in the future the versioning lifecycle of design deployments. For example, one a design evolves from one version to another, the design deployment will be able to accommodate the new changes.
description¶
This optional attribute that is a string that provides a high-level overview of the intend of the design job. This description is displayed int the design detail view.
docs¶
This attribute is also displayed on the design detail view. The docs attribute can utilize markdown format and should provide more detailed information than the description. This should help the users of the Design to understand the goal of the design and the impact of the input data.
Design Context¶
Primary Purpose:
- Organize data from multiple places
- Validate data
As previously stated, the design context is a combination of user supplied input and computed values. The design context should include any details needed to produce a design that can be built. Fundamentally, the design context is a Python class that extends the nautobot_design_builder.Context class. However, this context can be supplemented with YAML. Once Design Builder has created and populated the design context it passes this context off to a Jinja rendering environment to be used for variable lookups.
That's a lot to digest, so let's break it down to the net effect of the design context.
A context is essentially a mapping (similar to a dictionary) where the context's instance properties can be retrieved using the index operator ([]). YAML files that are included in the context will have their values added to the context as instance attributes. When design builder is rendering the design template it will use the context to resolve any unknown variables. One feature of the design context is that values in YAML contexts can include Jinja templates. For instance, consider the core site context from the design above:
from nautobot.dcim.models import Location
from netaddr import IPNetwork
from nautobot_design_builder.errors import DesignValidationError
from nautobot_design_builder.context import Context, context_file
@context_file("context.yaml")
class CoreSiteContext(Context):
"""Render context for core site design"""
region: Location
site_name: str
site_prefix: IPNetwork
def validate_new_site(self):
try:
Location.objects.get(name__iexact=str(self.site_name))
raise DesignValidationError(f"Another site exist with the name {self.site_name}")
except Location.DoesNotExist:
return
def get_serial_number(self, device_name):
# ideally this would be an API call, or some external
# process, to determine the serial number. This is just to
# demonstrate var lookup from the context object
return str(abs(hash(device_name)))
This context has instance variables region, site_name and site_prefix. These instance variables will be populated from the user input provided by the design job. Additionally note the class decorator @context_file. This decorator indicates that the core_site_context.yaml file should be used to also populate values of the design context. The context includes a method called validate_new_site to perform some pre-implementation validation (see the next section for details). The context also includes a method called get_serial_number. The implementation of this method is there only to demonstrate that some dynamic processing can occur to retrieve context values. For example, there may be an external CMDB that contains serial numbers for the devices. The get_serial_number method could connect to that system and lookup the serial number to populate the Nautobot object.
Now let's inspect the context YAML file:
---
core_1_loopback: "{{ site_prefix | network_offset('0.0.0.1/32') }}"
core_2_loopback: "{{ site_prefix | network_offset('0.0.1.1/32') }}"
This context YAML creates two variables that will be added to the design context: core_1_loopback and core_2_loopback. The values of both of these variables are computed using a jinja template. The template uses a jinja filter from the netutils project to compute the address using the user-supplied site_prefix. When the design context is created, the variables will be added to the context. The values (from the jinja template) are rendered when the variables are looked up during the design template rendering process.
Note: The
Contextclass also contains a property to retrieve theTagassociated with the design and attached to all the objects with full_control. With this tag you can check for data in objects already created when the design is updated, for example:.filter(tags__in=[self.design_instance_tag].
Context Validations¶
Sometimes design data needs to be validated before a design can be built. The Design Builder provides a means for a design context to determine if it is valid and can/should the implementation proceed. After a design job creates and populates a design context, the job will call any methods on the context where the method name begins with validate_. These methods should not accept any arguments other than self and should either return None when valid or should raise nautobot_design_builder.DesignValidationError. In the above Context example, the design context checks to see if a site with the same name already exists, and if so it raises an error. Any number of validation methods can exist in a design context. Each will be called in the order it is defined in the class.
Design Templates¶
Primary Purpose:
- Generate YAML files that conform to Design Builder's design file format.
Design templates are Jinja templates that render to YAML. The YAML file represents a dictionary of objects that the design builder will create or update. The design builder supports all data models that exist within Nautobot, including any data models that are defined by applications installed within Nautobot. Top level keys in a design file map to the verbose plural name of the model. For instance, the dcim.Device model maps to the top level devices key within a design. Similarly, dcim.Location maps to locations.
Design templates support object hierarchy where relationship mapping takes place automatically during the design build process. For instance, consider the following:
locations:
- "name": "US-East-1"
location_type__name: "Region"
children:
- "name": "IAD5"
location_type__name: "Site"
status__name: "Active"
- "name": "LGA1"
location_type__name: "Site"
status__name: "Active"
This design template will create location types and several locations. The Design Builder automatically takes care of the underlying relationships so that IAD5 and LGA1 are correctly associated with the US-East-1 location. All relationships that are defined on the underlying database models are supported as nested objects within design templates.
Special Syntax - Query Fields¶
Syntax: field__<relatedfield>
Double underscores between a field and a relatedfield cause design builder to attempt to query a related object using the relatedfield as a query parameter. This query must return only one object. The returned object is then assigned to the field of the object being created or updated. For instance:
This template will attempt to find the platform with the name Arista EOS and then assign the object to the platform field on the device. The value for query fields can be a scalar or a dictionary. In the case above (platform__name) the scalar value "Arista EOS" expands the the equivalent ORM query: Platform.objects.get(name="Arista EOS") with the returned object being assigned to the platform attribute of the device.
If a query field's value is a dictionary, then more complex lookups can be performed. For instance:
The above query expands to the following ORM code: Platform.objects.get(name="Arista EOS", napalm_driver="eos") with the returned value being assigned to the platform attribute of the device.
Special Syntax - Action Tag¶
In addition to the object mapping described above, some additional syntax, in what are called Action Tags are available in YAML design templates. These actions tags have special meaning to the design builder. Sometimes additional information is needed before the Design Builder can determine how to update the database or how to map a relationship. These special cases are handled with action tags within the YAML design templates. Any mapping key that begins with an exclamation point (!) is considered an action tag and carries special meaning in the design template. An example of a situation where an action tag is required is when updating, rather than creating, an object in the database. If Nautobot already has an instance of a data model and a design only requires updating that model then the update: action tag can be used. This instructs the builder to update the object, rather than try to create it. The available action tags are documented below.
Note: Any YAML key that begins with an exclamation point (
!) must be quoted or a YAML syntax error will be raised.
Action Tag - Update¶
Syntax: !update:<field>
This syntax is used when you know an object already exists and indicates to design builder that the object should be updated, rather than created. An example of this is updating the description on an interface.
This template will find the interface with the name Ethernet1/1 and then set the description field.
Action Tag - Update or Create¶
Syntax: !create_or_update:<field>
Similar to !update this is used before a field name but will also create the object if it does not already exist. For example:
devices:
- "!create_or_update:name": bb-rtr-1
"interfaces":
- "!create_or_update:name": Ethernet1/1
description: "Uplink to provider"
This template will cause design builder to attempt to first lookup the device by the name bb-rtr-1, if not found it will be created. Subsequently, the device interface named Ethernet1/1 will also be either created or updated. Note that when being created all required fields must be specified. The above example would fail during creation since both the device and the interface are missing required fields. Design Builder performs model validation prior to saving any model to the database.
Action Tag - Git Context¶
Syntax: !git_context
Nautobot supports assigning config contexts from a git repository. Therefore the design builder supports generating config context data for an associated git repository. Using the !git_context key will indicate that design builder should store the rendered config context in a git repository. This functionality requires that a destination and data key are nested within the !git_context key, such as the below example:
"!git_context":
destination: "config_contexts/devices/{{ device_name }}.yml"
data:
{% include 'templates/spine_switch_config_context.yaml.j2' %}
In this example, the included template will be rendered and a new file will be created named config_context/devices/{{ device_name }}.yml, and the content of the rendered data template will be written to the file. Note that the context_repository configuration key must be set in the nautobot_config.py in order for this feature to work. More information can be found in the Git-Based Config Context documentation
Action Tag - Reference¶
Syntax: !ref
When used as a YAML mapping key, !ref will store a reference to the current Nautobot object for use later in the design template. The value of the !ref key is used as the reference name within design builder. This feature is useful when you need to set a relationship field to the value of a previously created or updated object. One use-case where this can come up is when uplinks on a device are dynamically assigned based on the next available ports. The exact interfaces used may not be known but references to them can be created and then reused later to create cable terminations.
# Creating a reference to spine interfaces.
#
# In the rendered YAML this ends up being something like
# "spine_switch1:Ethernet1", "spine_switch1:Ethernet2", etc
#
#
- "!create_or_update:name": "{{ interface }}"
"!ref": "{{ spine.name }}:{{ interface }}"
Special YAML Values¶
In addition to the special syntax provided for mapping keys, there are also some action tags provided for values. The following document the available value tags within the Design Builder.
Action Tag Value - Reference¶
Syntax: !ref
When used as the value for a key !ref:<reference_name> will return the the previously stored object. In our example of cabling the reference lookup will look something like this:
# Looking up a reference to previously created spine interfaces.
#
# In the rendered YAML "!ref:{{ spine.name }}:{{ interface }}" will become something like
# "!ref:spine_switch1:Ethernet1", "!ref:spine_switch1:Ethernet2", etc
# ObjectCreator will be able to assign the cable termination A side to the previously created objects.
#
#
{% for leaf,interface in spine_to_leaf[loop.index0].items() %}
- termination_a: "!ref:{{ spine.name }}:{{ interface }}"
termination_b: "!ref:{{ leaf }}:{{ leaf_to_spine[leaf_index][spine_index] }}"
status__name: "Planned"
{% endfor %}
In this case, both termination_a and termination_b will be assigned the objects by looking for references by their name.
Additional Jinja Control Syntax¶
The following section includes some additional information about Jinja syntax as well as describing some extensions to the Jinja parser that are provided by design builder.
The {% include %} statement¶
include is a built-in jinja expression that provides the ability to separate sections of a template into individual files and then produce a completed template as a composition of those files. This is useful when you want to split up a large template or it makes logical sense to keep certain parts separate, or if some of the template might be shared between two parent templates.
Of particular note when using this Jinja expression is that the included files must have the same indentation as the point of inclusion, otherwise a YAML syntax error is introduced. The following example demonstrates the include expression:
---
devices:
{% for device_name in device_names %}
- name: "{{ device_name }}"
{% include 'design_files/templates/switch_template.yaml.j2' %}
{% endfor %}
The path to the included template is relative to the directory where the design class is defined for the particular design job. Using the example layout defined above, this path would be designs/design_files/templates/switch_template.yaml.j2.
Extensions¶
Custom action tags can be created using template extensions. If a design needs custom functionality implemented as an action tag, the design developer can simply create a new tag (see the extension documentation). The new tag class can be added to the design using the extensions attribute in the design Meta class:
class DesignJobWithExtensions(DesignJob):
class Meta:
name = "Design with Custom Extensions"
design_file = "templates/simple_design.yaml.j2"
extensions = [CustomExtension]
Several additional extensions ship with Design Builder and are located in the nautobot_design_builder.contrib.ext module. This module includes several useful extensions to help with things like connecting cables or creating BGP peers. However, these extensions may not be supported in all versions of Nautobot or in all configurations. For instance, the bgp_peering action tag requires that the BGP models plugin be installed. Given that these extensions may require optional packages, and are not supported across the entire Nautobot ecosystem they are distributed in the contrib package.
In order to use any of these contributed packages, simply import the ext module and include the necessary extensions in the design job: