Caching¶

A fundamental trade-off in dynamic websites like Nautobot is that, well, they’re dynamic. Each time a user requests a page, the Web server makes all sorts of calculations – from database queries to template rendering to business logic – to create the page that your site’s visitor sees. This is a lot more expensive, from a processing-overhead perspective, than your standard read-a-file-off-the-filesystem server arrangement.

That’s where caching comes in.

To cache something is to save the result of an expensive calculation so that you don’t have to perform the calculation next time.

Nautobot makes extensive use of caching; this is not a simple topic but it's a useful one for a Nautobot administrator to understand, so read on if you please.

How it Works¶

Nautobot supports database query caching using django-cacheops and Redis. When a query is made, the results are cached in Redis for a short period of time, as defined by the CACHEOPS_DEFAULTS parameter (15 minutes by default). Within that time, all recurrences of that specific query will return the pre-fetched results from the cache. Caching can be completely disabled by toggling CACHEOPS_ENABLED to False (it is True by default).

If a change is made to any of the objects returned by the cached query within that time, or if the timeout expires, the cached results are automatically invalidated and the next request for those results will be sent to the database.

Caching is a complex topic and there are some important details to clarify with how caching is implemented and configureed in Nautobot.

Caching in Django¶

Django includes with its own cache framework that works for common cases, but does not work well for the wide array of use-cases within Nautobot. For that reason, Django's built-in caching is not used for the caching of web UI views, API results, and underlying database queries. Instead, we use django-cacheops. Please see below for more on this.

CACHES and django-redis¶

The CACHES setting is used to, among other things, configure Django's built-in caching. You'll observe that, even though we aren't using Django's built-in caching, we still have this as a required setting. Here's why:

Nautobot uses the django-redis Django plugin which allows it to use Redis as a backend for caching and session storage. This is used to provide a concurrent write lock for preventing race conditions when allocating IP address objects, and also to define centralized Redis connection settings that will be used by RQ.

django-redis also uses the CACHES setting, in its case to simplify the configuration for establishing concurrent write locks, and also for referencing the correct Redis connection information when defining RQ task queues using the RQ_QUEUES setting.

Again: CACHES is not used for Django's built-in caching at this time, but it is still a required setting for django-redis to function properly.

Django Cacheops¶

Cacheops (aka django-cacheops) is a Django plugin that does some very advanced caching, but does not leverage the built-in cache framework. Instead it uses a technique called "monkey patching". By monkey patching, a library can inject its own functionality into the core code behind the scenes.

This technique allows Cacheops to do more advanced caching operations that are not provided by the Django built-in cache framework without requiring Nautobot to also include some elaborate code of its own. This is accomplished by intercepting calls to the underlying queryset methods that get and set cached results in Redis.

For this purpose, Cacheops has its own CACHEOPS_* settings required to configure it that are not related to the CACHES setting.

For more information on the required settings needed to configure Cacheops, please see the Caching section of the required settings documentation.

The optional settings include:

• CACHEOPS_DEFAULTS: To define the cache timeout value (Defaults to 15 minutes)
• CACHEOPS_ENABLED : To turn on/off caching (Defaults to True)

Invalidating Cached Data¶

Although caching is performed automatically and rarely requires administrative intervention, Nautobot provides the invalidate management command to force invalidation of cached results. This command can reference a specific object my its type and UUID:

$ nautobot-server invalidate dcim.Device.84ae706d-c189-4d13-a898-9737648e34b3

Alternatively, it can also delete all cached results for an object type:

$ nautobot-server invalidate dcim.Device

Finally, calling it with the all argument will force invalidation of the entire cache database:

$ nautobot-server invalidate all

High Availability Caching¶

Redis provides two different methods to achieve high availability, the first is Redis Sentinel and the second is the newer Redis Clustering feature. Unfortunately, due to an issue with django-cacheops Nautobot is unable to support Redis Clustering at this time. Nautobot can however support Redis Sentinel.

Using Redis Sentinel¶

The installation/configuration of the Redis Sentinel cluster itself is outside the scope of this document, this section is intended to provide the steps necessary to configure Nautobot to connect to a Sentinel cluster.

We need to configure django-redis, django-cacheops, and celery to use Sentinel, of course each library is configured differently so pay close attention to the details:

django-redis Sentinel Configuration¶

Notable settings:

• SENTINELS: List of tuples or tuple of tuples with each inner tuple containing the name or IP address of the Redis server and port for each sentinel instance to connect to
• LOCATION: Similar to a redis URL, however, the hostname in the URL is the master/service name in redis sentinel
• SENTINEL_KWARGS: Options which will be passed directly to Redis Sentinel
• PASSWORD: The redis password (if set), the SENTINEL_KWARGS["password"] setting is the password for Sentinel

Example:

DJANGO_REDIS_CONNECTION_FACTORY = "django_redis.pool.SentinelConnectionFactory"
CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": "redis://nautobot/0",  # in this context 'nautobot' is the redis master/service name
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.SentinelClient",
            "CONNECTION_POOL_CLASS": "redis.sentinel.SentinelConnectionPool",
            "PASSWORD": "",
            "SENTINEL_KWARGS": {
                "password": "",  # likely the same password from above
            },
            "SENTINELS": [
                ("mysentinel.redis.example.com", 26379),
                ("othersentinel.redis.example.com", 26379),
                ("thirdsentinel.redis.example.com", 26379)
            ],
        },
    },
}

For more details on configuring django-redis with Redis Sentinel, please see the documentation for Django Redis.

django-cacheops Sentinel Configuration¶

Notable settings:

• locations: List of tuples or tuple of tuples with each inner tuple containing the name or IP address of the Redis server and port for each sentinel instance to connect to
• service_name: the master/service name in redis sentinel
• Additional parameters may be specified in the CACHEOPS_SENTINEL dictionary which are passed directly to Sentinel

Example:

CACHEOPS_REDIS = False
CACHEOPS_SENTINEL = {
    "db": 0,
    "locations": [
        ("mysentinel.redis.example.com", 26379),
        ("othersentinel.redis.example.com", 26379),
        ("thirdsentinel.redis.example.com", 26379)
    ],
    "service_name": "nautobot",
    "socket_timeout": 10,
    "sentinel_kwargs": {
        "password": ""
    },
    "password": "",
    # Everything else is passed to `Sentinel()`
}

For more details on how to configure Cacheops to use Redis Sentinel see the documentation for Cacheops setup.

celery Sentinel Configuration¶

Celery Sentinel configuration is controlled by 4 variables BROKER_URL, BROKER_TRANSPORT_OPTIONS, RESULT_BACKEND, and RESULT_BACKEND_TRANSPORT_OPTIONS. These parameters can be specified in the django settings in nautobot_config.py by prefixing these variable names with CELERY_. By default Nautobot configures the celery broker and results backend with the same configuration.

redis_password = ""
sentinel_password = ""

CELERY_BROKER_URL = (
    f"sentinel://:{redis_password}@mysentinel.redis.example.com:26379;"
    f"sentinel://:{redis_password}@othersentinel.redis.example.com:26379;"
    # The final entry must not have the `;` delimiter
    f"sentinel://:{redis_password}@thirdsentinel.redis.example.com:26379"
)
CELERY_BROKER_TRANSPORT_OPTIONS = {
    "master_name": "nautobot",
    "sentinel_kwargs": {"password": sentinel_password},
}

CELERY_RESULT_BACKEND = CELERY_BROKER_URL
CELERY_RESULT_BACKEND_TRANSPORT_OPTIONS = CELERY_BROKER_TRANSPORT_OPTIONS

For more details on how to configure Celery to use Redis Sentinel see the documentation for Celery.