Skip to content

Redis for Caching and Celery

Redis is an in-memory data store similar to memcached. It is required to support Nautobot's caching, task queueing, and webhook features. The connection settings are explained here, allowing Nautobot to connect to different Redis instances/databases per feature.

Warning

It is highly recommended to keep the Redis databases for caching and tasks separate. Using the same database number on the same Redis instance for both may result in queued background tasks being lost during cache flushing events. For this reason, the default settings utilize database 1 for caching and database 0 for tasks.

Tip

The default Redis settings in your nautobot_config.py should be suitable for most deployments and should only require customization for more advanced configurations. See below for examples.

Caching

Out of the box you do not need to make any changes to utilize caching with Redis. The default settings are sufficient for most installations.

Django includes its own cache framework. Nautobot uses this cache framework, and specifically the extension django-redis, which allows it to use Redis as a backend for caching and session storage. This extension is also used to provide a concurrent write lock for preventing race conditions when allocating IP address objects.

The CACHES setting is used to, among other things, configure Django's built-in caching and the django-redis extension to appropriately use Redis.

Task Queuing with Celery

Out of the box you do not need to make any changes to utilize task queueing with Celery and Redis. The default settings are sufficient for most installations.

In the event you do need to make customizations to how Celery interacts with the message broker such as for more advanced clustered deployments, the following settings may be changed:

High Availability Using Redis Sentinel

Redis provides two different methods to achieve high availability: The first is Redis Sentinel and the second is the newer Redis Clustering feature. Currently, Nautobot only supports Redis Sentinel for high availability.

The installation/configuration of the Redis Sentinel cluster itself is outside the scope of this document, this section is intended to provide the steps necessary to configure Nautobot to connect to a Sentinel cluster.

We need to configure django-redis and celery to use Sentinel. Each library is configured differently, so please pay close attention to the details.

django-redis Sentinel Configuration

Notable settings:

  • SENTINELS: List of tuples or tuple of tuples with each inner tuple containing the name or IP address of the Redis server and port for each sentinel instance to connect to
  • LOCATION: Similar to a redis URL, however, the hostname in the URL is the master/service name in redis sentinel
  • SENTINEL_KWARGS: Options which will be passed directly to Redis Sentinel
  • PASSWORD: The redis password (if set), the SENTINEL_KWARGS["password"] setting is the password for Sentinel

Example:

DJANGO_REDIS_CONNECTION_FACTORY = "django_redis.pool.SentinelConnectionFactory"
CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": "redis://nautobot/0",  # in this context 'nautobot' is the redis master/service name
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.SentinelClient",
            "CONNECTION_POOL_CLASS": "redis.sentinel.SentinelConnectionPool",
            "PASSWORD": "",
            "SENTINEL_KWARGS": {
                "password": "",  # likely the same password from above
            },
            "SENTINELS": [
                ("mysentinel.redis.example.com", 26379),
                ("othersentinel.redis.example.com", 26379),
                ("thirdsentinel.redis.example.com", 26379)
            ],
        },
    },
}

For more details on configuring django-redis with Redis Sentinel, please see the documentation for Django Redis.

celery Sentinel Configuration

Changed in version 2.0.0 — Do not change CELERY_RESULT_BACKEND or CELERY_RESULT_BACKEND_TRANSPORT_OPTIONS

Celery now stores results in the Nautobot database. The CELERY_RESULT_BACKEND and CELERY_RESULT_BACKEND_TRANSPORT_OPTIONS should not be changed from their default values.

Celery Sentinel configuration is controlled by two settings within your nautobot_config.py:

redis_password = ""
sentinel_password = ""

CELERY_BROKER_URL = (
    f"sentinel://:{redis_password}@mysentinel.redis.example.com:26379;"
    f"sentinel://:{redis_password}@othersentinel.redis.example.com:26379;"
    # The final entry must not have the `;` delimiter
    f"sentinel://:{redis_password}@thirdsentinel.redis.example.com:26379"
)
CELERY_BROKER_TRANSPORT_OPTIONS = {
    "master_name": "nautobot",
    "sentinel_kwargs": {"password": sentinel_password},
}

Please see the official Celery documentation for more information on how to configure Celery to use Redis Sentinel.