How to Implement Warm-Up Caching in Apache Superset Using Redis and Celery for Dashboards with multiple JINJA params ?

[cellou59]

Hello everyone,

I am currently working on optimizing the performance of my Apache Superset dashboards by implementing a warm-up cache strategy. My goal is to pre-load the cache for multiple frequently accessed dashboards to improve load times for users. Here's a detailed breakdown of my setup and requirements:

Current Setup:

1. Superset Version: [Specify your version]
2. Environment: Superset running on a server with Apache
3. Caching Backend: Redis
4. Task Queue: Celery
5. Databases: MySQL for Superset metadata, various analytical databases for data sources

Requirements:

• I need to implement a warm-up caching strategy that pre-loads cache for multiple dashboards.
• These dashboards have specific IDs, e.g., [240, 123, 34].
• The warm-up process should be scheduled to run every day at 6 AM.
• Each dashboard may have variable parameters such as user_id, which should be included in the cache warming process.

Current Configuration in superset_config.py:

from celery import Celery
from celery.schedules import crontab
from flask_caching import Cache
from superset import app
from superset.tasks.cache import cache_warmup_task

# General Superset configuration
SECRET_KEY = "your_secret_key"
SQLALCHEMY_DATABASE_URI = 'mysql://superset:password@hostname:3306/superset'

# Redis Cache Configuration
CACHE_CONFIG = {
    'CACHE_TYPE': 'redis',
    'CACHE_REDIS_URL': 'redis://localhost:6379/0',
    'CACHE_DEFAULT_TIMEOUT': 86400,
    'CACHE_KEY_PREFIX': 'superset_'
}

DATA_CACHE_CONFIG = {
    'CACHE_TYPE': 'redis',
    'CACHE_REDIS_URL': 'redis://localhost:6379/0',
    'CACHE_DEFAULT_TIMEOUT': 86400,
    'CACHE_KEY_PREFIX': 'superset_results_cache_'
}

FILTER_STATE_CACHE_CONFIG = {
    'CACHE_TYPE': 'redis',
    'CACHE_REDIS_URL': 'redis://localhost:6379/0',
    'CACHE_DEFAULT_TIMEOUT': 86400,
    'CACHE_KEY_PREFIX': 'superset_filter_cache'
}

# Celery Configuration
class CeleryConfig:
    broker_url = 'redis://localhost:6379/0'
    result_backend = 'redis://localhost:6379/0'
    imports = ('superset.tasks.cache',)
    task_acks_late = True
    worker_prefetch_multiplier = 1
    task_annotations = {
        'sql_lab.get_sql_results': {
            'rate_limit': '100/s',
        },
    }
    beat_schedule = {
        'warmup-cache-every-day': {
            'task': 'warmup_dashboards_cache',
            'schedule': crontab(minute=0, hour=6),  # Run daily at 6 AM
            'args': (???)  **# I have replace this with actual dashboard and JINJA params like (user IDs) ??**
        },
    }

CELERY_CONFIG = CeleryConfig

# Warm-up cache task
celery_app = Celery('superset.tasks', broker=CELERY_CONFIG.broker_url)
celery_app.conf.update(app.config)

@celery_app.task
def warmup_dashboard_cache(dashboard_id, user_id):
    with app.app_context():
        cache_warmup_task(dashboard_id=dashboard_id, user_id=user_id)

@celery_app.task
def warmup_dashboards_cache(dashboard_ids, user_id):
    for dashboard_id in dashboard_ids:
        warmup_dashboard_cache.delay(dashboard_id, user_id)

Steps Taken:

1. Redis and Celery Configuration: Redis is configured as the cache backend and Celery as the task queue.
2. Cache Warm-up Task: Defined a Celery task to warm-up the cache for specific dashboards.
3. Scheduling: Used Celery Beat to schedule the warm-up task to run daily at 6 AM.

Issues/Questions:

1. Is this approach optimal for pre-loading caches for multiple dashboards?
2. How can I ensure that variable parameters (e.g., user_id) are correctly handled during the cache warm-up process?
3. Are there any additional best practices or configurations I should consider to improve performance and reliability?
4. How can I monitor and verify that the cache warm-up process is working as expected?

Any guidance, suggestions, or examples from similar implementations would be greatly appreciated.

Thank you!