Managing access to Amazon ElastiCache for Redis clusters with IAM

[nathanielDuro]

Hi everyone. I'm having a trouble on deploying Superset with Redis Auth. I'm a beginner to all of these so bare with me. Apache Superset 2.1.1 is deployed on Amazon EC2 Ubuntu 20.04. Redis Cluster 7.0.7 on AWS Elasticache. So far everything works with no Auth. Now since Redis 7 has auth capability, we are testing it out specially the IAM management. This is the CloudFormation Template based on this website:

RedisSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties: 
      GroupDescription: "Security group for Redis"
      VpcId: !Ref VpcId
      SecurityGroupIngress: 
        - IpProtocol: tcp
          FromPort: '6379'
          ToPort: '6379'
          SourceSecurityGroupId: !Ref EC2SecurityGroup
      SecurityGroupEgress:
        - IpProtocol: tcp
          FromPort: 0
          ToPort: 65535
          CidrIp: 0.0.0.0/0
  RedisSubnetGroup:
    Type: AWS::ElastiCache::SubnetGroup
    Properties:
      CacheSubnetGroupName: analytics-test-subnet
      Description: "Cache Subnet Group for analytics test redis"
      SubnetIds:
        - !Ref SubnetId
        - !Ref SubnetId2
  RedisUser:
    Type: AWS::ElastiCache::User
    Properties: 
      Engine: 'redis'
      AccessString: 'on ~* +@all'
      AuthenticationMode: 
        { Type: iam }
      UserId: 'supersettestuser' # UserId == UserName when AutheticationMode == iam
      UserName: 'supersettestuser'
  RedisDefaultUser:
    Type: AWS::ElastiCache::User
    Properties:
      Engine: 'redis'
      UserId: 'supersettestdefault'
      UserName: 'default'
      AccessString: 'off -@all'
      AuthenticationMode: 
        { 
        Type: password,
        Passwords: ["fRX0nEHjiUuTp68RL56Fk7VJ09739dLF"]
        }
  RedisUserGroup:
    Type: AWS::ElastiCache::UserGroup
    Properties: 
      Engine: 'redis'
      UserGroupId: "supersettestgroup"
      UserIds:
        - !Ref RedisDefaultUser
        - !Ref RedisUser
  Redis:
    Type: AWS::ElastiCache::ReplicationGroup
    Properties: 
      AtRestEncryptionEnabled: True
      AutomaticFailoverEnabled: False # Automatic failover requires 2 or more cache clusters. Create more cache clusters or disable automatic failover.
      AutoMinorVersionUpgrade: True
      #AuthToken: fRX0nEHjiUuTp68RL56Fk7VJ09739dLF
      CacheNodeType: cache.t4g.micro
      # CacheParameterGroupName: default.redis7.cluster.on
      CacheSubnetGroupName: !Ref RedisSubnetGroup
      ClusterMode: Disabled
      DataTieringEnabled: False
      Engine: Redis
      EngineVersion: 7.0
      IpDiscovery: ipv4
      MultiAZEnabled: False
      NetworkType: ipv4
      NumCacheClusters: 1
      #NumNodeGroups: 0 # Number of shards for Cluster mode Enabled
      Port: 6379
      #ReplicasPerNodeGroup: 0 # additional node not including the primary node for   Clusteer mode Enabled
      ReplicationGroupId: analytics-test-redis
      ReplicationGroupDescription: "Redis analytics test"
      SecurityGroupIds: 
        - !Ref RedisSecurityGroup
      TransitEncryptionEnabled: True
      UserGroupIds:
        - !Ref RedisUserGroup

This is the config file (which is based on one of the post here in slack):

CACHE_CONFIG = {
    'CACHE_TYPE': 'RedisCache',
    'CACHE_DEFAULT_TIMEOUT': 84600,
    'CACHE_KEY_PREFIX': 'superset_config_',
    'CACHE_REDIS_URL': 'redis://someurl.cache.amazonaws.com:6379/0'
}

DATA_CACHE_CONFIG = {
    'CACHE_TYPE': 'RedisCache',
    'CACHE_DEFAULT_TIMEOUT': 84600,
    'CACHE_KEY_PREFIX': 'superset_data_',
    'CACHE_REDIS_URL': 'redis://someurl.cache.amazonaws.com:6379/1'
}

FILTER_STATE_CACHE_CONFIG = {
    'CACHE_TYPE': 'RedisCache',
    'CACHE_DEFAULT_TIMEOUT': 84600,
    'CACHE_KEY_PREFIX': 'superset_filter_',
    'CACHE_REDIS_URL': 'redis://someurl.cache.amazonaws.com:6379/2'
}

EXPLORE_FORM_DATA_CACHE_CONFIG = {
    'CACHE_TYPE': 'RedisCache',
    'CACHE_DEFAULT_TIMEOUT': 84600,
    'CACHE_KEY_PREFIX': 'superset_explore_',
    'CACHE_REDIS_URL': 'redis://someurl.cache.amazonaws.com:6379/3'
}

The error is that the Superset webpage is not loading. Server not responding to any request.

-- Logs begin at Wed 2023-10-18 05:12:18 UTC, end at Wed 2023-10-18 06:22:33 UTC. --
Oct 18 06:20:14 ip-172-31-116-146 systemd[1]: Started Apache Superset service.
Oct 18 06:20:16 ip-172-31-116-146 superset[47268]: logging was configured successfully
Oct 18 06:20:16 ip-172-31-116-146 superset[47268]: 2023-10-18 06:20:16,947:INFO:superset.utils.logging_configurator:logging was configured successfully
Oct 18 06:20:16 ip-172-31-116-146 superset[47268]: 2023-10-18 06:20:16,954:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
Oct 18 06:20:18 ip-172-31-116-146 superset[47268]:  * Serving Flask app 'superset' (lazy loading)
Oct 18 06:20:18 ip-172-31-116-146 superset[47268]:  * Environment: production
Oct 18 06:20:18 ip-172-31-116-146 superset[47268]:    WARNING: This is a development server. Do not use it in a production deployment.
Oct 18 06:20:18 ip-172-31-116-146 superset[47268]:    Use a production WSGI server instead.
Oct 18 06:20:18 ip-172-31-116-146 superset[47268]:  * Debug mode: off
Oct 18 06:20:18 ip-172-31-116-146 superset[47268]: /home/ubuntu/venv/lib/python3.8/site-packages/flask_appbuilder/models/sqla/interface.py:64: SAWarning: relationship 'SqlaTable.slices' will copy column tables.id to column slices.datasource_id, which conflicts with relationship(s): 'Slice.table' (copies tables.id to slices.datasource_id). If this is not the intention, consider if these relationships should be linked with back_populates, or if viewonly=True should be applied to one or more if they are read-only. For the less common case that foreign key constraints are partially overlapping, the orm.foreign() annotation can be used to isolate the columns that should be written towards.   To silence this warning, add the parameter 'overlaps="table"' to the 'SqlaTable.slices' relationship. (Background on this error at: https://sqlalche.me/e/14/qzyx)
Oct 18 06:20:18 ip-172-31-116-146 superset[47268]:   for prop in class_mapper(obj).iterate_properties:
Oct 18 06:20:18 ip-172-31-116-146 superset[47268]: 2023-10-18 06:20:18,565:INFO:werkzeug:WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
Oct 18 06:20:18 ip-172-31-116-146 superset[47268]:  * Running on all addresses (0.0.0.0)
Oct 18 06:20:18 ip-172-31-116-146 superset[47268]:  * Running on http://127.0.0.1:8088
Oct 18 06:20:18 ip-172-31-116-146 superset[47268]:  * Running on http://172.31.116.146:8088
Oct 18 06:20:18 ip-172-31-116-146 superset[47268]: 2023-10-18 06:20:18,565:INFO:werkzeug:Press CTRL+C to quit
Oct 18 06:20:18 ip-172-31-116-146 superset[47268]: 2023-10-18 06:20:18,567:INFO:werkzeug: * Restarting with stat
Oct 18 06:20:20 ip-172-31-116-146 superset[47273]: logging was configured successfully
Oct 18 06:20:20 ip-172-31-116-146 superset[47273]: 2023-10-18 06:20:20,483:INFO:superset.utils.logging_configurator:logging was configured successfully
Oct 18 06:20:20 ip-172-31-116-146 superset[47273]: 2023-10-18 06:20:20,490:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
Oct 18 06:20:22 ip-172-31-116-146 superset[47273]: /home/ubuntu/venv/lib/python3.8/site-packages/flask_appbuilder/models/sqla/interface.py:64: SAWarning: relationship 'SqlaTable.slices' will copy column tables.id to column slices.datasource_id, which conflicts with relationship(s): 'Slice.table' (copies tables.id to slices.datasource_id). If this is not the intention, consider if these relationships should be linked with back_populates, or if viewonly=True should be applied to one or more if they are read-only. For the less common case that foreign key constraints are partially overlapping, the orm.foreign() annotation can be used to isolate the columns that should be written towards.   To silence this warning, add the parameter 'overlaps="table"' to the 'SqlaTable.slices' relationship. (Background on this error at: https://sqlalche.me/e/14/qzyx)
Oct 18 06:20:22 ip-172-31-116-146 superset[47273]:   for prop in class_mapper(obj).iterate_properties:
Oct 18 06:20:22 ip-172-31-116-146 superset[47273]: 2023-10-18 06:20:22,119:WARNING:werkzeug: * Debugger is active!
Oct 18 06:20:22 ip-172-31-116-146 superset[47273]: 2023-10-18 06:20:22,120:INFO:werkzeug: * Debugger PIN: 106-682-528
Oct 18 06:20:33 ip-172-31-116-146 superset[47273]: 2023-10-18 06:20:33,016:ERROR:werkzeug:104.28.220.30 - - [18/Oct/2023 06:20:33] code 400, message Bad request version ('2¯\x9a\x88')
Oct 18 06:20:33 ip-172-31-116-146 superset[47273]: 2023-10-18 06:20:33,016:INFO:werkzeug:104.28.220.30 - - [18/Oct/2023 06:20:33] "\x16\x03\x01\x024\x01\x00\x020\x03\x03±:@\x16\x1cúNþÚZ\x11¦¬Ö\x092¯\x9a\x88" HTTPStatus.BAD_REQUEST -
Oct 18 06:20:33 ip-172-31-116-146 superset[47273]: 2023-10-18 06:20:33,252:INFO:werkzeug:172.31.127.135 - - [18/Oct/2023 06:20:33] "GET / HTTP/1.1" 302 -
Oct 18 06:20:33 ip-172-31-116-146 superset[47273]: 2023-10-18 06:20:33,491:ERROR:werkzeug:104.28.220.30 - - [18/Oct/2023 06:20:33] code 400, message Bad request version ('Le\x8cu\x00з\x16aô}\x9fSµ»@åZf\x127.\x82µn¸¡')
Oct 18 06:20:33 ip-172-31-116-146 superset[47273]: 2023-10-18 06:20:33,491:INFO:werkzeug:104.28.220.30 - - [18/Oct/2023 06:20:33] "\x16\x03\x01\x02T\x01\x00\x02P\x03\x03\x9f\x1b\x97nÔ¥¨\x97\x02ÖvZ\x11\\f$÷ÝÝö[&\x0b2Í\x10\x80|ÅUÄò X@«Á nW#\x10¯ñ|Ûct ³'ú±L\x82«£¡íú+¬]\x04Ã\x00 ÊÊ\x13\x01\x13\x02\x13\x03À+À/À,À0̨̩À\x13À\x14\x00\x9c\x00\x9d\x00/\x005\x01\x00\x01çZZ\x00\x00þ\x0d\x00ú\x00\x00\x01\x00\x01n\x00 U&-\x1dÿÄ\x86TÜ\x0fÈ"Á¤_¨æÈ{Y¥¶¶\x9a y\x87 Le\x8cu\x00з\x16aô}\x9fSµ»@åZf\x127.\x82µn¸¡" HTTPStatus.BAD_REQUEST -
Oct 18 06:20:33 ip-172-31-116-146 superset[47273]: 2023-10-18 06:20:33,756:INFO:werkzeug:104.28.220.30 - - [18/Oct/2023 06:20:33] "GET / HTTP/1.1" 302 -
Oct 18 06:20:34 ip-172-31-116-146 superset[47273]: Class 'werkzeug.local.LocalProxy' is not mapped
Oct 18 06:20:34 ip-172-31-116-146 superset[47273]: 2023-10-18 06:20:34,037:INFO:werkzeug:104.28.220.30 - - [18/Oct/2023 06:20:34] "GET /superset/welcome/ HTTP/1.1" 302 -
Oct 18 06:20:44 ip-172-31-116-146 superset[47273]: 2023-10-18 06:20:44,849:INFO:werkzeug:172.31.108.161 - - [18/Oct/2023 06:20:44] "GET / HTTP/1.1" 302 -
Oct 18 06:21:03 ip-172-31-116-146 superset[47273]: 2023-10-18 06:21:03,282:INFO:werkzeug:172.31.127.135 - - [18/Oct/2023 06:21:03] "GET / HTTP/1.1" 302 -
Oct 18 06:21:14 ip-172-31-116-146 superset[47273]: 2023-10-18 06:21:14,880:INFO:werkzeug:172.31.108.161 - - [18/Oct/2023 06:21:14] "GET / HTTP/1.1" 302 -
Oct 18 06:21:33 ip-172-31-116-146 superset[47273]: 2023-10-18 06:21:33,293:INFO:werkzeug:172.31.127.135 - - [18/Oct/2023 06:21:33] "GET / HTTP/1.1" 302 -
Oct 18 06:21:44 ip-172-31-116-146 superset[47273]: 2023-10-18 06:21:44,910:INFO:werkzeug:172.31.108.161 - - [18/Oct/2023 06:21:44] "GET / HTTP/1.1" 302 -
Oct 18 06:22:03 ip-172-31-116-146 superset[47273]: 2023-10-18 06:22:03,325:INFO:werkzeug:172.31.127.135 - - [18/Oct/2023 06:22:03] "GET / HTTP/1.1" 302 -
Oct 18 06:22:14 ip-172-31-116-146 superset[47273]: 2023-10-18 06:22:14,941:INFO:werkzeug:172.31.108.161 - - [18/Oct/2023 06:22:14] "GET / HTTP/1.1" 302 -
Oct 18 06:22:33 ip-172-31-116-146 superset[47273]: 2023-10-18 06:22:33,331:INFO:werkzeug:172.31.127.135 - - [18/Oct/2023 06:22:33] "GET / HTTP/1.1" 302 -

[milashenko]

For those who find this discussion, searching for the same problem, the answer would be either disable TransitEncryptionEnabled, or use "rediss://" protocol instead of "redis://".
This is so frustrating that superset doesn't log any way failed connection attempts. The only way to discover this is to remove all configuration, and then enable them one by one, to figure out that it is redis configuration responsible for this, ensure that host can actually connect, to the port, but... you know, it could be so lovely to have something in log about connection to redis timed out, but nop.