Enable Multi DB #20305
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,6 @@ | ||
| {% if INSTANCES %} | ||
| {% for redis_inst, redis_items in INSTANCES.items() %} | ||
|
|
||
|
There was a problem hiding this comment. Remove this unnecessary change? There was a problem hiding this comment. If there is no blank line here, all program:{{ redis_inst }} of critical_processes will be concentrated on one line, which will cause subsequent regular expression matching to fail. There was a problem hiding this comment. Does changing There was a problem hiding this comment. Changing {%- endfor %} to {% endfor %} works. I apply this change instead of adding blank line. |
||
| program:{{ redis_inst }} | ||
| {%- endfor %} | ||
| {%- endif %} | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -50,7 +50,11 @@ mkdir -p /etc/supervisor/conf.d/ | |
| if [ -f /etc/sonic/database_config$NAMESPACE_ID.json ]; then | ||
| cp /etc/sonic/database_config$NAMESPACE_ID.json $REDIS_DIR/sonic-db/database_config.json | ||
| else | ||
| if [ -f /etc/sonic/enable_multidb ]; then | ||
| HOST_IP=$host_ip REDIS_PORT=$redis_port DATABASE_TYPE=$DATABASE_TYPE BMP_DB_PORT=$BMP_DB_PORT j2 /usr/share/sonic/templates/multi_database_config.json.j2 > $REDIS_DIR/sonic-db/database_config.json | ||
| else | ||
| HOST_IP=$host_ip REDIS_PORT=$redis_port DATABASE_TYPE=$DATABASE_TYPE BMP_DB_PORT=$BMP_DB_PORT j2 /usr/share/sonic/templates/database_config.json.j2 > $REDIS_DIR/sonic-db/database_config.json | ||
| fi | ||
|
There was a problem hiding this comment. @Pan-XT any sonic-mgmt tests for multi DB? There was a problem hiding this comment. |
||
| fi | ||
|
|
||
| # on VoQ system, we only publish redis_chassis instance and CHASSIS_APP_DB when | ||
|
|
@@ -125,6 +129,12 @@ do | |
| else | ||
| echo -n > /var/lib/$inst/dump.rdb | ||
| fi | ||
|
|
||
| chown -R redis:redis /var/lib/$inst | ||
|
There was a problem hiding this comment. Add comments to this file, help other understand. There was a problem hiding this comment. Here are the necessary permissions for enable_multidb, see /etc/supervisor/conf.d/supervisord.conf.j2 There was a problem hiding this comment. Hi @Pan-XT , can you add your comments to this file? that will help other understand the propose of this code. There was a problem hiding this comment. ok，I add commnets “the Redis process is operating under the 'redis' user in supervisord and make redis user own /var/lib/$inst inside db container.” |
||
| sock_file="$REDIS_DIR/$inst.sock" | ||
| if [[ -f $sock_file ]]; then | ||
| chgrp -f redis $sock_file && chmod -f 0760 $sock_file | ||
| fi | ||
| done | ||
|
|
||
| TZ=$(cat /etc/timezone) | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,192 @@ | ||
| {% set include_remote_db = (REMOTE_DB_IP is defined and REMOTE_DB_PORT is defined) %} | ||
| { | ||
| "INSTANCES": { | ||
| "redis":{ | ||
| "hostname" : "{{HOST_IP}}", | ||
| "port" : {{REDIS_PORT}}, | ||
| "unix_socket_path" : "/var/run/redis{{DEV}}/redis.sock", | ||
| "persistence_for_warm_boot" : "yes" | ||
| {% if DATABASE_TYPE is defined and DATABASE_TYPE != "" %} | ||
| ,"database_type": "{{DATABASE_TYPE}}" | ||
|
There was a problem hiding this comment. I didn't find code in sonic-swss-common to parse and handle 'database_type', is there a PR? There was a problem hiding this comment. I search that PR, not find any code read "database_type" Otherwise, sonic-swss-common lib can't handle multi-DB, so please make sure sonic-swss-common also change to handle it. There was a problem hiding this comment. There was a problem hiding this comment. This field was added via It is current SONiC behavior Can you help to check with him to see what the need for this field? There was a problem hiding this comment. #Closed, I will offline sync with Ze |
||
| {% endif %} | ||
| }, | ||
| "redis1": { | ||
| "hostname" : "{{HOST_IP}}", | ||
| "port" : 6378, | ||
|
There was a problem hiding this comment. pls note that env REDIS_PORT/BMP_DB_PORT need to be reserved, and port > 6400 is not used currently. |
||
| "unix_socket_path" : "/var/run/redis{{DEV}}/redis1.sock", | ||
| "persistence_for_warm_boot" : "yes" | ||
| {% if DATABASE_TYPE is defined and DATABASE_TYPE != "" %} | ||
| ,"database_type": "{{DATABASE_TYPE}}" | ||
| {% endif %} | ||
| }, | ||
| "redis2": { | ||
| "hostname" : "{{HOST_IP}}", | ||
| "port" : 6377, | ||
| "unix_socket_path" : "/var/run/redis{{DEV}}/redis2.sock", | ||
| "persistence_for_warm_boot" : "yes" | ||
| {% if DATABASE_TYPE is defined and DATABASE_TYPE != "" %} | ||
| ,"database_type": "{{DATABASE_TYPE}}" | ||
| {% endif %} | ||
| }, | ||
| "redis3": { | ||
| "hostname" : "{{HOST_IP}}", | ||
| "port" : 6376, | ||
| "unix_socket_path" : "/var/run/redis{{DEV}}/redis3.sock", | ||
| "persistence_for_warm_boot" : "yes" | ||
| {% if DATABASE_TYPE is defined and DATABASE_TYPE != "" %} | ||
| ,"database_type": "{{DATABASE_TYPE}}" | ||
| {% endif %} | ||
| }, | ||
| "redis4": { | ||
| "hostname" : "{{HOST_IP}}", | ||
| "port" : 6375, | ||
| "unix_socket_path" : "/var/run/redis{{DEV}}/redis4.sock", | ||
| "persistence_for_warm_boot" : "yes" | ||
| {% if DATABASE_TYPE is defined and DATABASE_TYPE != "" %} | ||
| ,"database_type": "{{DATABASE_TYPE}}" | ||
| {% endif %} | ||
| }, | ||
| "redis_chassis":{ | ||
| "hostname" : "redis_chassis.server", | ||
| "port": 6380, | ||
| "unix_socket_path": "/var/run/redis-chassis/redis_chassis.sock", | ||
| "persistence_for_warm_boot" : "yes" | ||
| } | ||
| {% if include_remote_db %} | ||
| ,"remote_redis":{ | ||
| "hostname" : "{{REMOTE_DB_IP}}", | ||
| "port" : {{REMOTE_DB_PORT}}, | ||
| "unix_socket_path": "", | ||
| "persistence_for_warm_boot" : "yes" | ||
| } | ||
| {% endif %}, | ||
| "redis_bmp":{ | ||
| "hostname" : "{{HOST_IP}}", | ||
| "port" : {{BMP_DB_PORT}}, | ||
| "unix_socket_path" : "/var/run/redis{{DEV}}/redis_bmp.sock", | ||
| "persistence_for_warm_boot" : "yes" | ||
| } | ||
| }, | ||
| "DATABASES" : { | ||
| "APPL_DB": { | ||
| "id": 0, | ||
| "separator": ":", | ||
| "instance": "redis1" | ||
| }, | ||
| "ASIC_DB": { | ||
| "id": 1, | ||
| "separator": ":", | ||
| "instance": "redis2" | ||
| }, | ||
| "COUNTERS_DB": { | ||
| "id": 2, | ||
| "separator": ":", | ||
| "instance": "redis3" | ||
| }, | ||
| "LOGLEVEL_DB": { | ||
| "id": 3, | ||
| "separator": ":", | ||
| "instance": "redis" | ||
| }, | ||
| "CONFIG_DB": { | ||
| "id": 4, | ||
| "separator": "|", | ||
| "instance": "redis" | ||
| }, | ||
| "PFC_WD_DB": { | ||
| "id": 5, | ||
| "separator": ":", | ||
| "instance": "redis3" | ||
| }, | ||
| "FLEX_COUNTER_DB": { | ||
| "id": 5, | ||
| "separator": ":", | ||
| "instance": "redis3" | ||
| }, | ||
| "STATE_DB": { | ||
| "id": 6, | ||
| "separator": "|", | ||
| "instance": "redis" | ||
| }, | ||
| "SNMP_OVERLAY_DB": { | ||
| "id": 7, | ||
| "separator": "|", | ||
| "instance": "redis" | ||
| }, | ||
| "SYSMON_DB": { | ||
| "id": 10, | ||
| "separator": "|", | ||
| "instance": "redis" | ||
| }, | ||
| "BMC_DB": { | ||
| "id": 12, | ||
| "separator": ":", | ||
| "instance": "redis4" | ||
| }, | ||
| "RESTAPI_DB": { | ||
| "id": 8, | ||
| "separator": "|", | ||
| "instance": "redis" | ||
| }, | ||
| "GB_ASIC_DB": { | ||
| "id": 9, | ||
| "separator": ":", | ||
| "instance": "redis" | ||
| }, | ||
| "GB_COUNTERS_DB": { | ||
| "id": 10, | ||
| "separator": ":", | ||
| "instance": "redis" | ||
| }, | ||
| "GB_FLEX_COUNTER_DB": { | ||
| "id": 11, | ||
| "separator": ":", | ||
| "instance": "redis" | ||
| }, | ||
| "APPL_STATE_DB": { | ||
| "id": 14, | ||
| "separator": ":", | ||
| "instance": "redis" | ||
| }, | ||
| "CHASSIS_APP_DB" : { | ||
| "id" : 12, | ||
| "separator": "|", | ||
| "instance" : "redis_chassis" | ||
| }, | ||
| "CHASSIS_STATE_DB" : { | ||
| "id" : 13, | ||
| "separator": "|", | ||
| "instance" : "redis_chassis" | ||
| } | ||
| {% if DATABASE_TYPE is defined and DATABASE_TYPE == "dpudb" %} | ||
| , | ||
| "DPU_APPL_DB" : { | ||
| "id" : 15, | ||
| "separator": ":", | ||
| "instance" : {% if include_remote_db %} "remote_redis" {% else %} "redis" {% endif %}, | ||
|
There was a problem hiding this comment. Suggest define a variable for all these dupe template There was a problem hiding this comment. Currently, the multi_database_config.json.j2 is consistent with the database_config.json.j2, except for the added content related to redis1,redis2,redis3,redis4. Defining a variable for all these dupe templates will be implemented in future modifications. |
||
| "format": "proto" | ||
| }, | ||
| "DPU_APPL_STATE_DB" : { | ||
| "id" : 16, | ||
| "separator": "|", | ||
| "instance" : {% if include_remote_db %} "remote_redis" {% else %} "redis" {% endif %} | ||
| }, | ||
| "DPU_STATE_DB" : { | ||
| "id" : 17, | ||
| "separator": "|", | ||
| "instance" : {% if include_remote_db %} "remote_redis" {% else %} "redis" {% endif %} | ||
| }, | ||
| "DPU_COUNTERS_DB" : { | ||
| "id" : 18, | ||
| "separator": ":", | ||
| "instance" : {% if include_remote_db %} "remote_redis" {% else %} "redis" {% endif %} | ||
| } | ||
| {% endif %}, | ||
| "BMP_STATE_DB" : { | ||
| "id" : 20, | ||
| "separator": "|", | ||
| "instance" : "redis_bmp" | ||
| } | ||
| }, | ||
| "VERSION" : "1.0" | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -31,6 +31,8 @@ dependent_startup=true | |
| {% if INSTANCES %} | ||
| {% for redis_inst, redis_items in INSTANCES.items() %} | ||
| {%- if redis_inst != 'remote_redis' %} | ||
|
|
||
|
There was a problem hiding this comment. Suggest remove unnecessary change in this file There was a problem hiding this comment. This change is the same as #20545 , so I remove it. |
||
|
|
||
| [program:{{ redis_inst }}] | ||
| {% if redis_items['hostname'] != '127.0.0.1' %} | ||
| {%- set ADDITIONAL_OPTS = '--protected-mode no' %} | ||
|
|
@@ -53,6 +55,7 @@ stderr_logfile=syslog | |
| {% endfor %} | ||
| {% endif %} | ||
|
|
||
|
|
||
| [program:flushdb] | ||
| command=/bin/bash -c "sleep 300 && /usr/local/bin/flush_unused_database" | ||
| priority=3 | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Pan-XT Could you upload a write-up for this multi-DB including the current issues and usage.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
https://github.com/dzhangalibaba/SONiC/blob/master/doc/database/multi_database_instances.md
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
#Closed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Pan-XT Could you update the HLD with some performance numbers with the multi DB testing?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Pan-XT Also provide a table with number of redis instances used during testing and what is the ideal number of instances used.