Run Time Buffers Configuration Update design
Run-time Buffers Configuration feature allows to adjust buffer parameters on port when port speed changes.
Current buffers configuration implementation performs only initial configuration on switch start.
An update suggested in this design document enables dynamic buffer profile selection based on current port parameters (speed and attached cable length) which allows to guarantee lossless traffic flows after port speed change.
PG profile update on port speed change includes:
- Get cable length for this port
- Get speed for this port (speed )
- Using speed&cable-to-pg_profile look-up table determine new profile.
Port speed and cable length are the two parameters which uniquely define the profile to be used on the port. - If profile is different from the old one – replace profile configuration on port
This also includes:- Check whether old profile is still used on other ports – if not – delete it
- Check whether new profile is already used. If not – create it first
- Recalculate and update PG Pool size
Three Databases are involved:
- BUFFERS_DB: Stores all information needed for buffer configuration. Used by BufferConfigDaemon only.
- CONFIG_DB: Stores information expected to be applied on the switch.
- STATE_DB: Stores information actually applied on the switch.
CONFIG_DB and STATE_DB represent former APPL_DB. CONFIG_DB contains requested configuration and STATE_DB - configuration actually applied on the ASIC possibly including some run-time specific details.
The template used for buffers config generation should be split into several parts
- generic - containing information about cable length
- vendor/platform specific. This includes buffer pools, profiles, mappings, look-up table for pg
- static part - tables which do not use any variables (could be generic or vendor/platform specific)
If not yet done in scope of some other task, swssconfig needs to be updated to allow inserting data from the provided json to some specific DB.
Partially Buffer Configuration Daemon(BCD) is already implemented in buffersorch.cpp. Only few modifications needed:
- update existing PG creation code to create PG profiles on-demand.
- add subscription to BUFFERS_DB:* to access buffers related tables
- add subscription to STATE_DB:PORT_TABLE to monitor port speed change
- subscribe to CONFIG_DB instead of APPL_DB to track desired buffers configuration
- write buffers configuration to STATE_DB after it is successfully applied on ASIC
swssconfig utility will write information to this DB. The following data will go there:
-
cables.json
Tables:{ "CABLE_LENGTH_TABLE": { "Ethernet0":"40m", "Ethernet4":"100m", ... }, "OP": "SET" }
-
msn27xx.buffers.json (just like it is now, file name could contain platform name/model) Static json file provided by vendor. Contains all non-PG related pools, profiles, mappings.
-
msn27xx.pg_lossless_buffers.json
Generated from j2 template provided by vendor. Template should ensure generation of the following tables:-
Profile for each port BUFFER_PG_TABLE:
E.g.{ "BUFFER_PG_TABLE:Ethernet0:3-4": { "profile" : "[BUFFER_PROFILE_TABLE:pg_lossless_100G_300m_profile]" }, "OP": "SET" } -
All possible profiles declaration BUFFER_PROFILE_TABLE: E.g.
{ "BUFFER_PROFILE_TABLE:pg_lossless_100G_300m_profile": { "pool":"[BUFFER_POOL_TABLE:ingress_lossless_pool]", "xon":"18432", "xoff":"165888", "size":"184320", "dynamic_th":"1" }, "OP": "SET" }
-
Optionally The last two configs can be combined to one file.
BUFFERS_DB is owned and used by the Buffer Configuration Daemon #1 (see chapter 3.4)
Buffer Config Daemon performs unconditional copy of the tables added to BUFFERS_DB from json msn27xx.buffers.json to the CONFIG_DB.
Tables related to PG lossless queues are created in CONFIG_DB basing on tables received from cables.json, pg_lossless_buffers and port speed from the STATE_DB.
Pool table for PG lossless
Data in STATE_DB will be created/updated after configuration requested in CONFIG_DB is successfully created on ASIC. TODO: Clarify schema.
Split into several parts:
-
generic static
Cable length table if it is predefined:{ "CABLE_LENGTH_TABLE": { "Ethernet0":"40m", "Ethernet4":"100m", ... }, "OP": "SET" } -
generic generated
Cable length if elaborated from the switch and neighbor role (field: type){# Template to generate table with cables length #} {%- macro cable_length(port_name) -%} {%- set cable_len = [] -%} {%- for local_port in DEVICE_NEIGHBOR -%} {%- if local_port == port_name -%} {%- if DEVICE_NEIGHBOR_METADATA[DEVICE_NEIGHBOR[local_port].name] -%} {%- set neighbor = DEVICE_NEIGHBOR_METADATA[DEVICE_NEIGHBOR[local_port].name] -%} {%- set neighbor_role = neighbor.type -%} {%- set roles1 = switch_role + '_' + neighbor_role %} {%- set roles2 = neighbor_role + '_' + switch_role -%} {%- if roles1 in ports2cable -%} {%- if cable_len.append(ports2cable[roles1]) -%}{%- endif -%} {%- elif roles2 in ports2cable -%} {%- if cable_len.append(ports2cable[roles2]) -%}{%- endif -%} {%- endif -%} {% endif %} {% endif %} {%- endfor -%} {%- if cable_len -%} {{ cable_len.0 }} {%- else -%} {{ supported_cable | last }} {%- endif -%} {% endmacro %} [ {% if PORT %} {% for port in PORT %} {% if PORT[port].has_key('speed') %} { "CABLE_LENGTH_TABLE: { "{{ port }}": "cable_length(port)" }, "OP": "SET" }{% if not loop.last %},{% endif %} {% endif %} {% endfor %} {% endif %} ]The output of this template is the table like the one shown in section "generic static"
-
vendor/platform (static or generated) for pg lossless
-
non pg-lossless configuration (pools, profiles, mapping)
All static tables from the existing template: ingress/egress lossy/lossless profiles, corresponding pools. -
pg lossless profile tables
"BUFFER_PROFILE_TABLE for pg profiles, port-config to port profile look-up table. If buffer brofiles specified as static tables, they should look like shown in section 3.1.1 for profile "BUFFER_PROFILE_TABLE:pg_lossless_100G_300m_profile". If there are many profiles used for current platform it could be more convenient to store and maintain them in the template and generate json on init. E.g.{%- set pg_profiles = { 'pg_lossless_10G_5m_profile': { 'xon': 18432, 'xoff': 16384, 'size': 34816, 'dynamic_th': 1 }, 'pg_lossless_25G_5m_profile': { 'xon': 18432, 'xoff': 16384, 'size': 34816, 'dynamic_th': 1 }, 'pg_lossless_40G_5m_profile': { 'xon': 18432, 'xoff': 16384, 'size': 34816, 'dynamic_th': 1 }, 'pg_lossless_50G_5m_profile': { 'xon': 18432, 'xoff': 16384, 'size': 34816, 'dynamic_th': 1 }, 'pg_lossless_100G_5m_profile': { 'xon': 18432, 'xoff': 18432, 'size': 36864, 'dynamic_th': 1 }, 'pg_lossless_10G_40m_profile': { 'xon': 18432, 'xoff': 18432, 'size': 36864, 'dynamic_th': 1 }, 'pg_lossless_25G_40m_profile': { 'xon': 18432, 'xoff': 21504, 'size': 39936, 'dynamic_th': 1 }, 'pg_lossless_40G_40m_profile': { 'xon': 18432, 'xoff': 23552, 'size': 41984, 'dynamic_th': 1 }, 'pg_lossless_50G_40m_profile': { 'xon': 18432, 'xoff': 23552, 'size': 41984, 'dynamic_th': 1 }, 'pg_lossless_100G_40m_profile': { 'xon': 18432, 'xoff': 35840, 'size': 54272, 'dynamic_th': 1 }, 'pg_lossless_10G_300m_profile': { 'xon': 18432, 'xoff': 30720, 'size': 49152, 'dynamic_th': 1 }, 'pg_lossless_25G_300m_profile': { 'xon': 18432, 'xoff': 53248, 'size': 71680, 'dynamic_th': 1 }, 'pg_lossless_40G_300m_profile': { 'xon': 18432, 'xoff': 75776, 'size': 94208, 'dynamic_th': 1 }, 'pg_lossless_50G_300m_profile': { 'xon': 18432, 'xoff': 75776, 'size': 94208, 'dynamic_th': 1 }, 'pg_lossless_100G_300m_profile':{ 'xon': 18432, 'xoff': 165888,'size': 184320,'dynamic_th': 1 }, } -%} {# PG profiles declaration #} {% for profile_config in pg_profiles %} { "BUFFER_PROFILE_TABLE:{{ profile_name }}": { "pool":"[BUFFER_POOL_TABLE:ingress_lossless_pg_pool]", "xon":"{{ profile_config['xon'] }}", "xoff":"{{ profile_config['xoff'] }}", "size":"{{ profile_config['size'] }}", "dynamic_th":"{{ profile_config['dynamic_th'] }}" }, "OP": "SET" }, {% endfor -%} -
pg lossless look-up table(s) This table (or tables) holds mapping of port parameters (speed and attached cable length) to th ebuffer profile to be used with these parameters. The simplest implementation of such look-up table looks like this:
{# The key in this table consist of two parts: (port speed)_(cable length) #} { "PG_LOSSLESS_PROFILE_LOOKUP_TABLE:": { '10000_5' : 'pg_lossless_10G_5m_profile', '25000_5' : 'pg_lossless_25G_5m_profile', '40000_5' : 'pg_lossless_40G_5m_profile', '50000_5' : 'pg_lossless_50G_5m_profile', '100000_5' : 'pg_lossless_100G_5m_profile', '10000_40' : 'pg_lossless_10G_40m_profile', '25000_40' : 'pg_lossless_25G_40m_profile', '40000_40' : 'pg_lossless_40G_40m_profile', '50000_40' : 'pg_lossless_50G_40m_profile', '100000_40' : 'pg_lossless_100G_40m_profile', '10000_300' : 'pg_lossless_10G_300m_profile', '25000_300' : 'pg_lossless_25G_300m_profile', '40000_300' : 'pg_lossless_40G_300m_profile', '50000_300' : 'pg_lossless_50G_300m_profile', '100000_300': 'pg_lossless_100G_300m_profile' } "OP": "SET" }
The drawback of the single-table solution shown above is that the key is combined of two values using string concatination. An alternative way to implement this mapping is to use several lookup tables where one of the input keys will be in the table name:
{ "PG_LOSSLESS_PROFILE_LOOKUP_TABLE:5": { '10000' : 'pg_lossless_10G_5m_profile', '25000' : 'pg_lossless_25G_5m_profile', '40000' : 'pg_lossless_40G_5m_profile', ... } "OP": "SET" }, { "PG_LOSSLESS_PROFILE_LOOKUP_TABLE:40": { '10000' : 'pg_lossless_10G_40m_profile', '25000' : 'pg_lossless_25G_40m_profile', '40000' : 'pg_lossless_40G_40m_profile', ... } "OP": "SET" } { "PG_LOSSLESS_PROFILE_LOOKUP_TABLE:300": { '10000' : 'pg_lossless_10G_300m_profile', '25000' : 'pg_lossless_25G_300m_profile', '40000' : 'pg_lossless_40G_300m_profile', ... } "OP": "SET" } -
Logically Buffer Configuration Daemon could be split into two parts.
First part is responsible for:
- handling tables in BUFFERS_DB
- generate or transfer with or without changes tables from BUFFERS_DB to CONFIG_DB
- subscribe to port speed in STATE_DB and update pg lossless related tables in CONFIG_DB
- track usage of buffer profiles (at least - pg lossless) and remove unused from CONFIG_DB
- recalculate and update pool size on profile change
Second part is responsible for the following:
- apply configuration requested in CONFIG_DB on ASIC (already implemented in buffersorch.cpp)
- reflect successfully applied config in STATE_DB
- delete profiles from ASIC and STATE_DB on delete from CONFIG_DB
- handle pool size update in CONFIG_DB
Initial buffers configuration is validated on compile stage by the unit test.
Validations to be implemented:
- no unused pools defined
- all profiles referenced in the look-up table are declared
- all pools referenced by profiles are declared
?