Skip to content

Commit 51df513

Browse files
r12fr12f
authored
[doc] Add metering HLD (#544)
1 parent 8b7a0d5 commit 51df513

File tree

2 files changed

+163
-0
lines changed

2 files changed

+163
-0
lines changed

‎documentation/metering/README.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# DASH Metering Documents
2+
3+
This folder contains DASH Metering design and requirements documents.
4+
5+
## Design
6+
7+
| Document | Description |
8+
| --- | --- |
9+
| [Metering](metering.md) | Metering design |

‎documentation/metering/metering.md

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
# DASH Metering HLD
2+
3+
| Rev | Date | Author | Change Description |
4+
| --- | ---- | ------ | ------------------ |
5+
| 0.1 | 03/30/2024 | Riff Jiang | Initial version |
6+
| 0.2 | 04/03/2024 | Riff Jiang | Updated meter class from 16 bits to 32 bits. Added capability for meter bucket. |
7+
8+
1. [1. Background](#1-background)
9+
2. [2. Resource modeling, requirement, and SLA](#2-resource-modeling-requirement-and-sla)
10+
1. [2.1. Resource modeling](#21-resource-modeling)
11+
2. [2.2. Scaling requirement](#22-scaling-requirement)
12+
3. [2.3. Reliability requirements](#23-reliability-requirements)
13+
3. [3. SAI API design](#3-sai-api-design)
14+
1. [3.1. Meter bucket](#31-meter-bucket)
15+
2. [3.2. Policy-based meter attributes](#32-policy-based-meter-attributes)
16+
1. [3.2.1. Route attributes](#321-route-attributes)
17+
2. [3.2.2. Mapping attributes](#322-mapping-attributes)
18+
3. [3.2.3. Tunnel attributes](#323-tunnel-attributes)
19+
4. [3.2.4. Inbound route rule attributes](#324-inbound-route-rule-attributes)
20+
3. [3.3. Global meter policy](#33-global-meter-policy)
21+
1. [3.3.1. ENI](#331-eni)
22+
2. [3.3.2. Meter policy](#332-meter-policy)
23+
3. [3.3.3. Meter rule](#333-meter-rule)
24+
4. [3.4. Capability](#34-capability)
25+
4. [4. Metering bucket selection in DASH pipeline](#4-metering-bucket-selection-in-dash-pipeline)
26+
27+
## 1. Background
28+
29+
To support billing, DASH introduced metering related objects as traffic counters. These counters are only used for billing purposes and not related to traffic policer or shaping.
30+
31+
## 2. Resource modeling, requirement, and SLA
32+
33+
### 2.1. Resource modeling
34+
35+
- Each ENI will allocates a set of metering bucket for billing purposes.
36+
- Metering bucket is indexed by a UINT32 number called metering class, which starts from 1. Meter class 0 will be reversed and considered as not set.
37+
- Each metering bucket will contain 1 inbound counter and 1 outbound counter for at least counting bytes.
38+
- Metering bucket shall reflect the traffic volume of the customer. This means:
39+
- It shall only count the size of the overlay packet.
40+
- On the outbound direction, it shall count the packets before the SDN transformation.
41+
- On the inbound direction, it shall count the packets after the SDN transformation.
42+
43+
### 2.2. Scaling requirement
44+
45+
The scaling requirement for metering are listed as below:
46+
47+
| Metric | Requirement |
48+
| --- | --- |
49+
| # of meter buckets per ENI | 4095 (2^12 – 1, 0 is considered as not set) |
50+
| # of meter rules per meter policy | (TBD) |
51+
52+
### 2.3. Reliability requirements
53+
54+
In HA setup, the metering info should be stored as part of flow and replicated to the standby side. Whenever the primary failover, the metering class id should be still the same for each flow.
55+
56+
The high level flow replication follows the same approach as the SmartSwitch HA design, hence omitted here.
57+
58+
## 3. SAI API design
59+
60+
The following attributes will be involved in determining the final metering buckets in DASH.
61+
62+
### 3.1. Meter bucket
63+
64+
| Attribute | Type | Default Value | Description |
65+
| --- | --- | --- | --- |
66+
| SAI_METER_BUCKET_ATTR_ENI_ID | `sai_object_id_t` | SAI_NULL_OBJECT_ID | ENI ID of this metering class. |
67+
| SAI_METER_BUCKET_ATTR_METER_CLASS | `sai_uint32_t` | 0 | Meter class of this meter bucket. |
68+
69+
To fetch the metering data from each meter bucket, we are going to leverage the SAI stats APIs, which provides get, get and clear and other frequently used semantics. It will also reduce the work in SONiC stack, as SONiC already have good support over the stats APIs.
70+
71+
| Attribute | Description |
72+
| --- | --- |
73+
| SAI_METER_BUCKET_STAT_OUTBOUND_BYTES | Total outbound traffic in bytes. |
74+
| SAI_METER_BUCKET_STAT_INBOUND_BYTES | Total inbound traffic in bytes. |
75+
76+
### 3.2. Policy-based meter attributes
77+
78+
#### 3.2.1. Route attributes
79+
80+
| Attribute | Type | Default Value | Description |
81+
| --- | --- | --- | --- |
82+
| SAI_OUTBOUND_ROUTING_ENTRY_ATTR_METER_CLASS_OR | `sai_uint32_t` | 0 | Meter class OR bits. |
83+
| SAI_OUTBOUND_ROUTING_ENTRY_ATTR_METER_CLASS_AND | `sai_uint32_t` | UINT32_MAX | Meter class AND bits. |
84+
85+
#### 3.2.2. Mapping attributes
86+
87+
| Attribute | Type | Default Value | Description |
88+
| --- | --- | --- | --- |
89+
| SAI_OUTBOUND_CA_TO_PA_ENTRY_ATTR_METER_CLASS_OR | `sai_uint32_t` | 0 | Meter class OR bits. |
90+
91+
#### 3.2.3. Tunnel attributes
92+
93+
| Attribute | Type | Default Value | Description |
94+
| --- | --- | --- | --- |
95+
| SAI_DASH_TUNNEL_ATTR_METER_CLASS_OR | `sai_uint32_t` | 0 | Meter class OR bits. |
96+
97+
#### 3.2.4. Inbound route rule attributes
98+
99+
| Attribute | Type | Default Value | Description |
100+
| --- | --- | --- | --- |
101+
| SAI_INBOUND_ROUTING_ENTRY_ATTR_METER_CLASS_OR | `sai_uint32_t` | 0 | Meter class OR bits. |
102+
| SAI_INBOUND_ROUTING_ENTRY_ATTR_METER_CLASS_AND | `sai_uint32_t` | UINT32_MAX | Meter class AND bits. |
103+
104+
### 3.3. Global meter policy
105+
106+
#### 3.3.1. ENI
107+
108+
| Attribute | Type | Default Value | Description |
109+
| --- | --- | --- | --- |
110+
| SAI_ENI_ATTR_V4_METER_POLICY_ID | `sai_object_id_t` | SAI_NULL_OBJECT_ID | Global IPv4 meter policy ID for this ENI. |
111+
| SAI_ENI_ATTR_V6_METER_POLICY_ID | `sai_object_id_t` | SAI_NULL_OBJECT_ID | Global IPv6 meter policy ID for this ENI. |
112+
113+
#### 3.3.2. Meter policy
114+
115+
| Attribute | Type | Default Value | Description |
116+
| --- | --- | --- | --- |
117+
| SAI_METER_POLICY_ATTR_IP_ADDR_FAMILY | `sai_ip_addr_family_t` | SAI_IP_ADDR_FAMILY_IPV4 | IP address family of the metering policy |
118+
119+
#### 3.3.3. Meter rule
120+
121+
| Attribute | Type | Default Value | Description |
122+
| --- | --- | --- | --- |
123+
| SAI_METER_RULE_ATTR_METER_POLICY_ID | `sai_object_id_t` | SAI_NULL_OBJECT_ID | Meter policy ID of this meter rule. |
124+
| SAI_METER_RULE_ATTR_METER_CLASS | `sai_uint32_t` | UINT32_MAX | Meter class when this meter rule is hit. |
125+
| SAI_METER_RULE_ATTR_DIP | `sai_ip_address_t` | NA | Destination IP for ternary match. |
126+
| SAI_METER_RULE_ATTR_DIP_MASK | `sai_ip_address_t` | NA | Destination IP mask for ternary match. |
127+
| SAI_METER_RULE_ATTR_PRIORITY | `sai_uint32_t` | NA | Priority required for ternary match. |
128+
129+
### 3.4. Capability
130+
131+
To enable the DASH providers be able to tell the host how much metering buckets are supported, we are going to introduce a new capability attributes:
132+
133+
| Attribute | Type | Description |
134+
| --- | --- | --- |
135+
| SAI_SWITCH_ATTR_DASH_CAPS_MAX_METER_BUCKET_COUNT_PER_ENI | `sai_uint32_t` | Max number of meter buckets per ENI supported by the DASH implementation. |
136+
137+
## 4. Metering bucket selection in DASH pipeline
138+
139+
In DASH, the packet shall be metered following the approach below.
140+
141+
When a packet arrives at an ENI, it will go through the steps below to find its metering bucket:
142+
143+
1. **Init**: `AggregatedMeterClassOR` = 0, `AggregatedMeterClassAND` = UINT32_MAX.
144+
2. **Conntrack Lookup**: In Conntrack Lookup stage, if a valid flow is hit, the meter class stored in that flow shall be used for locating the metering bucket.
145+
3. **Policy match stages (Routing/Mapping)**: When flow is missing, the packet will go to slow path and walk through all the policy match stages. Depends on the stage it goes through, it will pick up the meter class OR bits and AND bits, and these 2 bits shall be aggregated separately:
146+
1. `AggregatedMeterClassOR` = `AggregatedMeterClassOR` | `MeterClassOR`
147+
2. `AggregatedMeterClassAND` = `AggregatedMeterClassAND` & `MeterClassAND`
148+
4. **After policy match stages**: Now we calculates the meter class as below:
149+
1. `MeterClass` = `AggregatedMeterClassOR` & `AggregatedMeterClassAND`
150+
2. This allows us to use the info from routing entry to override the some meter class bits set in the mapping.
151+
5. **Metering**: If `MeterClass` is 0 at this moment, meter policy will be used for determining which meter class shall be used:
152+
1. Meter policy v4 or v6 will be selected based on the IP family of the original overlay packet.
153+
2. The overlay destination (outbound pipeline) / source (inbound pipeline) IP will be used for ternary match against the meter rules in the meter policy to find the meter class.
154+
6. **Meter Update**: The final meter class will be used for update the counters in meter bucket. If final meter class is 0, no meter bucket will be updated.

0 commit comments

Comments
 (0)