*** Disclaimer ***
s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment.
For any reference links to other websites we encourages you to read the privacy statements of the third-party websites.
The names of actual companies and products mentioned herein may be the trademarks of their respective owners.
Monitor Priority and Pause Frames for RDMA (RoCE) with Mellanox QoS
Examples of Physical Switch configuration for Mellanox, Cisco, HPE, Fujitsu and Dell
For all the physical switch examples, always get your hardware vendor to verify your settings before use.
Storages Spaces Direct (S2D) clusters with 2,4 or 6 nodes, Windows Server 2016 and Windows Server 2019
The Host and Switch need to be configured for DCB/PFC/ETS
Note: Microsoft does not currently support DCBx/LLDP if you use SET. You need to be sure that the Switch is not set to "PFC Auto" on the used ports. But use "PFC Enable" or "PFC On"
Use the Microsoft Deployment Guide and Validate DCB Scripts for Host validation
In the Examples we use Priority 3 for Storage and Priority 7 for Cluster Traffic.
- Storage Traffic = SMB 445 Traffic (Storage, CSV and Live Migration)
- Storage Traffic need PFC/ETS
- Cluster Traffic = UDP 3343 Traffic
- Cluster Traffic only use the ETS
In the lab I use Jumbo Frames for Ethernet and auto size for RDMA.(RDMA/RoCE Max Frame Size is in Auto Mode).
Sent Pause Frames
The total number of pause frames sent from this priority to the far-end port. The untagged instance indicates the number of global pause frames that were sent.
Sent Pause Duration
The total duration of packets transmission being paused on this priority in microseconds.
Received Pause Frames
The number of pause frames that were received to this priority from the far-end port. The untagged instance indicates the number of global pause frames that were received.
Received Pause Duration
The total duration that far-end port was requested to pause for the transmission of packets in microseconds.
Note: 1 second is equal to 1000000 microsecond.
Sent Discard Frames
The number of packets discarded by the transmitter. Note: this counter is per Traffic Class (TC) and not per priority.
The pictures below shows the Priority 3 (SMB) and also Priority 7 (Cluster)
(There need to be high load on the Physical Switch Port and/or Network Adapter before you will see that values change for Pause Frames).
New-NetQosPolicy "Cluster" -Cluster -PriorityValue8021Action 7
New-NetQosPolicy "SMB" -NetDirectPortMatchCondition 445 -PriorityValue8021Action 3
New-NetQosPolicy "DEFAULT" -Default -PriorityValue8021Action 0
Set-VMNetworkAdapter -ManagementOS -Name "SMB1" -IeeePriorityTag on
Set-VMNetworkAdapter -ManagementOS -Name "SMB2" -IeeePriorityTag on
The pictures below shows the Priority 3 (SMB) and Pause Frames (Rcv/Sent) on the pNICs with 10 minuttes between each other. We can see that the Frames increased, so the system sent and recived it correct. (There need to be high load on the Physical Switch Port and/or Network Adapter before you will see that values change).