Upgrade Storage Spaces Direct (S2D) from Windows Server 2016 to Windows Server 2019

s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

Upgrade Storage Spaces Direct (S2D) from Windows Server 2016 to Windows Server 2019

How to upgrade a Storage Spaces Direct (S2D) cluster to Windows Server 2019. The four approaches to upgrading a S2D cluster from Windows Server 2016 to Windows Server 2019

• In-place upgrade while VMs are running
• Clean-OS installation while VMs are running
• In-place upgrade while VMs are stopped
• Clean-OS install while VMs are stopped*

The Video shows the "Clean-OS installation while VMs are running" with the use of Host OS boot on VHDX (Dual boot lab systems)

*For the last two Hosts in Part 2 I used the "Clean-OS install while VMs are stopped" for Host S046035 and S046036

• Upgrade a Storage Spaces Direct cluster to Windows Server 2019

The lab system used is a 6 Node S2D Cluster

• List Host Version and the Last Reboot time

After the upgrade enable Cluster Performance History

• Performance history for Storage Spaces Direct

Windows Server 2019 and SR-IOV

s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

How to use single root input/output virtualization (SR-IOV) in Windows Server 2019 with Guest VMs running Windows Server 2012R2, Windows Server 2016 and Windows Server 2019

SR-IOV #3 - Upgrade Drivers for SR-IOV in Windows Server 2012R2 Guest VMs

The Windows Server 2019 Host Network configuration

The demo/lab Host configuration with three physical network interface controller (pNIC)
The use of three pNIC is only to show the diffrent amount of Virtual Functions (VF)

1. Get-VMSwitch
2. Get-VMSwitch | ft Name,NetAdapterInterfaceDescriptions
3. Get-NetAdapter -Name *NIC* | sort name | ft name, InterfaceDescription, LinkSpeed -AutoSize
4. Get-NetAdapterSriov | sort name | ft Name, InterfaceDescription, SriovSupport, NumVFs -AutoSize
5. Note
6. Get-NetAdapterSriovVf | sort name | ft -AutoSize
7. Note

Validate-DCB Disconnected installation

s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

Validator for RDMA Configuration and Best Practices

Validate-DCB Disconnected installation

Validate-DCB Fix 58 errors

More information about installation and review the result for Validate-DCB

• How to review the result from Validate-DCB
• How to do the Installation and first run in 5 minutes with Validate-DCB

More information on GitHub

• Validate-DCB
• WSLab Scripts for Validate-DCB

Validate-DCB Disconnected installation (Step-by-Step)

1. The installation failed do to no internet access (Disconnected)
2. From a Host with Internet access run "Save-Module Validate-DCB -Path C:\Temp"
3. The packages are downloaded
4. The Download complete
5. The six folders are created with all the modules
6. Select and copy the six folders
7. Change back to the disconnected Host, change to the folder "C:\Program Files\WindowsPowerShell\Modules"
8. Paste the files to the folder
9. Files are copied
10. Check that the files and folders are copied
    (Restart PowerShell CLI, to load with the new modules)
11. Now you can run the "Validate-DCB"
12. The prerequisite part "Global Unit" is check
13. If the "Global Unit" pass, the wizard will start and you are ready to add the information about your system
14. For more information about the next step go to the online installation guide [Link]
    (The next part is the same for online and disconnected systems)

Validate-DCB Behind the Scenes

s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

Validator for RDMA Configuration and Best Practices

Validate-DCB - Behind the Scenes

The setup used for the recording of the Validate-DCB Videos

Validate-DCB - Behind the Scenes

Hardware:

• Microsoft Surface Pro 7
• Microsoft Surface Pro 5
• Microsoft Surface Pro 5 LTE
• Microsoft Surface Arc Mouse and Pen

Monitor:

• EIZO 24"
• Beetronics 10"

Camera:

• GoPro HERO8 Black - 4K
• Logitech Brio Ultra HD Pro Webcam - 4K
• iPhone 11 Pro - 4K

Microphone, Speaker and Headphone

• Røde Wireless GO
• Røde Lavalier GO
• Røde VideoMic Me-L
• Røde SC6-L Mobile Interview Kit
• Creative Sound Blaster Play! 3
• Jabra Speak 510
• Jabra Move Wireless

Light/Control

• Elgato Key Light
• Elgato Control Center for Windows
• Philips HUE
• LK IHC (Schneider Electric)

Software:

• Microsoft Windows 10
• Microsoft Garage Mouse without Borders
• TechSmith Camtasia 2019
• Logitech Capture
• Voicemeeter Potato Ultimate Mixer

Part 2

Server Hardware:

• Mellanox SN2100 - 100 GiB (10, 25, 40, 50, 56 and 100 GiB)
• Mellanox SX1012 - 56GiB (10, 40 and 56 GiB)
• Mellanox ConnectX-5
• Mellanox ConnectX-3-Pro VPI
• Dell, Fujitsu, Lenovo, SuperMicro, Cisco and HP Servers

Software:

• Microsoft Windows Server 2016
• Microsoft Windows Server 2019
• Microsoft Windows 10
• Microsoft PowerBI

***

Validator for RDMA Configuration and Best Practices

• Validator for RDMA Configuration and Best Practices
• Part 1
• Part 2
• Part 3
• Part 4
• How to review the result from Validate-DCB

How to review the result from Validate-DCB

s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

Validator for RDMA Configuration and Best Practices

How to review the result from Validate-DCB

• How to do the Installation and first run in 5 minutes with Validate-DCB

More information on GitHub

• Validate-DCB
• WSLab Scripts for Validate-DCB

The results output consists of two unit

1. [Global Unit] Prerequisites and requirements to run the modal tests. 
2. [Modal Unit] Each Cluster Host configuration and best practices feedback

How to review the Error(s)

Example 1: Jumbo Packet Missing from vNIC SMB02

1. The Validate-DCB list one FailedCount
2. Scroll back to we find the red lines. Expect 9014 but the value was 1514 looks like the Jumbo Packet was not Enabled on the vNIC
3. Review the Properties for the vNIC SMB02 on Host S046036, The finding is correct, the Jumbo Packet was Disabled. To fix the Error just enable Jumbo Packet
4. Change the Value to 9014 Bytes
5. Scan the Host S046036 again with Validate-DCB
6. Now the Failed Count is back on 0

Review with PowerBI

Run the Validate-DCB and add the Report Path, copy the XML to the machine with PowerBI
Download the Validate-DCB.pbix fle from GitHub (Link)

1. Run "validate-dcb -ConfigFilePath C:\Temp\S046036.ps1 -ReportPath C:\Temp"
2. Copy the XML to the machine with PowerBI
3. Click on "Validate-DCB.pbix" to open PowerBI
4. Import the XML
   A: Edit Queries
   B: Data source settings
   C: Change Source... Select the XML
5. Review the "Failure" and make the necessary changes

Tilføj billedtekst

List Version and Reboot Time

s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

PowerShell Script to list the Windows Version and Last Reboot Time:

List my three S2D Cluster versions and the last reboot time

“Thanks to Steffen Brandsborg, Sentia Denmark for creating the PowerShell used below”

# Cluster S2D01
 $ComputerList = "S046031","S046032","S046033","S046034","S046035","S046036"
# Cluster S2D02
 $ComputerList += "S047027","S047028"
# Cluster S2D03
 $ComputerList += "S048001","S048002","S048003","S048004"

$InvokeResult = invoke-command -ComputerName $ComputerList -ScriptBlock {
    $BuildBranch = (Get-ItemProperty "HKLM:\Software\Microsoft\Windows NT\CurrentVersion").BuildBranch
    $CurrentVersion = (Get-ItemProperty "HKLM:\Software\Microsoft\Windows NT\CurrentVersion").CurrentVersion
    $CurrentBuildNumber = (Get-ItemProperty "HKLM:\Software\Microsoft\Windows NT\CurrentVersion").CurrentBuildNumber
    $UBR = (Get-ItemProperty "HKLM:\Software\Microsoft\Windows NT\CurrentVersion").UBR
    $BootTime = ((Get-WmiObject Win32_OperatingSystem).ConvertToDateTime((Get-WmiObject Win32_OperatingSystem).LastBootUpTime)).ToString("yyyy-MM-dd HH:mm:ss")
    $ProductName = (Get-ItemProperty "HKLM:\Software\Microsoft\Windows NT\CurrentVersion").ProductName

    $Result =  @{
        Computername = $env:COMPUTERNAME
        ProductName = $ProductName
        WindowsVersion = ("$CurrentVersion.$CurrentBuildNumber.$UBR")
        BootTime = $BootTime
    }
    New-Object -TypeName PSObject -Property $Result
}
$InvokeResult | Sort Computername | Select ComputerName, ProductName, WindowsVersion, BootTime | ft 

Validator for RDMA Configuration and Best Practices

s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

Validator for RDMA Configuration and Best Practices

How to do the Installation and first run in 5 minutes with Validate-DCB

• How to review the result from Validate-DCB

More information on GitHub

• Validate-DCB
• WSLab Scripts for Validate-DCB

Video part 1 to 4

Part 1 - Installation

Part 2

Part 3

Part 4

Installation

• Install-Module Validate-DCB

After installation of the Validate-DCB Module you can run the wizard:

1. Run the Validate-DCB from PowerShell and the wizard will start <Next>
2. Add the Cluster name
3. Resolve
4. The Cluster Nodes are listed
5. Next
6. Add vSwitch Name
7. Add pNIC, vNIC and vlan information
8. Select RDMA type iWARP or RoCE
9. Select the Jumbo Frames (Ethernet MTU size)
   Note: the MTU for RDMA is not the same as Ethernet MTU
   For Mellanox list the Ethernet and RoCE/RDMA MTU with "mlx5cmd.exe -Stat"
   Both the pNIC and the vNIC for SMB need to have the same MTU Size
10. Next
11. Policy Name
12. Bandwidth Reservation
13. Next
14. Save the configuration file
15. Export (the job will start)

RoCE Troubleshooting

s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

RoCE Troubleshooting

The RDMA Activity Performance Counter can give you some useful information

RDMA Accepted Connections will increase for each new connection. The Picture show a lab system with a RoCE Error. RDMA Accepted Connections will normally only change if you move VMs (SOFS), disable/enable pNIC/vNIC or reboot a Node.

The picture show that I have thousands of Accepted Connections and after just a few seconds I have 4 Accepted Connections. With no change in the Cluster.

Active Connections

Show the Active Connections on each vNIC/pNIC to see the connect use the "netstat -xan"

Monitor RoCE - Windows Server

s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

Performance Monitor

Monitor Priority and Pause Frames for RDMA (RoCE) with Mellanox QoS

Examples of Physical Switch configuration for Mellanox, Cisco, HPE, Fujitsu and Dell

For all the physical switch examples, always get your hardware vendor to verify your settings before use.

1. Mellanox
2. Cisco
3. HPE
4. Fujitsu
5. Dell
6. Lenovo

Lab setup:
Storages Spaces Direct (S2D) clusters with 2,4 or 6 nodes, Windows Server 2016 and Windows Server 2019

The Host and Switch need to be configured for DCB/PFC/ETS

Note: Microsoft does not currently support DCBx/LLDP if you use SET. You need to be sure that the Switch is not set to "PFC Auto" on the used ports. But use "PFC Enable" or "PFC On"

Use the Microsoft Deployment Guide and Validate DCB Scripts for Host validation

• Windows Server 2016 and 2019 RDMA Deployment Guide
• Validate DCB

In the Examples we use Priority 3 for Storage and Priority 7 for Cluster Traffic.

• Storage Traffic = SMB 445 Traffic (Storage, CSV and Live Migration)
• Storage Traffic need PFC/ETS
• Cluster Traffic = UDP 3343 Traffic
• Cluster Traffic only use the ETS

In the lab I use Jumbo Frames for Ethernet and auto size for RDMA.

(RDMA/RoCE Max Frame Size is in Auto Mode).

Sent Pause Frames	The total number of pause frames sent from this priority to the far-end port. The untagged instance indicates the number of global pause frames that were sent.
Sent Pause Duration	The total duration of packets transmission being paused on this priority in microseconds.
Received Pause Frames	The number of pause frames that were received to this priority from the far-end port. The untagged instance indicates the number of global pause frames that were received.
Received Pause Duration	The total duration that far-end port was requested to pause for the transmission of packets in microseconds.
Sent Discard Frames	The number of packets discarded by the transmitter.  Note: this counter is per Traffic Class (TC) and not per priority.

Performance Monitor:
The pictures below shows the Priority 3 (SMB) and also Priority 7 (Cluster)
(There need to be high load on the Physical Switch Port and/or Network Adapter before you will see that values change for Pause Frames).

New-NetQosPolicy "Cluster" -Cluster -PriorityValue8021Action 7
New-NetQosPolicy "SMB" -NetDirectPortMatchCondition 445 -PriorityValue8021Action 3
New-NetQosPolicy "DEFAULT" -Default -PriorityValue8021Action 0

Set-VMNetworkAdapter -ManagementOS -Name "SMB1" -IeeePriorityTag on
Set-VMNetworkAdapter -ManagementOS -Name "SMB2" -IeeePriorityTag on

Performance Monitor:
The pictures below shows the Priority 3 (SMB) and Pause Frames (Rcv/Sent) on the pNICs with 10 minuttes between each other. We can see that the Frames increased, so the system sent and recived it correct. (There need to be high load on the Physical Switch Port and/or Network Adapter before you will see that values change).

Monitor RoCE - Mellanox

s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

Performance Monitor
Monitor Priority and Pause Frames for RDMA (RoCE) on Mellanox Switch.

Microsoft references:

• Windows Server 2016 and 2019 RDMA Deployment Guide
• Validate DCB

For the screen captures I use two different Mellanox Switch.

• Mellanox SX1012 (SwitchX - Onyx 3.6.8010)
• Mellanox SN2100 (Spectrum - Onyx 3.6.8010)
• The SwitchX and Spectrum are configured in different ways when we look on L2 ETS
• The Spectrum also support to use L3 (DSCP) Note: Microsoft configuration guide use RoCE Layer 2 (L2) for S2D that use SET (NDKm2)

******************************************************************************
Mellanox Show Commands
******************************************************************************

S2D# show dcb priority-flow-control detail
S2D# show dcb priority-flow-control
S2D# show dcb priority-flow-control interface ethernet 1/1
S2D# show dcb ets
S2D# show dcb ets interface ethernet 1/1


References

Microsoft:
Windows Server 2016 and 2019 RDMA Deployment Guide
Validate DCB

Mellanox L2 PCP TC:
How to Install Windows Server 2016 with RoCEv2 and Switch Embedded Teaming over HA Mellanox Network Solution
Understanding QoS Classification (Trust) on Spectrum Switche
Understanding Traffic Class (TC) Scheduling on Mellanox Spectrum Switches (WRR,SP)
Understanding RoCEv2 Congestion Management
DCBX Versions and Support on Mellanox  Ethernet Switches
HowTo Configure PFC on ConnectX-4

Mellanox L3 - DSCP:
Recommended Network Configuration Examples for RoCE Deployment
Lossless RoCE Configuration for MLNX-OS Switches in DSCP-Based QoS Mode
Lossless RoCE Configuration for MLNX-OS Switches in DSCP-Based QoS Mode (advanced mode)
Lossless RoCE Configuration for WinOF2 driver in DSCP-based QoS mode
HowTo Configure Mellanox Spectrum Switch for Lossless RoCE
HowTo Configure ECN on Mellanox Ethernet Switches (Spectrum)

Terminology:
pNIC Physical NIC, the physical hardware that exchanges packets with the TOR
vNIC    Host vNIC – Virtual NIC from vSwitch exposed in the host partition
tNIC Host tNIC - Team Interface NIC from LBFO Team 
vmNIC Virtual Machine NIC – Virtual NIC from vSwitch exposed in a guest partition
vSwitch Hyper-V virtual switch
SET Switch Embedded Teaming, Hyper-V virtual switch supported in Windows Server 2016 and 2019

ToR Top of Rack switch
RDMA Remote Direct Memory Access
RoCE    RDMA over Converged Ethernet
RoCEv2 2nd generation RoCE using UDP/IP for routability (a.k.a. Routable RoCE)

DCB     Data Center Bridging
LLDP    Link Layer Data Protocol
DCBx    Data Center Bridging Capability Exchange protocol (DCBX) is an extension of LLDP.
PFC     Priority Flow Control
ETS     Enhanced Transmission Service
TC      Traffic Class
ECN     Explicit Congestion Notification
RED     Random Early Detection
CNP     Congestion Notification Packet. CNP control frames (congestion ACK)
SP      Strict Priority
WRR     Weighted Round Robin


******************************************************************************
Mellanox SX1012 (SwitchX)
******************************************************************************
S2D# show dcb priority-flow-control detail

PFC enabled
Priority Enabled List    :3
Priority Disabled List   :0 1 2 4 5 6 7

PFC Port Eth1/x            Information
-----------------------------------------------
PFC Port Mode       :On
PFC Oper State      :On
No Remote Entry is Present
-----------------------------------------------

******************************************************************************

S2D# show dcb priority-flow-control

PFC enabled
Priority Enabled List    :3
Priority Disabled List   :0 1 2 4 5 6 7

TC     Lossless
---    ----------
0           N
1           Y
2           Y
3           N

Interface      PFC admin        PFC oper
------------   --------------   -------------
Eth1/1           On               Enabled
Eth1/2           On               Enabled
Eth1/3           On               Enabled
Eth1/4           On               Enabled
Eth1/5           On               Enabled
Eth1/6           On               Enabled
Eth1/7           On               Enabled
Eth1/8           On               Enabled
Eth1/9           On               Enabled
Eth1/10          On               Enabled
Eth1/11          On               Enabled
Eth1/12          On               Enabled


******************************************************************************

S2D# show dcb priority-flow-control interface ethernet 1/1

PFC enabled
Priority Enabled List    :3
Priority Disabled List   :0 1 2 4 5 6 7

TC     Lossless
---    ----------
0           N
1           Y
2           Y
3           N

Interface      PFC admin        PFC oper
------------   --------------   -------------
Eth1/1           On               Enabled

******************************************************************************

S2D# show dcb ets
(Note: Default value before change)

ETS enabled

TC        Bandwidth
--------------------------
0         25%
1         25%
2         25%
3         25%

Number of Traffic Class: 4

******************************************************************************

S2D# show dcb ets interface ethernet 1/1
(Note: Default value before change)

ETS Port Mode             :AUTO MODE
ETS Oper State            :INIT STATE
ETS State Machine Type    :Assymetric
-----------------------------------------------
ETS Local Port Info
-----------------------------------------------
TC bandwidth table
-----------------------------------------------
TC        Bandwidth        RecomBandwidth
-----------------------------------------------
0         25%              25%
1         25%              25%
2         25%              25%
3         25%              25%

priority assignment table
--------------------------------------
Priority     TC
--------------------------------------
0            0
1            0
2            1
3            1
4            2
5            2
6            3
7            3

Number of Traffic Class: 4

Willing Status:  Disable
-----------------------------------------------
ETS Admin Port Info
-----------------------------------------------
TC        Bandwidth        RecomBandwidth
-----------------------------------------------
0         25%              25%
1         25%              25%
2         25%              25%
3         25%              25%
-----------------------------------------------
ETS Remote Port Info
-----------------------------------------------
No Remote Entry is Present
-----------------------------------------------

******************************************************************************
Configuration of SX1012 (SwitchX)
******************************************************************************

Mapping of priority to traffic classes (TC)
Priority 0 and 1 mapped to TC 0
Priority 2 and 3 mapped to TC 1
Priority 4 and 5 mapped to TC 2
Priority 6 and 7 mapped to TC 3

TC 0 and TC 3 are lossy TCs.
TC 1 and TC 2 are lossless TCs.

For the PFC enabled priorities we need to use the lossless TCs

So for the Microsoft Example use case we change the default to:
Priority 0 Default traffic 39%
Priority 3 SMB traffic 60%
Priority 5 Cluster traffic 1%

S2D (config)# dcb ets tc bandwidth 39 60 1 0

******************************************************************************

S2D# show dcb ets interface ethernet 1/1
(Note: Value after change)

S2D# sh dcb ets interface ethernet 1/1

ETS Port Mode             :AUTO MODE
ETS Oper State            :INIT STATE
ETS State Machine Type    :Assymetric
-----------------------------------------------
ETS Local Port Info
-----------------------------------------------
TC bandwidth table
-----------------------------------------------
TC        Bandwidth        RecomBandwidth
-----------------------------------------------
0         39%              39%
1         60%              60%
2          1%               1%
3          0%               0%

priority assignment table
--------------------------------------
Priority     TC
--------------------------------------
0            0
1            0
2            1
3            1
4            2
5            2
6            3
7            3

Number of Traffic Class: 4

Willing Status:  Disable
-----------------------------------------------
ETS Admin Port Info
-----------------------------------------------
TC        Bandwidth        RecomBandwidth
-----------------------------------------------
0         39%              39%
1         60%              60%
2          1%               1%
3          0%               0%
-----------------------------------------------
ETS Remote Port Info
-----------------------------------------------
No Remote Entry is Present
-----------------------------------------------

******************************************************************************
Mellanox SN-2100 Spectrum
******************************************************************************

S2D# show dcb ets interface ethernet 1/1
(Note: Default value before change)

Eth1/1
 Interface Bandwidth Shape [Mbps]: N/A
 Multicast unaware mapping : disabled

 ETS per TC :
 TC Scheduling Mode Weight Weight (%)
 -- --------------- ------ ----------
 0  WRR             12     12
 1  WRR             13     13
 2  WRR             12     12
 3  WRR             13     13
 4  WRR             12     12
 5  WRR             13     13
 6  WRR             12     12
 7  WRR             13     13

 Bandwidth Shape per TC:
 TC Bandwidth Shape [Mbps]
 -- ----------------------
 0  N/A
 1  N/A
 2  N/A
 3  N/A
 4  N/A
 5  N/A
 6  N/A
 7  N/A

 Bandwidth Guarantee per TC:
 TC Bandwidth Guaranteed [Mbps]
 -- ---------------------------
 0  0
 1  0
 2  0
 3  0
 4  0
 5  0
 6  0
 7  0

 Switch Priority to TC mapping:
 Switch Priority TC
 --------------- --
 0               0
 1               1
 2               2
 3               3
 4               4
 5               5
 6               6
 7               7

******************************************************************************

******************************************************************************
Mellanox Switch configuration example (L2):
******************************************************************************

Mellanox SN2100 (Spectrum)

S2D> enable
S2D# configure terminal 
S2D (Config)# interface ethernet 1/1-1/12 traffic-class 3 dcb ets Strict
S2D (Config)# interface ethernet 1/1-1/12 traffic-class 7 dcb ets Strict

S2D (Config)# interface ethernet 1/1-1/x flowcontrol send off force
S2D (Config)# interface ethernet 1/1-1/x flowcontrol receive off force

S2D (Config)# dcb priority-flow-control enable force
S2D (Config)# dcb priority-flow-control priority 3 enable
(Priority 3 is used for Storage (SMB) traffic)
S2D (Config)# interface ethernet 1/1-1/x dcb priority-flow-control mode on force
S2D (Config)# interface ethernet 1/x switchport mode hybrid
(Need to be repeated for each port from 1 to 12/16)
S2D# interface ethernet 1/x switchport hybrid allowed-vlan all
Need to be repeated for each port from 1 to 12/16. Only allow the needed vlans, the "allowed-vlan all" is not recommended, this is from a lab/test system).
S2D (Config)# exit
S2D# write memory

******************************************************************************
S2D# show dcb ets interface ethernet 1/1
Eth1/1
 Interface Bandwidth Shape [Mbps]: N/A
 Multicast unaware mapping : disabled

 ETS per TC :
 TC Scheduling Mode Weight Weight (%)
 -- --------------- ------ ----------
 0  WRR             12     12
 1  WRR             13     13
 2  WRR             12     12
 3  Strict          0      0
 4  WRR             12     12
 5  WRR             13     13
 6  WRR             12     12
 7  Strict          0      0

 Bandwidth Shape per TC:
 TC Bandwidth Shape [Mbps]
 -- ----------------------
 0  N/A
 1  N/A
 2  N/A
 3  N/A
 4  N/A
 5  N/A
 6  N/A
 7  N/A

 Bandwidth Guarantee per TC:
 TC Bandwidth Guaranteed [Mbps]
 -- ---------------------------
 0  0
 1  0
 2  0
 3  0
 4  0
 5  0
 6  0
 7  0

 Switch Priority to TC mapping:
 Switch Priority TC
 --------------- --
 0               0
 1               1
 2               2
 3               3
 4               4
 5               5
 6               6
 7               7

******************************************************************************
Mellanox Switch configuration example with DSCP:

Draft - Test only last update 2019.01.19
Note:
#1: The example use both L2/L3. SwitchX use L2 and Spectrum use both L2/L3
#2: Microsoft configuration guide use RoCE Layer 2 (L2) for S2D.
#3: DSCP is supported by Windows Server OS. However to use it for S2D is not recommended right now by Microsoft and for test you need to work close with the Hardware vendors. 

Lossless RoCE Configuration for MLNX-OS Switches in DSCP-Based QoS Mode
Lossless RoCE Configuration for MLNX-OS Switches in DSCP-Based QoS Mode (Old version)
******************************************************************************

S2D (Config)# interface ethernet 1/1-1/x qos trust ?

       port       based on port default settings
       L2         based on PCP, DEI fields
       L3         based on EXP, DSCP fields
       both       based on PCP, DEI and EXP, DSCP fields

S2D (Config)# interface ethernet 1/1-1/x qos trust both
(Note: In my lab I used both SwitchX and Spectrum. The Spectrum Switch need to accepts PCP and DSCP in my example)

******************************************************************************

S2D (config) # interface ethernet 1/1-1/x traffic-class 3 congestion-control ecn minimum-absolute 150 maximum-absolute 1500
S2D (config) # traffic pool roce type lossless
S2D (config) # traffic pool roce memory percent 50.00
S2D (config) # traffic pool roce map switch-priority 3
S2D (config) # interface ethernet 1/1-1/x traffic-class 3 dcb ets strict
S2D (config) # interface ethernet 1/1-1/x traffic-class 7 dcb ets strict
S2D (config) # interface ethernet 1/1-1/x qos trust both

******************************************************************************

S2D (Config)# interface ethernet 1/1-1/x traffic-class 0 bind switch-priority 0
S2D (Config)# interface ethernet 1/1-1/x traffic-class 3 bind switch-priority 3
S2D (Config)# interface ethernet 1/1-1/x traffic-class 7 bind switch-priority 7

******************************************************************************

S2D (Config)# interface ethernet 1/1-1/x flowcontrol send off force
S2D (Config)# interface ethernet 1/1-1/x flowcontrol receive off force

S2D (Config)# dcb priority-flow-control enable force
S2D (Config)# dcb priority-flow-control priority 3 enable
(Priority 3 is used for Storage (SMB) traffic)
S2D (Config)# interface ethernet 1/1-1/x dcb priority-flow-control mode on force
S2D (Config)# interface ethernet 1/x switchport mode hybrid
(Need to be repeated for each port from 1 to x)
S2D (Config)# interface ethernet 1/x switchport hybrid allowed-vlan all
Need to be repeated for each port from 1 to x. Only allow the needed vlans, the "allowed-vlan all" is not recommended, this is from a lab/test system).

******************************************************************************
Host pNIC configuration

Driver version 2.10 or newer

S2D PS> Mlx5Cmd.exe -QosConfig -SetupRoceQosConfig -Name NIC3 -Configure 2

Default DSCP to switch-priority mapping:
0-7 → 0
8-15 → 1
16-23 → 2
24-31 → 3
32-39 → 4
40-47 → 5
48-55 → 6
56-63 → 7

******************************************************************************