Featured Post

Links library

2019/11/12

Upgrade Storage Spaces Direct (S2D) from Windows Server 2016 to Windows Server 2019

s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

Upgrade Storage Spaces Direct (S2D) from Windows Server 2016 to Windows Server 2019

How to upgrade a Storage Spaces Direct (S2D) cluster to Windows Server 2019. The four approaches to upgrading a S2D cluster from Windows Server 2016 to Windows Server 2019
  • In-place upgrade while VMs are running
  • Clean-OS installation while VMs are running
  • In-place upgrade while VMs are stopped
  • Clean-OS install while VMs are stopped*
The Video shows the "Clean-OS installation while VMs are running" with the use of Host OS boot on VHDX (Dual boot lab systems)



*For the last two Hosts in Part 2 I used the "Clean-OS install while VMs are stopped" for Host S046035 and S046036





The lab system used is a 6 Node S2D Cluster



After the upgrade enable Cluster Performance History



















2019/11/11

Windows Server 2019 and SR-IOV

s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

How to use single root input/output virtualization (SR-IOV) in Windows Server 2019 with Guest VMs running Windows Server 2012R2, Windows Server 2016 and Windows Server 2019

SR-IOV #3 - Upgrade Drivers for SR-IOV in Windows Server 2012R2 Guest VMs



The Windows Server 2019 Host Network configuration

The demo/lab Host configuration with three physical network interface controller (pNIC)
The use of three pNIC is only to show the diffrent amount of Virtual Functions (VF)
  1. Get-VMSwitch
  2. Get-VMSwitch | ft Name,NetAdapterInterfaceDescriptions
  3. Get-NetAdapter -Name *NIC* | sort name | ft name, InterfaceDescription, LinkSpeed -AutoSize
  4. Get-NetAdapterSriov | sort name | ft Name, InterfaceDescription, SriovSupport, NumVFs -AutoSize
  5. Note
  6. Get-NetAdapterSriovVf | sort name | ft -AutoSize
  7. Note





2019/10/30

Validate-DCB Disconnected installation

s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

Validator for RDMA Configuration and Best Practices


Validate-DCB Disconnected installation




Validate-DCB Fix 58 errors




More information about installation and review the result for Validate-DCB
More information on GitHub

Validate-DCB Disconnected installation (Step-by-Step)
  1. The installation failed do to no internet access (Disconnected)
  2. From a Host with Internet access run "Save-Module Validate-DCB -Path C:\Temp"
  3. The packages are downloaded
  4. The Download complete
  5. The six folders are created with all the modules
  6. Select and copy the six folders
  7. Change back to the disconnected Host, change to the folder "C:\Program Files\WindowsPowerShell\Modules"
  8. Paste the files to the folder
  9. Files are copied
  10. Check that the files and folders are copied
    (Restart PowerShell CLI, to load with the new modules)
  11. Now you can run the "Validate-DCB"
  12. The prerequisite part "Global Unit" is check
  13. If the "Global Unit" pass, the wizard will start and you are ready to add the information about your system
  14. For more information about the next step go to the online installation guide [Link]
    (The next part is the same for online and disconnected systems)








                          2019/10/27

                          Validate-DCB Behind the Scenes

                          s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

                          Validator for RDMA Configuration and Best Practices


                          Validate-DCB - Behind the Scenes

                          The setup used for the recording of the Validate-DCB Videos



                          Validate-DCB - Behind the Scenes

                          Hardware:
                          • Microsoft Surface Pro 7
                          • Microsoft Surface Pro 5
                          • Microsoft Surface Pro 5 LTE
                          • Microsoft Surface Arc Mouse and Pen
                          Monitor:
                          • EIZO 24"
                          • Beetronics 10"
                          Camera:
                          • GoPro HERO8 Black - 4K
                          • Logitech Brio Ultra HD Pro Webcam - 4K
                          • iPhone 11 Pro - 4K
                          Microphone, Speaker and Headphone
                          • Røde Wireless GO
                          • Røde Lavalier GO
                          • Røde VideoMic Me-L
                          • Røde SC6-L Mobile Interview Kit
                          • Creative Sound Blaster Play! 3
                          • Jabra Speak 510
                          • Jabra Move Wireless
                          Light/Control
                          • Elgato Key Light
                          • Elgato Control Center for Windows
                          • Philips HUE
                          • LK IHC (Schneider Electric)
                          Software:
                          • Microsoft Windows 10
                          • Microsoft Garage Mouse without Borders
                          • TechSmith Camtasia 2019
                          • Logitech Capture
                          • Voicemeeter Potato Ultimate Mixer

                          Part 2

                          Server Hardware:
                          • Mellanox SN2100 - 100 GiB (10, 25, 40, 50, 56 and 100 GiB)
                          • Mellanox SX1012 - 56GiB (10, 40 and 56 GiB)
                          • Mellanox ConnectX-5
                          • Mellanox ConnectX-3-Pro VPI
                          • Dell, Fujitsu, Lenovo, SuperMicro, Cisco and HP Servers
                          Software:
                          • Microsoft Windows Server 2016
                          • Microsoft Windows Server 2019
                          • Microsoft Windows 10
                          • Microsoft PowerBI
                          ***

                          Validator for RDMA Configuration and Best Practices




                          2019/10/25

                          How to review the result from Validate-DCB

                          s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

                          Validator for RDMA Configuration and Best Practices


                          How to review the result from Validate-DCB


                          More information on GitHub
                          The results output consists of two unit
                          1. [Global Unit] Prerequisites and requirements to run the modal tests. 
                          2. [Modal Unit] Each Cluster Host configuration and best practices feedback




                          How to review the Error(s)

                          Example 1: Jumbo Packet Missing from vNIC SMB02
                          1. The Validate-DCB list one FailedCount
                          2. Scroll back to we find the red lines. Expect 9014 but the value was 1514 looks like the Jumbo Packet was not Enabled on the vNIC
                          3. Review the Properties for the vNIC SMB02 on Host S046036, The finding is correct, the Jumbo Packet was Disabled. To fix the Error just enable Jumbo Packet
                          4. Change the Value to 9014 Bytes
                          5. Scan the Host S046036 again with Validate-DCB
                          6. Now the Failed Count is back on 0


                          Review with PowerBI

                          Run the Validate-DCB and add the Report Path, copy the XML to the machine with PowerBI
                          Download the Validate-DCB.pbix fle from GitHub (Link)
                          1. Run "validate-dcb -ConfigFilePath C:\Temp\S046036.ps1 -ReportPath C:\Temp"
                          2. Copy the XML to the machine with PowerBI
                          3. Click on "Validate-DCB.pbix" to open PowerBI
                          4. Import the XML
                            A: Edit Queries
                            B: Data source settings
                            C: Change Source... Select the XML
                          5. Review the "Failure" and make the necessary changes


                          Tilføj billedtekst





                          2019/10/16

                          List Version and Reboot Time

                          s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

                          PowerShell Script to list the Windows Version and Last Reboot Time:

                          List my three S2D Cluster versions and the last reboot time

                          “Thanks to Steffen Brandsborg, Sentia Denmark for creating the PowerShell used below”

                          # Cluster S2D01
                           $ComputerList = "S046031","S046032","S046033","S046034","S046035","S046036"
                          # Cluster S2D02
                           $ComputerList += "S047027","S047028"
                          # Cluster S2D03
                           $ComputerList += "S048001","S048002","S048003","S048004"

                          $InvokeResult = invoke-command -ComputerName $ComputerList -ScriptBlock {
                              $BuildBranch = (Get-ItemProperty "HKLM:\Software\Microsoft\Windows NT\CurrentVersion").BuildBranch
                              $CurrentVersion = (Get-ItemProperty "HKLM:\Software\Microsoft\Windows NT\CurrentVersion").CurrentVersion
                              $CurrentBuildNumber = (Get-ItemProperty "HKLM:\Software\Microsoft\Windows NT\CurrentVersion").CurrentBuildNumber
                              $UBR = (Get-ItemProperty "HKLM:\Software\Microsoft\Windows NT\CurrentVersion").UBR
                              $BootTime = ((Get-WmiObject Win32_OperatingSystem).ConvertToDateTime((Get-WmiObject Win32_OperatingSystem).LastBootUpTime)).ToString("yyyy-MM-dd HH:mm:ss")
                              $ProductName = (Get-ItemProperty "HKLM:\Software\Microsoft\Windows NT\CurrentVersion").ProductName

                              $Result =  @{
                                  Computername = $env:COMPUTERNAME
                                  ProductName = $ProductName
                                  WindowsVersion = ("$CurrentVersion.$CurrentBuildNumber.$UBR")
                                  BootTime = $BootTime
                              }
                              New-Object -TypeName PSObject -Property $Result
                          }
                          $InvokeResult | Sort Computername | Select ComputerName, ProductName, WindowsVersion, BootTime | ft 




















                          2019/10/15

                          Validator for RDMA Configuration and Best Practices

                          s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

                          Validator for RDMA Configuration and Best Practices


                          How to do the Installation and first run in 5 minutes with Validate-DCB

                          More information on GitHub
                          Video part 1 to 4

                          Part 1 - Installation

                          Part 2

                          Part 3

                          Part 4

                          Installation
                          • Install-Module Validate-DCB


                          After installation of the Validate-DCB Module you can run the wizard:
                          1. Run the Validate-DCB from PowerShell and the wizard will start <Next>
                          2. Add the Cluster name
                          3. Resolve
                          4. The Cluster Nodes are listed
                          5. Next
                          6. Add vSwitch Name
                          7. Add pNIC, vNIC and vlan information
                          8. Select RDMA type iWARP or RoCE
                          9. Select the Jumbo Frames (Ethernet MTU size)
                            Note: the MTU for RDMA is not the same as Ethernet MTU
                            For Mellanox list the Ethernet and RoCE/RDMA MTU with "mlx5cmd.exe -Stat"
                            Both the pNIC and the vNIC for SMB need to have the same MTU Size
                          10. Next
                          11. Policy Name
                          12. Bandwidth Reservation
                          13. Next
                          14. Save the configuration file
                          15. Export (the job will start)








                          2019/10/09

                          RoCE Troubleshooting

                          s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

                          RoCE Troubleshooting

                          The RDMA Activity Performance Counter can give you some useful information

                          RDMA Accepted Connections will increase for each new connection. The Picture show a lab system with a RoCE Error. RDMA Accepted Connections will normally only change if you move VMs (SOFS), disable/enable pNIC/vNIC or reboot a Node.

                          The picture show that I have thousands of Accepted Connections and after just a few seconds I have 4 Accepted Connections. With no change in the Cluster.

                          Active Connections

                          Show the Active Connections on each vNIC/pNIC to see the connect use the "netstat -xan"











                          Monitor RoCE - Windows Server

                          s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

                          Performance Monitor

                          Monitor Priority and Pause Frames for RDMA (RoCE) with Mellanox QoS

                          Examples of Physical Switch configuration for Mellanox, Cisco, HPE, Fujitsu and Dell

                          For all the physical switch examples, always get your hardware vendor to verify your settings before use.
                          1. Mellanox
                          2. Cisco
                          3. HPE
                          4. Fujitsu
                          5. Dell
                          6. Lenovo
                          Lab setup:
                          Storages Spaces Direct (S2D) clusters with 2,4 or 6 nodes, Windows Server 2016 and Windows Server 2019
                            The Host and Switch need to be configured for DCB/PFC/ETS
                            Note: Microsoft does not currently support DCBx/LLDP if you use SET. You need to be sure that the Switch is not set to "PFC Auto" on the used ports. But use "PFC Enable" or "PFC On"

                            Use the Microsoft Deployment Guide and Validate DCB Scripts for Host validation
                            In the Examples we use Priority 3 for Storage and Priority 7 for Cluster Traffic.
                            • Storage Traffic = SMB 445 Traffic (Storage, CSV and Live Migration)
                              • Storage Traffic need PFC/ETS
                            • Cluster Traffic = UDP 3343 Traffic
                              • Cluster Traffic only use the ETS
                            In the lab I use Jumbo Frames for Ethernet and auto size for RDMA.
                            (RDMA/RoCE Max Frame Size is in Auto Mode).

                            Sent Pause Frames
                            The total number of pause frames sent from this priority to the far-end port. The untagged instance indicates the number of global pause frames that were sent.
                            Sent Pause Duration
                            The total duration of packets transmission being paused on this priority in microseconds.
                            Received Pause Frames
                            The number of pause frames that were received to this priority from the far-end port. The untagged instance indicates the number of global pause frames that were received.
                            Received Pause Duration
                            The total duration that far-end port was requested to pause for the transmission of packets in microseconds.
                            Sent Discard Frames 
                            The number of packets discarded by the transmitter.  Note: this counter is per Traffic Class (TC) and not per priority.


                            Performance Monitor:
                            The pictures below shows the Priority 3 (SMB) and also Priority 7 (Cluster)
                            (There need to be high load on the Physical Switch Port and/or Network Adapter before you will see that values change for Pause Frames).

                            New-NetQosPolicy "Cluster" -Cluster -PriorityValue8021Action 7
                            New-NetQosPolicy "SMB" -NetDirectPortMatchCondition 445 -PriorityValue8021Action 3
                            New-NetQosPolicy "DEFAULT" -Default -PriorityValue8021Action 0

                            Set-VMNetworkAdapter -ManagementOS -Name "SMB1" -IeeePriorityTag on
                            Set-VMNetworkAdapter -ManagementOS -Name "SMB2" -IeeePriorityTag on




                            Performance Monitor:
                            The pictures below shows the Priority 3 (SMB) and Pause Frames (Rcv/Sent) on the pNICs with 10 minuttes between each other. We can see that the Frames increased, so the system sent and recived it correct. (There need to be high load on the Physical Switch Port and/or Network Adapter before you will see that values change).














                            Monitor RoCE - Mellanox

                            s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

                            Performance Monitor
                            Monitor Priority and Pause Frames for RDMA (RoCE) on Mellanox Switch.

                            Microsoft references:
                            For the screen captures I use two different Mellanox Switch.
                            • Mellanox SX1012 (SwitchX - Onyx 3.6.8010)
                            • Mellanox SN2100 (Spectrum - Onyx 3.6.8010)
                            • The SwitchX and Spectrum are configured in different ways when we look on L2 ETS
                            • The Spectrum also support to use L3 (DSCP) Note: Microsoft configuration guide use RoCE Layer 2 (L2) for S2D that use SET (NDKm2)

                            ******************************************************************************
                            Mellanox Show Commands
                            ******************************************************************************

                            S2D# show dcb priority-flow-control detail
                            S2D# show dcb priority-flow-control
                            S2D# show dcb priority-flow-control interface ethernet 1/1
                            S2D# show dcb ets
                            S2D# show dcb ets interface ethernet 1/1


                            References

                            Microsoft:
                            Windows Server 2016 and 2019 RDMA Deployment Guide
                            Validate DCB

                            Mellanox L2 PCP TC:
                            How to Install Windows Server 2016 with RoCEv2 and Switch Embedded Teaming over HA Mellanox Network Solution
                            Understanding QoS Classification (Trust) on Spectrum Switche
                            Understanding Traffic Class (TC) Scheduling on Mellanox Spectrum Switches (WRR,SP)
                            Understanding RoCEv2 Congestion Management
                            DCBX Versions and Support on Mellanox  Ethernet Switches
                            HowTo Configure PFC on ConnectX-4

                            Mellanox L3 - DSCP:
                            Recommended Network Configuration Examples for RoCE Deployment
                            Lossless RoCE Configuration for MLNX-OS Switches in DSCP-Based QoS Mode
                            Lossless RoCE Configuration for MLNX-OS Switches in DSCP-Based QoS Mode (advanced mode)
                            Lossless RoCE Configuration for WinOF2 driver in DSCP-based QoS mode
                            HowTo Configure Mellanox Spectrum Switch for Lossless RoCE
                            HowTo Configure ECN on Mellanox Ethernet Switches (Spectrum)

                            Terminology:
                            pNIC Physical NIC, the physical hardware that exchanges packets with the TOR
                            vNIC    Host vNIC – Virtual NIC from vSwitch exposed in the host partition
                            tNIC Host tNIC - Team Interface NIC from LBFO Team 
                            vmNIC Virtual Machine NIC – Virtual NIC from vSwitch exposed in a guest partition
                            vSwitch Hyper-V virtual switch
                            SET Switch Embedded Teaming, Hyper-V virtual switch supported in Windows Server 2016 and 2019

                            ToR Top of Rack switch
                            RDMA Remote Direct Memory Access
                            RoCE    RDMA over Converged Ethernet
                            RoCEv2 2nd generation RoCE using UDP/IP for routability (a.k.a. Routable RoCE)

                            DCB     Data Center Bridging
                            LLDP    Link Layer Data Protocol
                            DCBx    Data Center Bridging Capability Exchange protocol (DCBX) is an extension of LLDP.
                            PFC     Priority Flow Control
                            ETS     Enhanced Transmission Service
                            TC      Traffic Class

                            ECN     Explicit Congestion Notification
                            RED     Random Early Detection
                            CNP     Congestion Notification Packet. CNP control frames (congestion ACK)
                            SP      Strict Priority
                            WRR     Weighted Round Robin


                            ******************************************************************************
                            Mellanox SX1012 (SwitchX)
                            ******************************************************************************
                            S2D# show dcb priority-flow-control detail

                            PFC enabled
                            Priority Enabled List    :3
                            Priority Disabled List   :0 1 2 4 5 6 7

                            PFC Port Eth1/x            Information
                            -----------------------------------------------
                            PFC Port Mode       :On
                            PFC Oper State      :On
                            No Remote Entry is Present
                            -----------------------------------------------

                            ******************************************************************************
                            S2D# show dcb priority-flow-control

                            PFC enabled
                            Priority Enabled List    :3
                            Priority Disabled List   :0 1 2 4 5 6 7

                            TC     Lossless
                            ---    ----------
                            0           N
                            1           Y
                            2           Y
                            3           N

                            Interface      PFC admin        PFC oper
                            ------------   --------------   -------------
                            Eth1/1           On               Enabled
                            Eth1/2           On               Enabled
                            Eth1/3           On               Enabled
                            Eth1/4           On               Enabled
                            Eth1/5           On               Enabled
                            Eth1/6           On               Enabled
                            Eth1/7           On               Enabled
                            Eth1/8           On               Enabled
                            Eth1/9           On               Enabled
                            Eth1/10          On               Enabled
                            Eth1/11          On               Enabled
                            Eth1/12          On               Enabled


                            ******************************************************************************
                            S2D# show dcb priority-flow-control interface ethernet 1/1

                            PFC enabled
                            Priority Enabled List    :3
                            Priority Disabled List   :0 1 2 4 5 6 7

                            TC     Lossless
                            ---    ----------
                            0           N
                            1           Y
                            2           Y
                            3           N

                            Interface      PFC admin        PFC oper
                            ------------   --------------   -------------
                            Eth1/1           On               Enabled

                            ******************************************************************************
                            S2D# show dcb ets
                            (Note: Default value before change)

                            ETS enabled

                            TC        Bandwidth
                            --------------------------
                            0         25%
                            1         25%
                            2         25%
                            3         25%

                            Number of Traffic Class: 4

                            ******************************************************************************
                            S2D# show dcb ets interface ethernet 1/1
                            (Note: Default value before change)

                            ETS Port Mode             :AUTO MODE
                            ETS Oper State            :INIT STATE
                            ETS State Machine Type    :Assymetric
                            -----------------------------------------------
                            ETS Local Port Info
                            -----------------------------------------------
                            TC bandwidth table
                            -----------------------------------------------
                            TC        Bandwidth        RecomBandwidth
                            -----------------------------------------------
                            0         25%              25%
                            1         25%              25%
                            2         25%              25%
                            3         25%              25%

                            priority assignment table
                            --------------------------------------
                            Priority     TC
                            --------------------------------------
                            0            0
                            1            0
                            2            1
                            3            1
                            4            2
                            5            2
                            6            3
                            7            3

                            Number of Traffic Class: 4

                            Willing Status:  Disable
                            -----------------------------------------------
                            ETS Admin Port Info
                            -----------------------------------------------
                            TC        Bandwidth        RecomBandwidth
                            -----------------------------------------------
                            0         25%              25%
                            1         25%              25%
                            2         25%              25%
                            3         25%              25%
                            -----------------------------------------------
                            ETS Remote Port Info
                            -----------------------------------------------
                            No Remote Entry is Present
                            -----------------------------------------------

                            ******************************************************************************
                            Configuration of SX1012 (SwitchX)
                            ******************************************************************************

                            Mapping of priority to traffic classes (TC)
                            Priority 0 and 1 mapped to TC 0
                            Priority 2 and 3 mapped to TC 1
                            Priority 4 and 5 mapped to TC 2
                            Priority 6 and 7 mapped to TC 3

                            TC 0 and TC 3 are lossy TCs.
                            TC 1 and TC 2 are lossless TCs.

                            For the PFC enabled priorities we need to use the lossless TCs

                            So for the Microsoft Example use case we change the default to:
                            Priority 0 Default traffic 39%
                            Priority 3 SMB traffic 60%
                            Priority 5 Cluster traffic 1%

                            S2D (config)# dcb ets tc bandwidth 39 60 1 0

                            ******************************************************************************

                            S2D# show dcb ets interface ethernet 1/1
                            (Note: Value after change)

                            S2D# sh dcb ets interface ethernet 1/1

                            ETS Port Mode             :AUTO MODE
                            ETS Oper State            :INIT STATE
                            ETS State Machine Type    :Assymetric
                            -----------------------------------------------
                            ETS Local Port Info
                            -----------------------------------------------
                            TC bandwidth table
                            -----------------------------------------------
                            TC        Bandwidth        RecomBandwidth
                            -----------------------------------------------
                            0         39%              39%
                            1         60%              60%
                            2          1%               1%
                            3          0%               0%

                            priority assignment table
                            --------------------------------------
                            Priority     TC
                            --------------------------------------
                            0            0
                            1            0
                            2            1
                            3            1
                            4            2
                            5            2
                            6            3
                            7            3

                            Number of Traffic Class: 4

                            Willing Status:  Disable
                            -----------------------------------------------
                            ETS Admin Port Info
                            -----------------------------------------------
                            TC        Bandwidth        RecomBandwidth
                            -----------------------------------------------
                            0         39%              39%
                            1         60%              60%
                            2          1%               1%
                            3          0%               0%
                            -----------------------------------------------
                            ETS Remote Port Info
                            -----------------------------------------------
                            No Remote Entry is Present
                            -----------------------------------------------

                            ******************************************************************************
                            Mellanox SN-2100 Spectrum
                            ******************************************************************************

                            S2D# show dcb ets interface ethernet 1/1
                            (Note: Default value before change)

                            Eth1/1
                             Interface Bandwidth Shape [Mbps]: N/A
                             Multicast unaware mapping : disabled

                             ETS per TC :
                             TC Scheduling Mode Weight Weight (%)
                             -- --------------- ------ ----------
                             0  WRR             12     12
                             1  WRR             13     13
                             2  WRR             12     12
                             3  WRR             13     13
                             4  WRR             12     12
                             5  WRR             13     13
                             6  WRR             12     12
                             7  WRR             13     13

                             Bandwidth Shape per TC:
                             TC Bandwidth Shape [Mbps]
                             -- ----------------------
                             0  N/A
                             1  N/A
                             2  N/A
                             3  N/A
                             4  N/A
                             5  N/A
                             6  N/A
                             7  N/A

                             Bandwidth Guarantee per TC:
                             TC Bandwidth Guaranteed [Mbps]
                             -- ---------------------------
                             0  0
                             1  0
                             2  0
                             3  0
                             4  0
                             5  0
                             6  0
                             7  0

                             Switch Priority to TC mapping:
                             Switch Priority TC
                             --------------- --
                             0               0
                             1               1
                             2               2
                             3               3
                             4               4
                             5               5
                             6               6
                             7               7

                            ******************************************************************************







                            ******************************************************************************

                            Mellanox Switch configuration example (L2):
                            ******************************************************************************

                            Mellanox SN2100 (Spectrum)

                            S2D> enable
                            S2D# configure terminal 
                            S2D (Config)# interface ethernet 1/1-1/12 traffic-class 3 dcb ets Strict
                            S2D (Config)# interface ethernet 1/1-1/12 traffic-class 7 dcb ets Strict

                            S2D (Config)# interface ethernet 1/1-1/x flowcontrol send off force
                            S2D (Config)# interface ethernet 1/1-1/x flowcontrol receive off force

                            S2D (Config)# dcb priority-flow-control enable force
                            S2D (Config)# dcb priority-flow-control priority 3 enable
                            (Priority 3 is used for Storage (SMB) traffic)
                            S2D (Config)# interface ethernet 1/1-1/x dcb priority-flow-control mode on force
                            S2D (Config)# interface ethernet 1/x switchport mode hybrid
                            (Need to be repeated for each port from 1 to 12/16)
                            S2D# interface ethernet 1/x switchport hybrid allowed-vlan all
                            Need to be repeated for each port from 1 to 12/16. Only allow the needed vlans, the "allowed-vlan all" is not recommended, this is from a lab/test system).
                            S2D (Config)# exit
                            S2D# write memory

                            ******************************************************************************
                            S2D# show dcb ets interface ethernet 1/1
                            Eth1/1
                             Interface Bandwidth Shape [Mbps]: N/A
                             Multicast unaware mapping : disabled

                             ETS per TC :
                             TC Scheduling Mode Weight Weight (%)
                             -- --------------- ------ ----------
                             0  WRR             12     12
                             1  WRR             13     13
                             2  WRR             12     12
                             3  Strict          0      0
                             4  WRR             12     12
                             5  WRR             13     13
                             6  WRR             12     12
                             7  Strict          0      0

                             Bandwidth Shape per TC:
                             TC Bandwidth Shape [Mbps]
                             -- ----------------------
                             0  N/A
                             1  N/A
                             2  N/A
                             3  N/A
                             4  N/A
                             5  N/A
                             6  N/A
                             7  N/A

                             Bandwidth Guarantee per TC:
                             TC Bandwidth Guaranteed [Mbps]
                             -- ---------------------------
                             0  0
                             1  0
                             2  0
                             3  0
                             4  0
                             5  0
                             6  0
                             7  0

                             Switch Priority to TC mapping:
                             Switch Priority TC
                             --------------- --
                             0               0
                             1               1
                             2               2
                             3               3
                             4               4
                             5               5
                             6               6
                             7               7



                            ******************************************************************************
                            Mellanox Switch configuration example with DSCP:

                            Draft - Test only last update 2019.01.19
                            Note:
                            #1: The example use both L2/L3. SwitchX use L2 and Spectrum use both L2/L3
                            #2: Microsoft configuration guide use RoCE Layer 2 (L2) for S2D.
                            #3: DSCP is supported by Windows Server OS. However to use it for S2D is not recommended right now by Microsoft and for test you need to work close with the Hardware vendors. 

                            Lossless RoCE Configuration for MLNX-OS Switches in DSCP-Based QoS Mode
                            Lossless RoCE Configuration for MLNX-OS Switches in DSCP-Based QoS Mode (Old version)
                            ******************************************************************************

                            S2D (Config)# interface ethernet 1/1-1/x qos trust ?
                                   port       based on port default settings
                                   L2         based on PCP, DEI fields
                                   L3         based on EXP, DSCP fields
                                   both       based on PCP, DEI and EXP, DSCP fields

                            S2D (Config)# interface ethernet 1/1-1/x qos trust both
                            (Note: In my lab I used both SwitchX and Spectrum. The Spectrum Switch need to accepts PCP and DSCP in my example)

                            ******************************************************************************

                            S2D (config) # interface ethernet 1/1-1/x traffic-class 3 congestion-control ecn minimum-absolute 150 maximum-absolute 1500
                            S2D (config) # traffic pool roce type lossless
                            S2D (config) # traffic pool roce memory percent 50.00
                            S2D (config) # traffic pool roce map switch-priority 3
                            S2D (config) # interface ethernet 1/1-1/x traffic-class 3 dcb ets strict
                            S2D (config) # interface ethernet 1/1-1/x traffic-class 7 dcb ets strict
                            S2D (config) # interface ethernet 1/1-1/x qos trust both

                            ******************************************************************************

                            S2D (Config)# interface ethernet 1/1-1/x traffic-class 0 bind switch-priority 0
                            S2D (Config)# interface ethernet 1/1-1/x traffic-class 3 bind switch-priority 3
                            S2D (Config)# interface ethernet 1/1-1/x traffic-class 7 bind switch-priority 7

                            ******************************************************************************

                            S2D (Config)# interface ethernet 1/1-1/x flowcontrol send off force
                            S2D (Config)# interface ethernet 1/1-1/x flowcontrol receive off force

                            S2D (Config)# dcb priority-flow-control enable force
                            S2D (Config)# dcb priority-flow-control priority 3 enable
                            (Priority 3 is used for Storage (SMB) traffic)
                            S2D (Config)# interface ethernet 1/1-1/x dcb priority-flow-control mode on force
                            S2D (Config)# interface ethernet 1/x switchport mode hybrid
                            (Need to be repeated for each port from 1 to x)
                            S2D (Config)# interface ethernet 1/x switchport hybrid allowed-vlan all
                            Need to be repeated for each port from 1 to x. Only allow the needed vlans, the "allowed-vlan all" is not recommended, this is from a lab/test system).

                            ******************************************************************************
                            Host pNIC configuration

                            Driver version 2.10 or newer

                            S2D PS> Mlx5Cmd.exe -QosConfig -SetupRoceQosConfig -Name NIC3 -Configure 2



                            Default DSCP to switch-priority mapping:
                            0-7 → 0
                            8-15 → 1
                            16-23 → 2
                            24-31 → 3
                            32-39 → 4
                            40-47 → 5
                            48-55 → 6
                            56-63 → 7


                            ******************************************************************************