Featured Post

Links library

2019/01/21

Monitor RoCE - Windows Server

s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

Performance Monitor

(DRAFT, work in progress last update 2019.06.11)

Monitor Priority and Pause Frames for RDMA (RoCE) with Mellanox QoS

Examples of Physical Switch configuration for Mellanox, Cisco, HPE, Fujitsu and Dell

For all the physical switch examples, always get your hardware vendor to verify your settings before use.
  1. Mellanox
  2. Cisco
  3. HPE
  4. Fujitsu
  5. Dell
  6. Lenovo
Lab setup:
Storages Spaces Direct (S2D) clusters with 2,4 or 6 nodes, Windows Server 2016 and Windows Server 2019
    The Host and Switch need to be configured for DCB/PFC/ETS
    Note: Microsoft does not currently support DCBx/LLDP. You need to be sure that the Switch is not set to "PFC Auto" on the used ports. But use "PFC Enable" or "PFC On"

    Use the Microsoft Deployment Guide and Validate DCB Scripts for Host validation
    In the Examples we use Priority 3 for Storage and Priority 5 or 7 for Cluster Traffic.
    • Storage Traffic = SMB 445 Traffic (Storage, CSV and Live Migration)
    • Cluster Traffic = UDP 3343 Traffic
    In the lab I use Jumbo Frames for Ethernet and auto size for RDMA.
    (RDMA/RoCE Max Frame Size is in Auto Mode).



    Performance Monitor:
    The pictures below shows the Priority 3 (SMB) and Pause Frames (Rcv/Sent) on the pNIC with 5-10 minuttes between each other. We can see that the Frames increased, so the system sent and recived it correct. (There need to be high load on the Physical Switch Port and/or Network Adapter before you will see that values change).



    Sent Pause Frames
    The total number of pause frames sent from this priority to the far-end port. The untagged instance indicates the number of global pause frames that were sent.
    Sent Pause Duration
    The total duration of packets transmission being paused on this priority in microseconds.
    Received Pause Frames
    The number of pause frames that were received to this priority from the far-end port. The untagged instance indicates the number of global pause frames that were received.
    Received Pause Duration
    The total duration that far-end port was requested to pause for the transmission of packets in microseconds.
    Sent Discard Frames 
    The number of packets discarded by the transmitter.  Note: this counter is per Traffic Class (TC) and not per priority.







    Performance Monitor:
    The pictures below shows the Priority 3 (SMB) and also Priority 5 (Cluster)
    (There need to be high load on the Physical Switch Port and/or Network Adapter before you will see that values change for Pause Frames).

    The Priority 5 counters are all 0 (Before configuration of the Cluster QoS Settings).

    Now we add the QoS for Cluster traffic

    # Create QoS policies and tag each type of traffic with the relevant priority
    New-NetQosPolicy "Cluster" -Cluster -PriorityValue8021Action 5
    New-NetQosPolicy "SMB" -NetDirectPortMatchCondition 445 -PriorityValue8021Action 3
    New-NetQosPolicy "DEFAULT" -Default -PriorityValue8021Action 0

    The Microsoft SET Switch will only tag SMB Traffic, to get the Cluster traffic tag you need to enable -IeeePriorityTag

    Set-VMNetworkAdapter -ManagementOS -Name "MGMT" -IeeePriorityTag on
    Set-VMNetworkAdapter -ManagementOS -Name "SMB1" -IeeePriorityTag on
    Set-VMNetworkAdapter -ManagementOS -Name "SMB2" -IeeePriorityTag on

    For the compleat QoS configuration please review the Microsoft Network Guide or my Blog that show the lab configuration.

    After the first Host is changed we can see that we sent traffic with Priority 5 (Can also be Priority 7)


    Changed all Host to tag Cluster Traffic. We now see Sent and also Received Traffic with Priority 5







    Monitor RoCE - Lenovo

    s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

    Performance Monitor

    Monitor Priority and Pause Frames for RDMA (RoCE) on Lenovo Switch.

    (DRAFT, work in progress last update 2019.06.11)


    Microsoft references:

    Lenovo:
    Lenovo System X

    ******************************************************************************
    Lenovo Display Commands
    ******************************************************************************

    Editing is done below, removed lines with repeated information replaced by x (port numbers)

    Note: G8332 running CNOS 10.4.3 – Syntax changed with CNOS 10.7 and higher

    The Example use priorities 3 and 5 (The Cluster traffic just need to have the highest priority 5 or 7)

    ****************************************************************************** 

    S2D# display cee ets information
    ETS Global Admin Configuration:
    PGID    BW%   COSq  Priorities          Description
    ==============================================================================
    0       49    0      0 1 2              Default
    1       0     NA
    2       0     2      4 6 7
    3       50    3      3                  RoCEv2_Traffic (SMB)
    4       0     4
    5       1     5      5                  Cluster
    6       0     6
    7       0     7
    15      NA    1

    ETS Global Operational Configuration:
    PGID    BW%   COSq  Priorities          Description
    ==============================================================================
    0       49    0      0 1 2              Default
    1       0     NA
    2       0     2      4 6 7
    3       50    3      3                  RoCEv2_Traffic (SMB)
    4       0     4
    5       1     5      5                  Cluster
    6       0     6
    7       0     7
    15      NA    1

    ETS Interface Admin Configuration
    --------------------
    ETS Willing mode is not supported on the Switch.

    Interface           Advertise
    ================================================================================
    Ethernet1/x         On
    Ethernet1/x         On

    ******************************************************************************
    S2D# display cee pfc
    Global Admin PFC State: On
    Priority  State  Description
    ============================
       0      Dis
       1      Dis
       2      Dis
       3      Ena    RoCEv2_Priority (SMB)
       4      Dis
       5      Dis    Cluster
       6      Dis
       7      Dis
    Admin Configuration:
    --------------------
    PFC Willing mode is not supported on the Switch.

    Interface      State  Advertise Cap   Priorities
    ================================================================================
    Ethernet1/x    On     On        2     3 5
    Ethernet1/x    On     On        2     3 5

    --------------------------------------------------------------------------------

    Operational Configuration
    -------------------------
    Global Operational PFC State: On
    Priority  State  Description
    ============================
       0      Dis
       1      Dis
       2      Dis
       3      Ena    RoCEv2_Priority (SMB)
       4      Dis
       5      Dis    Cluster
       6      Dis
       7      Dis
    Interface      State Cap   Priorities
    ================================================================================
    Ethernet1/x    On    2     3 5
    Ethernet1/x    On    2     3 5

    ******************************************************************************
    Ports used for S2D (PFC) need to show traffic/Packets on Rx/Tx

    S2D# display cee pfc counters
    Priority Flow Control Packets:
    Interface              Received(Rx)            Sent(Tx)
    =======================================================
    Ethernet1/x                       0                   0
    Ethernet1/x                       0                   0
    Ethernet1/7                 4988911            17067089
    Ethernet1/8                 1852123            12482442
    Ethernet1/9                 9800282            12982588
    Ethernet1/10                5229729            20209402
    Ethernet1/11                7244549            16976660
    Ethernet1/x                       0                   0
    Ethernet1/23               12245734            14795755
    Ethernet1/24               12245683             8309129
    Ethernet1/x                       0                   0

    ******************************************************************************

    S2D# display cee interface Ethernet 1/7

    ETS information for the interface Ethernet1/7:
    ETS Global Admin Configuration:
    PGID    BW%   COSq  Priorities          Description
    ==============================================================================
    0       49    0      0 1 2              Default
    1       0     NA
    2       0     2      4 6 7
    3       50    3      3                  RoCEv2_Traffic (SMB)
    4       0     4
    5       1     5      5                  Cluster
    6       0     6
    7       0     7
    15      NA    1

    ETS Global Operational Configuration:
    PGID    BW%   COSq  Priorities          Description
    ==============================================================================
    0       49    0      0 1 2              Default
    1       0     NA
    2       0     2      4 6 7
    3       50    3      3                  RoCEv2_Traffic (SMB)
    4       0     4
    5       1     5      5                  Cluster
    6       0     6
    7       0     7
    15      NA    1

    Admin Configuration
    --------------------
    Advertise Willing
    ================================================================================
    On        Off

    Operational Configuration
    --------------------
    Advertise Willing
    ================================================================================
    Off       Off

    PFC information for the interface Ethernet1/7:
    Admin Configuration
    --------------------
    Interface      State  Advertise Cap   Priorities
    ================================================================================
    Ethernet1/7    On     On        2     3 5

    --------------------------------------------------------------------------------

    Operational Configuration
    -------------------------
    State Cap   Priorities
    ================================================================================
    On    2     3 5

    ******************************************************************************







    2019/01/05

    Monitor RoCE - Mellanox

    s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.


    Performance Monitor
    Monitor Priority and Pause Frames for RDMA (RoCE) on Mellanox Switch.

    (DRAFT, work in progress last update 2019.06.11)


    Microsoft references:

    For the screen captures I use two different Mellanox Switch.
    • Mellanox SX1012 (SwitchX - Onyx 3.6.8010)
    • Mellanox SN2100 (Spectrum - Onyx 3.6.8010)
    • The SwitchX and Spectrum are configured in different ways when we look on L2 ETS
    • The Spectrum also support to use L3 (DSCP) Note: Microsoft configuration guide use RoCE Layer 2 (L2) for S2D.

    ******************************************************************************
    Mellanox Show Commands
    ******************************************************************************

    S2D# show dcb priority-flow-control detail
    S2D# show dcb priority-flow-control
    S2D# show dcb priority-flow-control interface ethernet 1/1
    S2D# show dcb ets
    S2D# show dcb ets interface ethernet 1/1


    References

    Microsoft:
    Windows Server 2016 and 2019 RDMA Deployment Guide
    Validate DCB

    Mellanox L2 PCP TC:
    How to Install Windows Server 2016 with RoCEv2 and Switch Embedded Teaming over HA Mellanox Network Solution
    Understanding QoS Classification (Trust) on Spectrum Switche
    Understanding Traffic Class (TC) Scheduling on Mellanox Spectrum Switches (WRR,SP)
    Understanding RoCEv2 Congestion Management
    DCBX Versions and Support on Mellanox  Ethernet Switches
    HowTo Configure PFC on ConnectX-4

    Mellanox L3 - DSCP:
    Recommended Network Configuration Examples for RoCE Deployment
    Lossless RoCE Configuration for MLNX-OS Switches in DSCP-Based QoS Mode
    Lossless RoCE Configuration for MLNX-OS Switches in DSCP-Based QoS Mode (advanced mode)
    Lossless RoCE Configuration for WinOF2 driver in DSCP-based QoS mode
    HowTo Configure Mellanox Spectrum Switch for Lossless RoCE
    HowTo Configure ECN on Mellanox Ethernet Switches (Spectrum)

    Terminology:
    pNIC Physical NIC, the physical hardware that exchanges packets with the TOR
    vNIC    Host vNIC – Virtual NIC from vSwitch exposed in the host partition
    tNIC Host tNIC - Team Interface NIC from LBFO Team 
    vmNIC Virtual Machine NIC – Virtual NIC from vSwitch exposed in a guest partition
    vSwitch Hyper-V virtual switch
    SET Switch Embedded Teaming, Hyper-V virtual switch supported in Windows Server 2016 and 2019

    ToR Top of Rack switch
    RDMA Remote Direct Memory Access
    RoCE    RDMA over Converged Ethernet
    RoCEv2 2nd generation RoCE using UDP/IP for routability (a.k.a. Routable RoCE)

    DCB     Data Center Bridging
    LLDP    Link Layer Data Protocol
    DCBx    Data Center Bridging Capability Exchange protocol (DCBX) is an extension of LLDP.
    PFC     Priority Flow Control
    ETS     Enhanced Transmission Service
    TC      Traffic Class

    ECN     Explicit Congestion Notification
    RED     Random Early Detection
    CNP     Congestion Notification Packet. CNP control frames (congestion ACK)
    SP      Strict Priority
    WRR     Weighted Round Robin


    ******************************************************************************
    Mellanox SX1012 (SwitchX)
    ******************************************************************************
    S2D# show dcb priority-flow-control detail

    PFC enabled
    Priority Enabled List    :3
    Priority Disabled List   :0 1 2 4 5 6 7

    PFC Port Eth1/x            Information
    -----------------------------------------------
    PFC Port Mode       :On
    PFC Oper State      :On
    No Remote Entry is Present
    -----------------------------------------------

    ******************************************************************************
    S2D# show dcb priority-flow-control

    PFC enabled
    Priority Enabled List    :3
    Priority Disabled List   :0 1 2 4 5 6 7

    TC     Lossless
    ---    ----------
    0           N
    1           Y
    2           Y
    3           N

    Interface      PFC admin        PFC oper
    ------------   --------------   -------------
    Eth1/1           On               Enabled
    Eth1/2           On               Enabled
    Eth1/3           On               Enabled
    Eth1/4           On               Enabled
    Eth1/5           On               Enabled
    Eth1/6           On               Enabled
    Eth1/7           On               Enabled
    Eth1/8           On               Enabled
    Eth1/9           On               Enabled
    Eth1/10          On               Enabled
    Eth1/11          On               Enabled
    Eth1/12          On               Enabled


    ******************************************************************************
    S2D# show dcb priority-flow-control interface ethernet 1/1

    PFC enabled
    Priority Enabled List    :3
    Priority Disabled List   :0 1 2 4 5 6 7

    TC     Lossless
    ---    ----------
    0           N
    1           Y
    2           Y
    3           N

    Interface      PFC admin        PFC oper
    ------------   --------------   -------------
    Eth1/1           On               Enabled

    ******************************************************************************
    S2D# show dcb ets
    (Note: Default value before change)

    ETS enabled

    TC        Bandwidth
    --------------------------
    0         25%
    1         25%
    2         25%
    3         25%

    Number of Traffic Class: 4

    ******************************************************************************
    S2D# show dcb ets interface ethernet 1/1
    (Note: Default value before change)

    ETS Port Mode             :AUTO MODE
    ETS Oper State            :INIT STATE
    ETS State Machine Type    :Assymetric
    -----------------------------------------------
    ETS Local Port Info
    -----------------------------------------------
    TC bandwidth table
    -----------------------------------------------
    TC        Bandwidth        RecomBandwidth
    -----------------------------------------------
    0         25%              25%
    1         25%              25%
    2         25%              25%
    3         25%              25%

    priority assignment table
    --------------------------------------
    Priority     TC
    --------------------------------------
    0            0
    1            0
    2            1
    3            1
    4            2
    5            2
    6            3
    7            3

    Number of Traffic Class: 4

    Willing Status:  Disable
    -----------------------------------------------
    ETS Admin Port Info
    -----------------------------------------------
    TC        Bandwidth        RecomBandwidth
    -----------------------------------------------
    0         25%              25%
    1         25%              25%
    2         25%              25%
    3         25%              25%
    -----------------------------------------------
    ETS Remote Port Info
    -----------------------------------------------
    No Remote Entry is Present
    -----------------------------------------------

    ******************************************************************************
    Configuration of SX1012 (SwitchX)
    ******************************************************************************

    Mapping of priority to traffic classes (TC)
    Priority 0 and 1 mapped to TC 0
    Priority 2 and 3 mapped to TC 1
    Priority 4 and 5 mapped to TC 2
    Priority 6 and 7 mapped to TC 3

    TC 0 and TC 3 are lossy TCs.
    TC 1 and TC 2 are lossless TCs.

    For the PFC enabled priorities we need to use the lossless TCs

    So for the Microsoft Example use case we change the default to:
    Priority 0 Default traffic 39%
    Priority 3 SMB traffic 60%
    Priority 5 Cluster traffic 1%

    S2D (config)# dcb ets tc bandwidth 39 60 1 0

    ******************************************************************************

    S2D# show dcb ets interface ethernet 1/1
    (Note: Value after change)

    S2D# sh dcb ets interface ethernet 1/1

    ETS Port Mode             :AUTO MODE
    ETS Oper State            :INIT STATE
    ETS State Machine Type    :Assymetric
    -----------------------------------------------
    ETS Local Port Info
    -----------------------------------------------
    TC bandwidth table
    -----------------------------------------------
    TC        Bandwidth        RecomBandwidth
    -----------------------------------------------
    0         39%              39%
    1         60%              60%
    2          1%               1%
    3          0%               0%

    priority assignment table
    --------------------------------------
    Priority     TC
    --------------------------------------
    0            0
    1            0
    2            1
    3            1
    4            2
    5            2
    6            3
    7            3

    Number of Traffic Class: 4

    Willing Status:  Disable
    -----------------------------------------------
    ETS Admin Port Info
    -----------------------------------------------
    TC        Bandwidth        RecomBandwidth
    -----------------------------------------------
    0         39%              39%
    1         60%              60%
    2          1%               1%
    3          0%               0%
    -----------------------------------------------
    ETS Remote Port Info
    -----------------------------------------------
    No Remote Entry is Present
    -----------------------------------------------

    ******************************************************************************
    Mellanox SN-2100 Spectrum
    ******************************************************************************

    S2D# show dcb ets interface ethernet 1/1
    (Note: Default value before change)

    Eth1/1
     Interface Bandwidth Shape [Mbps]: N/A
     Multicast unaware mapping : disabled

     ETS per TC :
     TC Scheduling Mode Weight Weight (%)
     -- --------------- ------ ----------
     0  WRR             12     12
     1  WRR             13     13
     2  WRR             12     12
     3  WRR             13     13
     4  WRR             12     12
     5  WRR             13     13
     6  WRR             12     12
     7  WRR             13     13

     Bandwidth Shape per TC:
     TC Bandwidth Shape [Mbps]
     -- ----------------------
     0  N/A
     1  N/A
     2  N/A
     3  N/A
     4  N/A
     5  N/A
     6  N/A
     7  N/A

     Bandwidth Guarantee per TC:
     TC Bandwidth Guaranteed [Mbps]
     -- ---------------------------
     0  0
     1  0
     2  0
     3  0
     4  0
     5  0
     6  0
     7  0

     Switch Priority to TC mapping:
     Switch Priority TC
     --------------- --
     0               0
     1               1
     2               2
     3               3
     4               4
     5               5
     6               6
     7               7

    ******************************************************************************







    ******************************************************************************

    Mellanox Switch configuration example (L2):
    ******************************************************************************

    Mellanox SN2100 (Spectrum)

    S2D> enable
    S2D# configure terminal 
    S2D (Config)# interface ethernet 1/1-1/12 traffic-class 0 dcb ets wrr 39
    S2D (Config)# interface ethernet 1/1-1/12 traffic-class 3 dcb ets wrr 60
    S2D (Config)# interface ethernet 1/1-1/12 traffic-class 5 dcb ets wrr 1

    S2D (Config)# interface ethernet 1/1-1/12 traffic-class 6 dcb ets strict

    S2D (Config)# interface ethernet 1/1-1/12 traffic-class 1 dcb ets wrr 0
    S2D (Config)# interface ethernet 1/1-1/12 traffic-class 2 dcb ets wrr 0
    S2D (Config)# interface ethernet 1/1-1/12 traffic-class 4 dcb ets wrr 0
    S2D (Config)# interface ethernet 1/1-1/12 traffic-class 7 dcb ets wrr 0
    S2D (Config)# interface ethernet 1/1-1/x flowcontrol send off force
    S2D (Config)# interface ethernet 1/1-1/x flowcontrol receive off force

    S2D (Config)# dcb priority-flow-control enable force
    S2D (Config)# dcb priority-flow-control priority 3 enable
    (Priority 3 is used for Storage (SMB) traffic)
    S2D (Config)# dcb priority-flow-control priority 5 enable
    (Priority 5 is used for Cluster traffic)
    S2D (Config)# interface ethernet 1/1-1/x dcb priority-flow-control mode on force
    S2D (Config)# interface ethernet 1/x switchport mode hybrid
    (Need to be repeated for each port from 1 to 12/16)
    S2D# interface ethernet 1/x switchport hybrid allowed-vlan all
    Need to be repeated for each port from 1 to 12/16. Only allow the needed vlans, the "allowed-vlan all" is not recommended, this is from a lab/test system).
    S2D (Config)# exit
    S2D# write memory

    ******************************************************************************
    S2D# show dcb ets interface ethernet 1/1
    Eth1/1
     Interface Bandwidth Shape [Mbps]: N/A
     Multicast unaware mapping : disabled

     ETS per TC :
     TC Scheduling Mode Weight Weight (%)
     -- --------------- ------ ----------
     0  WRR             39     39
     1  WRR             0      0
     2  WRR             0      0
     3  WRR             60     60
     4  WRR             0      0
     5  WRR             1      1
     6  Strict          0      0
     7  WRR             0      0

     Bandwidth Shape per TC:
     TC Bandwidth Shape [Mbps]
     -- ----------------------
     0  N/A
     1  N/A
     2  N/A
     3  N/A
     4  N/A
     5  N/A
     6  N/A
     7  N/A

     Bandwidth Guarantee per TC:
     TC Bandwidth Guaranteed [Mbps]
     -- ---------------------------
     0  0
     1  0
     2  0
     3  0
     4  0
     5  0
     6  0
     7  0

     Switch Priority to TC mapping:
     Switch Priority TC
     --------------- --
     0               0
     1               1
     2               2
     3               3
     4               4
     5               5
     6               6
     7               7



    ******************************************************************************
    Mellanox Switch configuration example with DSCP:

    Draft - Test only last update 2019.01.19
    Note:
    #1: The example use both L2/L3. SwitchX use L2 and Spectrum use both L2/L3
    #2: Microsoft configuration guide use RoCE Layer 2 (L2) for S2D.
    #3: DSCP is supported by Windows Server OS. However to use it for S2D is not recommended right now by Microsoft and for test you need to work close with the Hardware vendors. 

    Lossless RoCE Configuration for MLNX-OS Switches in DSCP-Based QoS Mode
    Lossless RoCE Configuration for MLNX-OS Switches in DSCP-Based QoS Mode (Old version)
    ******************************************************************************

    S2D (Config)# interface ethernet 1/1-1/x qos trust ?
           port       based on port default settings
           L2         based on PCP, DEI fields
           L3         based on EXP, DSCP fields
           both       based on PCP, DEI and EXP, DSCP fields

    S2D (Config)# interface ethernet 1/1-1/x qos trust both
    (Note: In my lab I used both SwitchX and Spectrum. The Spectrum Switch need to accepts PCP and DSCP in my example)

    ******************************************************************************

    S2D (config) # interface ethernet 1/1-1/x traffic-class 3 congestion-control ecn minimum-absolute 150 maximum-absolute 1500
    S2D (config) # traffic pool roce type lossless
    S2D (config) # traffic pool roce memory percent 50.00
    S2D (config) # traffic pool roce map switch-priority 3 
    S2D (config) # interface ethernet 1/1-1/x traffic-class 6 dcb ets strict
    S2D (config) # interface ethernet 1/1-1/x qos trust both

    ******************************************************************************

    S2D (Config)# interface ethernet 1/1-1/x traffic-class 0 bind switch-priority 0
    S2D (Config)# interface ethernet 1/1-1/x traffic-class 3 bind switch-priority 3
    S2D (Config)# interface ethernet 1/1-1/x traffic-class 6 bind switch-priority 6

    ******************************************************************************

    S2D (Config)# interface ethernet 1/1-1/x flowcontrol send off force
    S2D (Config)# interface ethernet 1/1-1/x flowcontrol receive off force

    S2D (Config)# dcb priority-flow-control enable force
    S2D (Config)# dcb priority-flow-control priority 3 enable
    (Priority 3 is used for Storage (SMB) traffic)
    S2D (Config)# dcb priority-flow-control priority 5 enable
    (Priority 5 is used for Cluster traffic)
    S2D (Config)# interface ethernet 1/1-1/x dcb priority-flow-control mode on force
    S2D (Config)# interface ethernet 1/x switchport mode hybrid
    (Need to be repeated for each port from 1 to x)
    S2D (Config)# interface ethernet 1/x switchport hybrid allowed-vlan all
    Need to be repeated for each port from 1 to x. Only allow the needed vlans, the "allowed-vlan all" is not recommended, this is from a lab/test system).

    ******************************************************************************
    Host pNIC configuration

    Driver version 2.10 or newer







    S2D PS> Mlx5Cmd.exe -QosConfig -SetupRoceQosConfig -Name NIC3 -Configure 2





    Default DSCP to switch-priority mapping:
    0-7 → 0
    8-15 → 1
    16-23 → 2
    24-31 → 3
    32-39 → 4
    40-47 → 5
    48-55 → 6
    56-63 → 7


    ******************************************************************************