Split Mellanox Port and connect to Cisco Switch

Split Mellanox Port from 1x100 Gigabit to 4x10 Gigabit ports and connect to Cisco Switch

*** Disclaimer ***
s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment.
For any reference links to other websites we encourages you to read the privacy statements of the third-party websites.
The names of actual companies and products mentioned herein may be the trademarks of their respective owners.
***

Split Mellanox Port from 1x100 Gigabit to 4x10 Gigabit ports and connect to Cisco Switch

The Example use a Cisco 2360G (4x10 Gigabit Ports) 
Mellanox Split Cable with Prolabs Transceiver, Cisco compatible and has been programmed to match the Cisco Switch 2360G

***

Split Mellanox Port 15

***

enable

configure terminal 

interface ethernet 1/15

shutdown

module-type qsfp-split-4

Yes

For port configuration or reset port, please go to:
Split Mellanox Port from 1x100 Gigabit to 4x10 Gigabit

Mellanox
HowTo Use Mellanox Interconnect Products with Cisco Switches

Cisco Switch show command for idprom and transceiver:
show interfaces tenGigabitEthernet 0/1 status

show idprom interface tenGigabitEthernet 0/1
show idprom interface tenGigabitEthernet 0/1 detail
show interfaces tenGigabitEthernet 0/1 transceiver
show interfaces tenGigabitEthernet 0/1 transceiver properties

Split Mellanox Port from 1x100 Gigabit to 4x10 Gigabit

Split Mellanox Port from 1x100 Gigabit to 4x10 Gigabit ports and connect to Windows Server Host

*** Disclaimer ***
s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment.
For any reference links to other websites we encourages you to read the privacy statements of the third-party websites.
The names of actual companies and products mentioned herein may be the trademarks of their respective owners.
***

Split Mellanox Port from 1x100 Gigabit to 4x10 Gigabit ports and connect to Windows Server Host

***

Split Mellanox Port 16

***

enable

configure terminal 

interface ethernet 1/16

shutdown

module-type qsfp-split-4

Yes

***

To used Port 16 (the 4 split ports) for Host running S2D  (RDMA/RoCE)

***

dcb priority-flow-control enable force

dcb priority-flow-control priority 3 enable

***

WRR or Strict - Example with Strict

***

interface ethernet 1/16/1-1/16/4 traffic-class 3 dcb ets strict
interface ethernet 1/16/1-1/16/4 traffic-class 7 dcb ets strict

interface ethernet 1/16/1-1/16/4 flowcontrol send off force

interface ethernet 1/16/1-1/16/4 flowcontrol receive off force

interface ethernet 1/16/1-1/16/4 speed 10000 force

interface ethernet 1/16/1-1/16/4 mtu 9216 force

interface ethernet 1/16/1-1/16/4 dcb priority-flow-control mode on force

interface ethernet 1/16/1 switchport mode hybrid

interface ethernet 1/16/2 switchport mode hybrid

interface ethernet 1/16/3 switchport mode hybrid

interface ethernet 1/16/4 switchport mode hybrid

interface ethernet 1/16/1 switchport hybrid allowed-vlan (your VLAN list for that port)

interface ethernet 1/16/2 switchport hybrid allowed-vlan (your VLAN list for that port)

interface ethernet 1/16/3 switchport hybrid allowed-vlan (your VLAN list for that port)

interface ethernet 1/16/4 switchport hybrid allowed-vlan (your VLAN list for that port)

***

To reset the port 16 back to one port
***
interface ethernet 1/16/1  shutdown
interface ethernet 1/16/2  shutdown
interface ethernet 1/16/3  shutdown
interface ethernet 1/16/4  shutdown
interface ethernet 1/16/1  module-type qsfp force

***

For more information about how to configure and monitor the Mellanox Switch with RoCE

• Mellanox SN2100
• Mellanox SX1012

1 or 2 vlan/subnets for S2D Cluster - Troubleshooting series

1 or 2 vlan/subnets for S2D Cluster - Troubleshooting series

*** Disclaimer ***
s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment.
For any reference links to other websites we encourages you to read the privacy statements of the third-party websites.
The names of actual companies and products mentioned herein may be the trademarks of their respective owners.
***

Configuration of VLAN/Subnets for Storage Spaces Direct (S2D) Cluster
Configuration of VLAN/Subnets for the Troubleshooting Series, Part 1-3
Shows the steps for the Part 1, how to change from 1x VLAN to 2x VLANs.
Configuration of System Center Virtual Machine Manager (SCVMM)

Demo Part #1 - Demo with two vlan/subnets.

The demo shows a Windows Server 2019 cluster that has been configured with two vlan/subnets. The subnet only exist in one physical switch, if the vNIC (SMB vNIC) has to move to the second physical NIC it will not have connectivity. The Failover Cluster will "know" that the network connectivity is down...

SET Switch, created from SCVMM

• vNIC "MGMT"
• vNIC "SMB11" - vlan 211 > Team Mapping to pNIC1
• vNIC "SMB12" - vlan 212 > Team Mapping to pNIC2

Demo Part #2 - Demo with two vlan/subnets but the vlans is present in both ToR Switches


The demo shows a Windows Server 2019 Storage Spaces Direct (S2D) Cluster that has been configured with two VLAN/Subnets. The subnet exist in both physical Switches, if the vNIC (SMB vNIC) has to move to the second physical NIC it will have Ethernet connectivity.


SET Switch, created from SCVMM

• vNIC "MGMT"
• vNIC "SMB01" - vlan 201 > Team Mapping to pNIC1
• vNIC "SMB02" - vlan 202 > Team Mapping to pNIC2

Demo Part #3 - Demo with one vlan/subnets


The demo shows a Windows Server 2019 cluster that has been configured with one vlan/subnet. The subnet exist in all physical switchs, if the vNIC (SMB vNIC) has to move to the second physical NIC it will have connectivity.


SET Switch, created from SCVMM

• vNIC "MGMT"
• vNIC "SMB01" - vlan 18 > Team Mapping to pNIC1
• vNIC "SMB02" - vlan 18 > Team Mapping to pNIC2

Replace Node - Troubleshooting series

Replace Node - Troubleshooting series

*** Disclaimer ***
s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment.
For any reference links to other websites we encourages you to read the privacy statements of the third-party websites.
The names of actual companies and products mentioned herein may be the trademarks of their respective owners.
***

Replace Storage Spaces Direct Cluster Node - Troubleshooting series for Storage Spaces Direct (S2D)

4 Node Storage Spaces Direct (S2D) Cluster with failed physical disks

• S048001
• S048002 (Failed Node)
• S048003
• S048004

Part 1: Removed the failed disks from the Storage Pool and removed the Cluster Node

Part 2: Add the Node back to the Cluster

Windows Admin Center list the failed and retired disks

Remove the failed disk from the Storage Pool
Get-StorageNode -Name S048002.contoso.com | Get-PhysicalDisk -PhysicallyConnected | sort PhysicalLocation | ft SerialNumber, OperationalStatus, PhysicalLocation

$Disk = Get-PhysicalDisk |? Usage -like Retired

# Remove from Pool Get-StoragePool *S2D* | Remove-PhysicalDisk -PhysicalDisks $Disk

Test RDMA Connectivity

Test RDMA Connectivity

*** Disclaimer ***
s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment.
For any reference links to other websites we encourages you to read the privacy statements of the third-party websites.
The names of actual companies and products mentioned herein may be the trademarks of their respective owners.
***

Test RDMA Connectivity
Note: The Test-RDMA Script is a connectivity tools and not for performance tests...

Use the Microsoft Deployment Guide and Validate DCB Scripts for Host validation

• Windows Server 2016 and 2019 RDMA Deployment Guide
• Validate DCB

Download the DiskSpd.exe utility and extract into C:\Temp

• DiskSpd.exe utility

Download the Test-RDMA powershell script to C:\Temp

• Test-RDMA script

Test with a two Node cluster

• Node 1 = S048033
• Node 2 = S048034

S048033

• SMB01 = Interface Index 7 = 172.18.148.33
• SMB02 = Interface Index 9 = 172.18.248.33

S048034

• SMB01 = Interface Index 9 = 172.18.148.34
• SMB02 = Interface Index 8 = 172.18.248.34

Test from S048033 to S048034

Get-SmbClientNetworkInterface

.\Test-RDMA.PS1 -IfIndex 9 -IsRoCE $true -RemoteIpAddress 172.18.248.34 -PathToDiskspd C:\Temp\

.\Test-RDMA.PS1 -IfIndex 7 -IsRoCE $true -RemoteIpAddress 172.18.148.34 -PathToDiskspd C:\Temp\

DCB, PFC and ETS Configuration for RDMA/RoCE.

DCB, PFC and ETS Configuration for RDMA/RoCE.

*** Disclaimer ***
s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment.
For any reference links to other websites we encourages you to read the privacy statements of the third-party websites.
The names of actual companies and products mentioned herein may be the trademarks of their respective owners.
***

DCB, PFC and ETS Configuration for RDMA/RoCE.

Use the Microsoft Deployment Guide and Validate DCB Scripts for Host validation

• Windows Server 2016 and 2019 RDMA Deployment Guide
• Validate DCB

(01:15) Design overview
(02:00) Rename the pNICs to match the Microsoft standard names NIC1 and NIC2
(02:50) Disable RSC
(03:30) Enable DCB
(03:45) Clear previous configurations
(04:00) Disable the DCBx
(04:30) Create QoS policies and tag each type of traffic with the relevant priority
(05:15) Enable Priority Flow Control (PFC) on a specific priority. Disable for others
(05:40) Enable PFC for Priority 3, SMB Traffic
(05:50) Enable QoS on the relevant interface
(06:00) Enable Bandwidth of 50% for SMB when sharing with VMs
(06:35) Create vSwitch from PowerShell with SR-IOV
(07:48) Rename the vNIC for Management traffic from “vSwitch” to “MGMT”
(08:00) Add the vNIC for SMB/RDMA Traffic
(08:58) Rename to “short” name, rename the "vEthernet (xxx)"
(09:37) SET Switch need the -IeeePriorityTag to add the "Cluster" PriorityTag of 7 on vNIC/MGMT
(10:45) Enable RDMA for the vNICs
(11:17) Add IP Address to the vNICs for SMB/MGMT traffic 
(12:25) Add the vlan infromation for the SMB vNICs
(14:30) Jumbo Packet
(15:15) Disable NetBIOS
(15:22) Map the vNICs to pNICs
(16:07) SMB Bandwidth Limits
(17:45) Review vSwitch Settings
(19:00) Review DCB/PFC/ETS Settings
(20:03) Review pNICs RSS/VMQ Settings
(21:20) Optimise the use of LPs
(22:35) Review SR-IOV Settings
(23:35) Check that SMB Signing is off

***

Links to Scripts on GitHub WSlab:
S2D Converged with Live Migration with SMB Bandwidth Limits

***
PowerShell Commands used in the Video

# ***************************************************************************************************************************************

# Rename the pNICs to match Microsoft Standard names NIC1 and NIC2
Get-NetAdapter | sort Name | ft -AutoSize
Rename-NetAdapter -Name "Ethernet" -NewName NIC1
Rename-NetAdapter -Name "Ethernet #2" -NewName NIC2
Get-NetAdapter | sort Name | ft -AutoSize

# Disable RSC (if the Validate-DCB highlight this as needed for the pNICs)
Disable-NetAdapterRsc -Name NIC1
Disable-NetAdapterRsc -Name NIC2
Get-NetAdapterRsc | sort name | ft name, IPv4Enabled, IPv6Enabled, IPv4OperationalState, IPv6OperationalState -AutoSize

# Enable Data Center Bridging (DCB)
Install-WindowsFeature Data-Center-Bridging

# Clear previous configurations
Remove-NetQosTrafficClass
Remove-NetQosPolicy -Confirm:$False

# Disable the DCBx setting:
Set-NetQosDcbxSetting -Willing 0

# Create QoS policies and tag each type of traffic with the relevant priority
New-NetQosPolicy "Cluster" -Cluster -PriorityValue8021Action 7
New-NetQosPolicy "SMB" -NetDirectPortMatchCondition 445 -PriorityValue8021Action 3
New-NetQosPolicy "DEFAULT" -Default -PriorityValue8021Action 0

# Enable Priority Flow Control (PFC) on priority 3 (SMB Traffic). Disable for all others
Disable-NetQosFlowControl 0,1,2,4,5,6,7
Enable-NetQosFlowControl -Priority 3

# Enable QoS on the relevant interface
Enable-NetAdapterQos -InterfaceAlias "NIC1"
Enable-NetAdapterQos -InterfaceAlias "NIC2"

# Enable Bandwidth of 50% for SMB when sharing with VMs
New-NetQosTrafficClass "Cluster" -priority 7 -bandwidthpercentage 1 -algorithm ETS
New-NetQosTrafficClass "SMB" -Priority 3 -BandwidthPercentage 50 -Algorithm ETS

# Create vSwitch from PowerShell with SR-IOV
New-VMSwitch -Name vSwitch -NetAdapterName "NIC1","NIC2" -AllowManagementOS $true -EnableEmbeddedTeaming $true -EnableIov $true

# Rename the vNIC for Management traffic from “vSwitch” to “MGMT”
Rename-VMNetworkAdapter –ManagementOS –VMNetworkAdapterName vSwitch -NewName MGMT

# Add the vNIC for SMB/vNIC RDMA Traffic
Add-VMNetworkAdapter -ManagementOS -VMnetworkAdapterName SMB01 -SwitchName vSwitch
Add-VMNetworkAdapter -ManagementOS -VMnetworkAdapterName SMB02 -SwitchName vSwitch

# List the current Names of the vNICs
Get-VMNetworkAdapter -ManagementOS
Get-NetAdapter | sort Name | ft -AutoSize

# Rename to “short” name, rename the "vEthernet (xxx)"
Rename-NetAdapter -Name "*MGMT*" -NewName MGMT
Rename-NetAdapter -Name "*SMB01*" -NewName SMB01
Rename-NetAdapter -Name "*SMB02*" -NewName SMB02

Get-VMNetworkAdapter -ManagementOS
Get-NetAdapter | sort Name | ft -AutoSize


# SET Switch need the -IeeePriorityTag to add the "Cluster" PriorityTag of 7
Set-VMNetworkAdapter -ManagementOS -Name "MGMT" -IeeePriorityTag on
Set-VMNetworkAdapter -ManagementOS -Name "SMB01" -IeeePriorityTag on
Set-VMNetworkAdapter -ManagementOS -Name "SMB02" -IeeePriorityTag on

# Enable vRDMA for the vNICs
Enable-NetAdapterRdma *SMB*
Get-NetAdapterRdma | Sort Name | ft -AutoSize


# Add IP Address to the vNICs for SMB traffic 
New-NetIPAddress -InterfaceAlias SMB01 -IPAddress x.x.x.x -PrefixLength 24
New-NetIPAddress -InterfaceAlias SMB02 -IPAddress x.x.x.x -PrefixLength 24

New-NetIPAddress -InterfaceAlias MGMT -IPAddress x.x.x.x -PrefixLength 16 -DefaultGateway x.x.x.x
Set-DnsClientServerAddress -InterfaceAlias "MGMT" -ServerAddresses x.x.x.x,x.x.x.x

# Add the vlan infromation
Set-VMNetworkAdaptervlan –VMNetworkAdapterName SMB01 –vlanid xxx –Managementos –access
Set-VMNetworkAdaptervlan –VMNetworkAdapterName SMB02 –vlanid yyy –Managementos –access

# List the new Names and check the vlan information
Get-VMNetworkAdapter -ManagementOS
Get-VMNetworkAdapterVlan -ManagementOS

# Set jumbo Packet to 9014
Get-NetAdapter | where {$_.Name -like "NIC1" -or $_.Name -like "NIC2" -or $_.Name -like "SMB*"} | Set-NetAdapterAdvancedProperty -RegistryKeyword '*JumboPacket' -RegistryValue '9014'

# Disable NetBIOS
$adapters=(gwmi win32_networkadapterconfiguration )
Foreach ($adapter in $adapters){
  Write-Host $adapter
  $adapter.settcpipnetbios(2)
}

# Map the vNICs to pNICs
Set-VMNetworkAdapterTeamMapping -VMNetworkAdapterName "SMB01" –ManagementOS –PhysicalNetAdapterName "NIC1"
Set-VMNetworkAdapterTeamMapping -VMNetworkAdapterName "SMB02" –ManagementOS –PhysicalNetAdapterName "NIC2"
Get-VMNetworkAdapterTeamMapping -ManagementOS

# ***************************************************************************************************************************************
# Note: SMB Bandwidth Limits

Install-WindowsFeature FS-SMBBW
Set-SmbBandwidthLimit -Category LiveMigration -BytesPerSecond 300MB

# ***************************************************************************************************************************************
# Review vSwitch Settings
Get-NetAdapter | sort Name | ft -AutoSize
Get-VMSwitch -Name vSwitch
Get-VMSwitch -Name vSwitch | ft NetAdapterInterfaceDescriptions
Get-VMSwitch -Name vSwitch | fl BandwidthReservationMode

# ***************************************************************************************************************************************
# Review DCB/PFC/ETS Settings
Get-NetAdapterRdma | Sort Name | ft -AutoSize
Get-VMNetworkAdapterTeamMapping -ManagementOS
Get-VMNetworkAdapter -ManagementOS
Get-VMNetworkAdapterVlan -ManagementOS
Get-NetQosDcbxSetting
Get-NetQosFlowControl
Get-NetQosPolicy
Get-NetQosTrafficClass | ft -AutoSize
Get-NetAdapterQos -Name NIC1
Get-NetAdapterQos -Name NIC2

# ***************************************************************************************************************************************
# Review pNICs RSS/VMQ Settings
Get-NetAdapter | sort Name | ft -AutoSize
Get-NetAdapterHardwareInfo | sort Name | ft -AutoSize

# Optimise the use of LPs (This examples only move away from LP 0, more advance configuration can sometime be needed)
Set-NetAdapterVmq -Name NIC1 -BaseProcessorNumber 2
Set-NetAdapterVmq -Name NIC2 -BaseProcessorNumber 2

Get-NetAdapterVmq | sort Name | ft -AutoSize
Get-NetAdapterRss | sort Name | ft -AutoSize
Get-NetAdapterRss -Name NIC1
Get-NetAdapterRss -Name NIC2

# ***************************************************************************************************************************************
# Review SR-IOV Settings
Get-NetAdapterSriov -Name NIC1,NIC2| sort name | ft -AutoSize
Get-NetAdapterSriovVf | sort name | ft -AutoSize
Get-NetAdapterSriovVf | sort VmFriendlyName | ft VmFriendlyName, name, FunctionID, VPortID -AutoSize

# ***************************************************************************************************************************************
# Check that SMB Signing is off (False)
Get-SmbServerConfiguration | fl EncryptData, RequireSecuritySignature

# List Metric for vNIC used for Cluster traffic
Get-ClusterNetwork | ft Name, Metric, AutoMetric, Role -AutoSize

# ***************************************************************************************************************************************

Monitor RoCE - Mellanox Switch SX1012

Monitor RoCE and configuration examples for Mellanox Switch SX1012

*** Disclaimer ***
s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment.
For any reference links to other websites we encourages you to read the privacy statements of the third-party websites.
The names of actual companies and products mentioned herein may be the trademarks of their respective owners.
***

Performance Monitor and configuration examples for Mellanox Switch SX1012

Monitor Priority and Pause Frames for RDMA (RoCE) on Mellanox Switch SX1012

• Blog on Mellanox SN2100 

For the screen captures in the blog I used the Mellanox Switch SX1012

• Mellanox SX1012 (SwitchX - Onyx 3.6.8010)

******************************************************************************
Mellanox Show Commands for DCB, PFC and ETS
******************************************************************************

show dcb priority-flow-control
show dcb priority-flow-control detail
show dcb ets

Review Interface
show dcb priority-flow-control interface ethernet 1/3
show dcb ets interface ethernet 1/3
show interfaces ethernet 1/3 counters priority 3

References

Microsoft:

• Windows Server 2016 and 2019 RDMA Deployment Guide
• Validate DCB

Mellanox L2 PCP TC:

• How to Install Windows Server 2016 with RoCEv2 and Switch Embedded Teaming over HA Mellanox Network Solution
• Understanding Traffic Class (TC) Scheduling on Mellanox Spectrum Switches (WRR,SP)
• Understanding RoCEv2 Congestion Management
• HowTo Configure PFC on ConnectX-4

Terminology:
pNIC Physical NIC, the physical hardware that exchanges packets with the TOR
vNIC    Host vNIC – Virtual NIC from vSwitch exposed in the host partition
tNIC Host tNIC - Team Interface NIC from LBFO Team 
vmNIC Virtual Machine NIC – Virtual NIC from vSwitch exposed in a guest partition
vSwitch Hyper-V virtual switch
SET Switch Embedded Teaming, Hyper-V virtual switch supported in Windows Server 2016 and 2019

ToR Top of Rack switch
RDMA Remote Direct Memory Access
RoCE    RDMA over Converged Ethernet
RoCEv2 2nd generation RoCE using UDP/IP for routability (a.k.a. Routable RoCE)

DCB     Data Center Bridging
LLDP    Link Layer Data Protocol
DCBx    Data Center Bridging Capability Exchange protocol (DCBX) is an extension of LLDP.
PFC     Priority Flow Control
ETS     Enhanced Transmission Service
TC      Traffic Class
ECN     Explicit Congestion Notification
RED     Random Early Detection
CNP     Congestion Notification Packet. CNP control frames (congestion ACK)
SP      Strict Priority
WRR     Weighted Round Robin


******************************************************************************
Mellanox Switch configuration example (L2):
******************************************************************************

Mellanox SX1012

enable
configure terminal 
interface ethernet 1/1-1/12 flowcontrol send off force
interface ethernet 1/1-1/12 flowcontrol receive off force

dcb priority-flow-control enable force
dcb priority-flow-control priority 3 enable
(Priority 3 is used for Storage (SMB) traffic)
interface ethernet 1/1-1/12 dcb priority-flow-control mode on force
interface ethernet 1/1 switchport mode hybrid
(Need to be repeated for each port)
S2D# interface ethernet 1/x switchport hybrid allowed-vlan all
Need to be repeated for each port. Only allow the needed vlans, the "allowed-vlan all" is not recommended, this is from a lab/test system).
dcb ets tc bandwidth 49 50 0 1
(More infromation below)
exit
write memory

******************************************************************************
Mellanox Switch Disable PFC

******************************************************************************

no dcb priority-flow-control priority 3 enable

******************************************************************************
Mellanox SX1012 (SwitchX)
******************************************************************************
show dcb priority-flow-control detail

******************************************************************************

show dcb priority-flow-control
show dcb priority-flow-control interface ethernet 1/1

******************************************************************************

******************************************************************************
show dcb ets
******************************************************************************

******************************************************************************

For the PFC enabled priorities we need to use the lossless TCs

So for the Microsoft use cases we change the default to:
Priority 0 Default traffic 49%
Priority 3 SMB traffic 50%
Priority 7 Cluster traffic 1%

dcb ets tc bandwidth 49 50 0 1

******************************************************************************
show dcb ets interface ethernet 1/1

******************************************************************************
show interfaces ethernet 1/3 counters priority 3
******************************************************************************