Featured Post

YouTube and link library for S2D.dk

2020/01/04

DCB, PFC and ETS Configuration for RDMA/RoCE.

DCB, PFC and ETS Configuration for RDMA/RoCE.

*** Disclaimer ***
s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment.
For any reference links to other websites we encourages you to read the privacy statements of the third-party websites.
The names of actual companies and products mentioned herein may be the trademarks of their respective owners.
***

DCB, PFC and ETS Configuration for RDMA/RoCE.

Use the Microsoft Deployment Guide and Validate DCB Scripts for Host validation



(01:15) Design overview
(02:00) Rename the pNICs to match the Microsoft standard names NIC1 and NIC2
(02:50) Disable RSC
(03:30) Enable DCB
(03:45) Clear previous configurations
(04:00) Disable the DCBx
(04:30) Create QoS policies and tag each type of traffic with the relevant priority
(05:15) Enable Priority Flow Control (PFC) on a specific priority. Disable for others
(05:40) Enable PFC for Priority 3, SMB Traffic
(05:50) Enable QoS on the relevant interface
(06:00) Enable Bandwidth of 50% for SMB when sharing with VMs
(06:35) Create vSwitch from PowerShell with SR-IOV
(07:48) Rename the vNIC for Management traffic from “vSwitch” to “MGMT”
(08:00) Add the vNIC for SMB/RDMA Traffic
(08:58) Rename to “short” name, rename the "vEthernet (xxx)"
(09:37) SET Switch need the -IeeePriorityTag to add the "Cluster" PriorityTag of 7 on vNIC/MGMT
(10:45) Enable RDMA for the vNICs
(11:17) Add IP Address to the vNICs for SMB/MGMT traffic 
(12:25) Add the vlan infromation for the SMB vNICs
(14:30) Jumbo Packet
(15:15) Disable NetBIOS
(15:22) Map the vNICs to pNICs
(16:07) SMB Bandwidth Limits
(17:45) Review vSwitch Settings
(19:00) Review DCB/PFC/ETS Settings
(20:03) Review pNICs RSS/VMQ Settings
(21:20) Optimise the use of LPs
(22:35) Review SR-IOV Settings
(23:35) Check that SMB Signing is off

***

Links to Scripts on GitHub WSlab:
S2D Converged with Live Migration with SMB Bandwidth Limits

***
PowerShell Commands used in the Video

# ***************************************************************************************************************************************

# Rename the pNICs to match Microsoft Standard names NIC1 and NIC2
Get-NetAdapter | sort Name | ft -AutoSize
Rename-NetAdapter -Name "Ethernet" -NewName NIC1
Rename-NetAdapter -Name "Ethernet #2" -NewName NIC2
Get-NetAdapter | sort Name | ft -AutoSize

# Disable RSC (if the Validate-DCB highlight this as needed for the pNICs)
Disable-NetAdapterRsc -Name NIC1
Disable-NetAdapterRsc -Name NIC2
Get-NetAdapterRsc | sort name | ft name, IPv4Enabled, IPv6Enabled, IPv4OperationalState, IPv6OperationalState -AutoSize

# Enable Data Center Bridging (DCB)
Install-WindowsFeature Data-Center-Bridging

# Clear previous configurations
Remove-NetQosTrafficClass
Remove-NetQosPolicy -Confirm:$False

# Disable the DCBx setting:
Set-NetQosDcbxSetting -Willing 0

# Create QoS policies and tag each type of traffic with the relevant priority
New-NetQosPolicy "Cluster" -Cluster -PriorityValue8021Action 7
New-NetQosPolicy "SMB" -NetDirectPortMatchCondition 445 -PriorityValue8021Action 3
New-NetQosPolicy "DEFAULT" -Default -PriorityValue8021Action 0

# Enable Priority Flow Control (PFC) on priority 3 (SMB Traffic). Disable for all others
Disable-NetQosFlowControl 0,1,2,4,5,6,7
Enable-NetQosFlowControl -Priority 3

# Enable QoS on the relevant interface
Enable-NetAdapterQos -InterfaceAlias "NIC1"
Enable-NetAdapterQos -InterfaceAlias "NIC2"

# Enable Bandwidth of 50% for SMB when sharing with VMs
New-NetQosTrafficClass "Cluster" -priority 7 -bandwidthpercentage 1 -algorithm ETS
New-NetQosTrafficClass "SMB" -Priority 3 -BandwidthPercentage 50 -Algorithm ETS

# Create vSwitch from PowerShell with SR-IOV
New-VMSwitch -Name vSwitch -NetAdapterName "NIC1","NIC2" -AllowManagementOS $true -EnableEmbeddedTeaming $true -EnableIov $true

# Rename the vNIC for Management traffic from “vSwitch” to “MGMT”
Rename-VMNetworkAdapter –ManagementOS –VMNetworkAdapterName vSwitch -NewName MGMT

# Add the vNIC for SMB/vNIC RDMA Traffic
Add-VMNetworkAdapter -ManagementOS -VMnetworkAdapterName SMB01 -SwitchName vSwitch
Add-VMNetworkAdapter -ManagementOS -VMnetworkAdapterName SMB02 -SwitchName vSwitch

# List the current Names of the vNICs
Get-VMNetworkAdapter -ManagementOS
Get-NetAdapter | sort Name | ft -AutoSize

# Rename to “short” name, rename the "vEthernet (xxx)"
Rename-NetAdapter -Name "*MGMT*" -NewName MGMT
Rename-NetAdapter -Name "*SMB01*" -NewName SMB01
Rename-NetAdapter -Name "*SMB02*" -NewName SMB02

Get-VMNetworkAdapter -ManagementOS
Get-NetAdapter | sort Name | ft -AutoSize


# SET Switch need the -IeeePriorityTag to add the "Cluster" PriorityTag of 7
Set-VMNetworkAdapter -ManagementOS -Name "MGMT" -IeeePriorityTag on
Set-VMNetworkAdapter -ManagementOS -Name "SMB01" -IeeePriorityTag on
Set-VMNetworkAdapter -ManagementOS -Name "SMB02" -IeeePriorityTag on

# Enable vRDMA for the vNICs
Enable-NetAdapterRdma *SMB*
Get-NetAdapterRdma | Sort Name | ft -AutoSize


# Add IP Address to the vNICs for SMB traffic 
New-NetIPAddress -InterfaceAlias SMB01 -IPAddress x.x.x.x -PrefixLength 24
New-NetIPAddress -InterfaceAlias SMB02 -IPAddress x.x.x.x -PrefixLength 24

New-NetIPAddress -InterfaceAlias MGMT -IPAddress x.x.x.x -PrefixLength 16 -DefaultGateway x.x.x.x
Set-DnsClientServerAddress -InterfaceAlias "MGMT" -ServerAddresses x.x.x.x,x.x.x.x

# Add the vlan infromation
Set-VMNetworkAdaptervlan –VMNetworkAdapterName SMB01 –vlanid xxx –Managementos –access
Set-VMNetworkAdaptervlan –VMNetworkAdapterName SMB02 –vlanid yyy –Managementos –access

# List the new Names and check the vlan information
Get-VMNetworkAdapter -ManagementOS
Get-VMNetworkAdapterVlan -ManagementOS

# Set jumbo Packet to 9014
Get-NetAdapter | where {$_.Name -like "NIC1" -or $_.Name -like "NIC2" -or $_.Name -like "SMB*"} | Set-NetAdapterAdvancedProperty -RegistryKeyword '*JumboPacket' -RegistryValue '9014'

# Disable NetBIOS
$adapters=(gwmi win32_networkadapterconfiguration )
Foreach ($adapter in $adapters){
  Write-Host $adapter
  $adapter.settcpipnetbios(2)
}

# Map the vNICs to pNICs
Set-VMNetworkAdapterTeamMapping -VMNetworkAdapterName "SMB01" –ManagementOS –PhysicalNetAdapterName "NIC1"
Set-VMNetworkAdapterTeamMapping -VMNetworkAdapterName "SMB02" –ManagementOS –PhysicalNetAdapterName "NIC2"
Get-VMNetworkAdapterTeamMapping -ManagementOS

# ***************************************************************************************************************************************
# Note: SMB Bandwidth Limits

Install-WindowsFeature FS-SMBBW
Set-SmbBandwidthLimit -Category LiveMigration -BytesPerSecond 300MB

# ***************************************************************************************************************************************
# Review vSwitch Settings
Get-NetAdapter | sort Name | ft -AutoSize
Get-VMSwitch -Name vSwitch
Get-VMSwitch -Name vSwitch | ft NetAdapterInterfaceDescriptions
Get-VMSwitch -Name vSwitch | fl BandwidthReservationMode

# ***************************************************************************************************************************************
# Review DCB/PFC/ETS Settings
Get-NetAdapterRdma | Sort Name | ft -AutoSize
Get-VMNetworkAdapterTeamMapping -ManagementOS
Get-VMNetworkAdapter -ManagementOS
Get-VMNetworkAdapterVlan -ManagementOS
Get-NetQosDcbxSetting
Get-NetQosFlowControl
Get-NetQosPolicy
Get-NetQosTrafficClass | ft -AutoSize
Get-NetAdapterQos -Name NIC1
Get-NetAdapterQos -Name NIC2

# ***************************************************************************************************************************************
# Review pNICs RSS/VMQ Settings
Get-NetAdapter | sort Name | ft -AutoSize
Get-NetAdapterHardwareInfo | sort Name | ft -AutoSize

# Optimise the use of LPs (This examples only move away from LP 0, more advance configuration can sometime be needed)
Set-NetAdapterVmq -Name NIC1 -BaseProcessorNumber 2
Set-NetAdapterVmq -Name NIC2 -BaseProcessorNumber 2

Get-NetAdapterVmq | sort Name | ft -AutoSize
Get-NetAdapterRss | sort Name | ft -AutoSize
Get-NetAdapterRss -Name NIC1
Get-NetAdapterRss -Name NIC2

# ***************************************************************************************************************************************
# Review SR-IOV Settings
Get-NetAdapterSriov -Name NIC1,NIC2| sort name | ft -AutoSize
Get-NetAdapterSriovVf | sort name | ft -AutoSize
Get-NetAdapterSriovVf | sort VmFriendlyName | ft VmFriendlyName, name, FunctionID, VPortID -AutoSize

# ***************************************************************************************************************************************
# Check that SMB Signing is off (False)
Get-SmbServerConfiguration | fl EncryptData, RequireSecuritySignature

# List Metric for vNIC used for Cluster traffic
Get-ClusterNetwork | ft Name, Metric, AutoMetric, Role -AutoSize

# ***************************************************************************************************************************************