Featured Post

Links library

2018/07/04

Mellanox SR-IOV


Enable Virtual Switch with SR-IOV for Mellanox ConnectX-3 Pro and Mellanox ConnectX-5

s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.
  1. List the Physical Network Adapter in the Host. NIC1, NIC2 and NIC3 is used in this blogpost. (NIC3 and NIC4 is used for the Chelsio blogpost #link will be add#)
    Get-NetAdapter | sort Name | ft Name, InterfaceDescription, LinkSpeed -AutoSize
  2. List the default number of VF in the OS
    (Note: This is not the same as the configured)
    Get-NetAdapterSriov | sort name | ft Name, InterfaceDescription, SriovSupport, NumVFs -AutoSize
  3. List the Mellanox ConnectX-3 Pro numbers of VF enabled for each pNIC Port.
    Mellanox ConnectX-3 Pro support 32 VF in total. For more information about the configuration of the SR-IOV in the pNIC BIOS and Speed, ses the blogpost #link will be add#)
    Mellanox 56 Gigabit
    Get-MlnxPCIDeviceSriovSetting
  4. The Mellanox ConnectX-5 support up to 1000 VF, the Physical Server don´t support that, therefor we have to change the number to a supported value. We used the Mellanox Tools to get the PCI Device Name. Link to Mellanox Tool
  5. PCI Device Name for the Mellanox ConnectX-5
    mlxfwmanager.exe --query
  6. Change the numbers of VF to 64 (1-127 is supported)
    mlxconfig.exe -d mt4119_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=64More information in Mellanox Firmware Tools (MFT) User Manual
  7. List the current configuration and the new values
  8. Reboot
  9. Create the Virtual Switch "iovSwitch1" with the two Mellanox ConnetX-3 Pro ports
    New-VMSwitch -Name iovSwitch1 -NetAdapterName NIC1, NIC2 -AllowManagementOS $false -EnableEmbeddedTeaming $true -EnableIov $true
  10. Create the Virtual Switch "iovSwitch3" with the two Mellanox ConnetX-5
    New-VMSwitch -Name iovSwitch3 -NetAdapterName NIC5 -AllowManagementOS $true -EnableEmbeddedTeaming $true -EnableIov $true
  11. List the two Virtual Switch with SR-IOV enabled
  12. Only the iovSwitch3 have "-AllowManagement $true"
  13. The vNIC created for Management Access (Default name is the vSwitch name)
    Get-VMNetworkAdapter -ManagementOS
  14. Rename the vNIC iovSwitch3 to MGMT
    Rename-VMNetworkAdapter -ManagementOS -VMNetworkAdapterName iovSwitch3 -NewName MGMT
    Rename-NetAdapter "*MGMT*" MGMT
  15. List the pNIC/vNIC now with the MGMT name for the vNIC
    Get-NetAdapter | sort Name | ft Name, InterfaceDescription, LinkSpeed -AutoSize
  16. Show the new Virtual Switch with the correct numbers of VF
    Get-NetAdapterSriov | sort name | ft Name, InterfaceDescription, SriovSupport, NumVFs -AutoSize