Featured Post

Links library

2018/07/04

Links library

s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

News

Storage Spaces Direct (S2D)

Storage

Micron Tools

Network

Microsoft Tools

Mellanox
Chelsio

Cisco

Dell



HPE

Virtual Disks resources are in No Redundancy or Detached status in a Storage Spaces Direct cluster

s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

The following are done in S2D test/demo environment with 4 Cluster Nodes.

The notes shows some of the steps from the Microsoft Support Help
The vDisk4 is Offline/Detached and can not go online
(Conscious actions have been taken to bring the vDisk in this failed state for the Demo)



  1. The vDisk4 is offline (Detached)
  2. Move the Disk to local Host and remove Cluster Share Volume (Local Disk)
  3. Change the Settings for the vDisk to run chkdsk or recovery
  4. Bring the vDisk online (Local Disk)
  5. Check the Scheduler
  6. Start the Scheduler and check that it is running
  7. Check the Storage Job
  8. Check the Virtual Disk Status
    • Each picture show the process of Repair Job
      • Active
      • Stale Metadata
      • Need Realloaction
      • Need Regeneration
      • Regeneration
      • Pending Deletion
  9. After 1 hour and 10 minutes the disk is OK again all 600 Extents are Active
  10. The 3 Way Mirror (3x200 Extents) for more information see the XXX (Link will be added)
  11. Offline the disk (Local Disk)
  12. Change the Settings for the vDisk back (Disable the chkdsk/recovery)
  13. Status before online the disk again after add back to cluster Share Volume
  14. Disk is online again
PowerShell:

Remove-clustersharedvolume -name "vDisk4"
Get-ClusterResource -Name "vDisk4" | Set-ClusterParameter -Name diskrunchkdsk -Value 7
Get-ClusterResource -Name "vDisk4" | Set-ClusterParameter -Name DiskRecoveryAction -Value 1
Start-clusterresource -Name "vDisk4"
Get-ScheduledTask -TaskName "Data Integrity Scan for Crash Recovery"
Get-ScheduledTask -TaskName "Data Integrity Scan for Crash Recovery" | Start-ScheduledTask
Get-ScheduledTask -TaskName "Data Integrity Scan for Crash Recovery"
Stop-clusterresource -Name "vDisk4"
Get-ClusterResource -Name "vDisk4" | Set-ClusterParameter -Name diskrecoveryaction -Value 0
Get-clusterresource "vDisk4" | set-clusterparameter -name diskrunchkdsk -value 0
Add-clustersharedvolume -name "vDisk4"


Monitor the Storage job process:

Get-VirtualDisk "vDisk4"
Get-StorageJob
$VirtualDisk4 = Get-VirtualDisk "vDisk4"
$Extents4 = Get-VirtualDisk "vDisk4" | Get-PhysicalExtent | Where VirtualDiskUniqueId -Eq $VirtualDisk4.UniqueId

$Extents4 | Group OperationalStatus -NoElement



You can also use Get-VirtualDisk | Get-PhysicalExtent to see exactly every extent. For example:


Get-StoragePool S2D* | Get-PhysicalDisk | Sort Model, OperationalStatus
Get-VirtualDisk "vDisk1"
$VirtualDisk = Get-VirtualDisk "vDisk1"
$Extents = Get-VirtualDisk "vDisk1" | Get-PhysicalExtent | Where VirtualDiskUniqueId -Eq $VirtualDisk.UniqueId
$Extents | Group OperationalStatus -NoElement
$Extents | Group Size -NoElement
$Extents | Group CopyNumber -NoElement


Get-VirtualDisk "vDisk1" | Get-PhysicalExtent
(Note: If the size of the vDisk is over 2-3TB the PowerShell can not show the information)

































Mellanox SR-IOV


Enable Virtual Switch with SR-IOV for Mellanox ConnectX-3 Pro and Mellanox ConnectX-5

s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.
  1. List the Physical Network Adapter in the Host. NIC1, NIC2 and NIC3 is used in this blogpost. (NIC3 and NIC4 is used for the Chelsio blogpost #link will be add#)
    Get-NetAdapter | sort Name | ft Name, InterfaceDescription, LinkSpeed -AutoSize
  2. List the default number of VF in the OS
    (Note: This is not the same as the configured)
    Get-NetAdapterSriov | sort name | ft Name, InterfaceDescription, SriovSupport, NumVFs -AutoSize
  3. List the Mellanox ConnectX-3 Pro numbers of VF enabled for each pNIC Port.
    Mellanox ConnectX-3 Pro support 32 VF in total. For more information about the configuration of the SR-IOV in the pNIC BIOS and Speed, ses the blogpost #link will be add#)
    Mellanox 56 Gigabit
    Get-MlnxPCIDeviceSriovSetting
  4. The Mellanox ConnectX-5 support up to 1000 VF, the Physical Server don´t support that, therefor we have to change the number to a supported value. We used the Mellanox Tools to get the PCI Device Name. Link to Mellanox Tool
  5. PCI Device Name for the Mellanox ConnectX-5
    mlxfwmanager.exe --query
  6. Change the numbers of VF to 64 (1-127 is supported)
    mlxconfig.exe -d mt4119_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=64More information in Mellanox Firmware Tools (MFT) User Manual
  7. List the current configuration and the new values
  8. Reboot
  9. Create the Virtual Switch "iovSwitch1" with the two Mellanox ConnetX-3 Pro ports
    New-VMSwitch -Name iovSwitch1 -NetAdapterName NIC1, NIC2 -AllowManagementOS $false -EnableEmbeddedTeaming $true -EnableIov $true
  10. Create the Virtual Switch "iovSwitch3" with the two Mellanox ConnetX-5
    New-VMSwitch -Name iovSwitch3 -NetAdapterName NIC5 -AllowManagementOS $true -EnableEmbeddedTeaming $true -EnableIov $true
  11. List the two Virtual Switch with SR-IOV enabled
  12. Only the iovSwitch3 have "-AllowManagement $true"
  13. The vNIC created for Management Access (Default name is the vSwitch name)
    Get-VMNetworkAdapter -ManagementOS
  14. Rename the vNIC iovSwitch3 to MGMT
    Rename-VMNetworkAdapter -ManagementOS -VMNetworkAdapterName iovSwitch3 -NewName MGMT
    Rename-NetAdapter "*MGMT*" MGMT
  15. List the pNIC/vNIC now with the MGMT name for the vNIC
    Get-NetAdapter | sort Name | ft Name, InterfaceDescription, LinkSpeed -AutoSize
  16. Show the new Virtual Switch with the correct numbers of VF
    Get-NetAdapterSriov | sort name | ft Name, InterfaceDescription, SriovSupport, NumVFs -AutoSize








Mellanox 56 Gigabit

56 Gigabit enabled on Mellanox SX1012 Switch

s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.
  1. The Windows Server Network Adapter Speed is listed as 40 Gigabit
  2. Login to the Mellanox Switch and go to Setup
  3. Select Licensing
  4. Type the Licens Key
  5. Select Add Licenses
  6. Eth port SW speed limit now list 56 Gigabit
  7. Save
  8. Check that the transceiver/cable support 56 Gigabit
  9. Select the Speed dropdown list (current speed 40 Gigabit)
  10. Change to 56 Gigabit and Save the change
  11. After setup the Actual speed show 56 Gigabit
  12. The Windows Server Network Adapter Speed is listed as 56 Gigabit












Remove Node

s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

The following Demo are performed in a S2D test environment with 6 Cluster Nodes.
Each Node have 5 SSDs used for the Storage Pool.

If you dont have access to Physical Lab Servers you can use WSLab and get some more information from the Scenarios


Task:
Remove Node from Cluster. Retire the Disk before the Node is removed.
(Note: The Node have a Hardware problem so it only list 4 of 5 SSDs and need to be replaced)

Columns:
Know you Columns count and free Disk space, before you start to remove Disk/Nodes. 
  • 6x5=30 Physical Disk
All vDisk are created with the default maximum columns count of 8 and 3 way Mirror
  • 3x8=24 Disk used for every Extents
When we remove a Cluster Node we go from 30 Physical disk to 25 Physical Disk so the Columns of 8 (Need 24 Disk) will not impact the Performance if we have free space on the remaining Physical Disk.

Before we remote the Node we create a new vDisk to follow the Extents 

PowerShell:
New-Volume -StoragePoolFriendlyName S2D* -FriendlyName vDisk4 -FileSystem CSVFS_ReFS -ProvisioningType Fixed -Size 200GB -Verbose
Get-VirtualDisk "vDisk4"
$VirtualDisk4 = Get-VirtualDisk "vDisk4"
$Extents4 = Get-VirtualDisk "vDisk4" | Get-PhysicalExtent | Where VirtualDiskUniqueId -Eq $VirtualDisk4.UniqueId
$Extents4 | Group OperationalStatus -NoElement
$Extents4 | Group CopyNumber -NoElement
$Extents4 | Group Size -NoElement


# Show the first Columns of Extents and Copyes (8x3=24)
Get-VirtualDisk "vDisk4" | Get-PhysicalExtent | Select-Object -First 24 

  1. Create a 200 GB vDisk (With the size information from below we need 200 Extents to create a 200GB vDisk)
  2. Show the total of Extents (3 way Mirror then we have 3x200 = 600 Extents)
  3. Show each copy of the Extents (200 in copy 0, 200 in copy 1 and 200 in copy 2)
    • The default will be 0,1,2 in a 3 way Mirror.
    • When we start to remove Disk/Node we will see copy 3,4 more information below
  4. Show the size of one of the Extents (1073741824 Bytes = 1024MB or 1GB)

The Cluster Node S047023 have some problems with one Physical Disk (The node only list 4 of the 5 Disks installed and need to be replaced)


Node S047023 only list the 4 of 5 diske. All 4 Disk are listed as Retired


After the Physical Disk have 0% of the Extents (Only 4 of 5 listed do to Hardware Problem and the Node need to be replaced)


Now it is time to remove the Node S047023 from the Cluster





  1. Show the Extents 600 Active Extents before retire the disk
  2. List the Extents in each Copy
  3. List the Storage job when the DIsks are retired
  4. List the Copy number in the process of the retire













Mellanox SR-IOV VF


Change the numbers of VFs for Mellanox ConnectX-5 (Draft, work in progress)


s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.


  1. List the current number of VFs
    • Get-NetAdapterSriov | sort name | ft Name, InterfaceDescription, SriovSupport, NumVFs -AutoSize
  2. List the PCI Device Name
    • mlxfwmanager.exe --query
  3. Example with 256 VF
    • mlxconfig.exe -d mt4119_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=254
  4. The 254 VFs are not supported
  5. Change the numbers of VF to 127
    • mlxconfig.exe -d mt4119_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=127
  6. 1-127 is supported
  7. List the current number of VFs after the reboot, list 123
    • Get-NetAdapterSriov | sort name | ft Name, InterfaceDescription, SriovSupport, NumVFs -AutoSize
  8. Review the NUM_OF_VF
    • mlxconfig.exe -d mt4119_pciconf0 q
  9. The NUM_OF_VF show 127
  10. List the VMs use of the VFs
    • Get-NetAdapterSriovVf | sort name | ft -AutoSize