Fault domain awareness

 Fault domain awareness in Microsoft Failover Cluster

*** Disclaimer ***
s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment.
For any reference links to other websites we encourages you to read the privacy statements of the third-party websites.
The names of actual companies and products mentioned herein may be the trademarks of their respective owners.
***

Fault domain awareness in Microsoft Failover Cluster

PowerShell used in the Video

# Cluster and Node information

Get-ClusterNode | sort name | ft Name, State, StatusInformation, FaultDomain

Get-ClusterNode | sort name | ft Name, State, StatusInformation, FaultDomain | Out-File -FilePath C:\Temp\Review\Get-ClusterNode-ft.txt

Get-ClusterNode | sort name | fl *

Get-ClusterNode | sort name | fl * | Out-File -FilePath C:\Temp\Review\Get-ClusterNode-fl.txt

Start-Cluster

Stop-Cluster

Start-ClusterNode –ClearQuarantine

cls

# *************************************************************************************************************

# Fault domain awareness

# https://docs.microsoft.com/en-us/windows-server/failover-clustering/fault-domains

#

# Defining fault domains with PowerShell

# *************************************************************************************************************

Get-cluster | fl AutoAssignNodeSite

(Get-Cluster).AutoAssignNodeSite=1

Get-ClusterFaultDomain | sort name | ft -AutoSize

Get-ClusterFaultDomain | sort name | ft -AutoSize | Out-File -FilePath C:\Temp\Review\Get-ClusterFaultDomain-ft.txt

Get-ClusterFaultDomain | sort name | fl * 

Get-ClusterFaultDomain | sort name | fl * | Out-File -FilePath C:\Temp\Review\Get-ClusterFaultDomain-fl.txt

New-ClusterFaultDomain -Type Site -Name "Copenhagen" -Description "Microsoft Denmark"

New-ClusterFaultDomain -Type Rack -Name "RACK45"

New-ClusterFaultDomain -Type Rack -Name "RACK46"

New-ClusterFaultDomain -Type Rack -Name "RACK47"

New-ClusterFaultDomain -Type Rack -Name "RACK48"

Set-ClusterFaultDomain -Name "Copenhagen","RACK45", "RACK46", "RACK47", "RACK48" -Location "Bulding 92 Room 1"

Set-ClusterFaultDomain -Name "RACK45", "RACK46", "RACK47", "RACK48" -Parent "Copenhagen"

Set-ClusterFaultDomain -Name "S047011","S047012","S047013" -Parent "RACK47"

Set-ClusterFaultDomain -Name "S047014","S047015","S047016" -Parent "RACK48"

# *************************************************************************************************************

# Remove fault domains

# *************************************************************************************************************

Set-ClusterFaultDomain -Name "S047011","S047012","S047013","S047014","S047015","S047016" -Parent ""

Set-ClusterFaultDomain -Name "RACK45", "RACK46", "RACK47", "RACK48" -Parent ""

Remove-ClusterFaultDomain -Name "Copenhagen","RACK45", "RACK46", "RACK47", "RACK48"

(Get-Cluster).AutoAssignNodeSite=0

# *************************************************************************************************************

DataON Azure Stack HCI - Public Preview

 Azure Stack HCI - Public Preview - Installation and Troubleshooting series with DataON

*** Disclaimer ***
s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment.
For any reference links to other websites we encourages you to read the privacy statements of the third-party websites.
The names of actual companies and products mentioned herein may be the trademarks of their respective owners.
***

Azure Stack HCI - Public Preview - Installation and Troubleshooting series with DataON

DataON Azure Stack HCI

• DataON Hosts
• Intel NVMe
• Mellanox Network

DataON Azure Stack HCI

Azure Stack HCI launch blog Windows Admin Center blog Azure Stack HCI Preview Access Virtual Lab Setup Evaluate Azure Stack HCI using Nested Virtualizatio

Part 1

Setup test Domain and Windows Admin Center

Installation of the physical DataON with the Azure Stack HCI - Public Preview

Configuration of network with one dual port Mellanox ConnectX-4 adapter for both Storage and Guest Traffic

Setup of VMs for Performance test with DiskSpd.exe

Se the impact on the Host CPUs for Storage, Network and Guest workload

(Video will come one day... when I have time to finish the editing)

Part 2

Configuration of network with two dual port Mellanox ConnectX-4 adapters. The Storage adapter are direct connected and the Guest Traffic use a SET switch connect to Mellanox SN2100 physical Switch

Notes and time agenda for the Video: 00:20 Agenda, Create one virtual switch for compute only and use direct connection for storage 00:45 Start the cleaning of the previous setup 00:50 Delete all the VMs (Was exported before the recording was started) 01:20 Delete all the vDisk 01:50 Disable-ClusterS2D the "destroy cluster" part is NOT included in this video... 02:10 Clear-Cluster Node was performed on all Nodes 02:44 Start the "Create new" "Server cluster" from WAC 02:56 WAC step 1.1 Check the prerequisites 03:05 WAC step 1.2 Add servers 04:35 WAC step 2.1 Verify network adapters (Remove Existing Switches, from last demo) 04:56 WAC step 2.2 Select management adapter (Use a 10Gbps pNIC, only 1Gbps connection in the Demo) 05:24 WAC step 2.3 Define networks (Add the Name, IP, Subnet and vlan for the Storage pNICs) 06:20 WAC step 2.4 Virtual Switch 07:00 WAC step 3.1 Validate cluster 07:30 WAC step 3.2 Create cluster 08:30 WAC step 4.1 Clean drives 09:18 WAC step 4.4 Enable Storage Spaces Direct More to come in the next videos about setup of vDisk and performance

Part 3

More to come

Rebuild Storage Spaces Direct (S2D) SOFS Cluster

Rebuild Storage Spaces Direct S2D and Scale-Out File Server (SOFS) - Troubleshooting series

*** Disclaimer ***
s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment.
For any reference links to other websites we encourages you to read the privacy statements of the third-party websites.
The names of actual companies and products mentioned herein may be the trademarks of their respective owners.
***

Rebuild Storage Spaces Direct S2D and Scale-Out File Server (SOFS) - Troubleshooting series

Rebuild a 4 Node Storage Spaces Direct (S2D) SOFS Cluster. Reconfigured the Network and use Validate-DCB to ensure the RoCE setup of DCB, PFC, ETS. Enable the S2D again on the "new" cluster Reuse/add the "old" Storage Pool back to the "new" cluster Add the "old" vDisks and share it again with the SOFS Role If it work, I get my VMs back... no need to restore from Backup... The "old" Cluster was deleted and one Host was complet reinstalled. The "Clear-ClusterNode" was used to clear the cluster configuration from a node. This ensure that the failover cluster configuration has been completely removed from a node. Sit back, enjoy 1 hour and 12 minutes of planned and unplanned challenges. Troubleshooting with Validate-DCB and the use of SCVMM/PowerShell to configure the Cluster Network

If you don't have time to see it all (recommended to see it all, it is fun to see me get into trouble) Then I have created some time agenda, that can help to jump in the Video 0:03:08 See the old disk Pool info. on one Host from "Disk Management" 0:04:05 SCVMM Add the Host to SCVMM 0:05:05 SCVMM Add the Logical Switch to the Host 0:06:43 See the vNIC be created on the Host from SCVMM 0:08:14 See the vNIC for Storage 0:09:02 See the configuration of DCB, PFC and ETS (PowerShell) 0:13:56 The new cluster dont use the same vlan as the old, will give problems later 0:14:10 Use the wrong vNIC name when configure the MTU (See if Validate-DCB will detect the mistake 0:24:54) 0:16:44 See how I make a mistake. Add the wrong Subnet Mask for my Management vNIC... 0:21:59 Validate-DCB Installation of the Module 0:22:55 Validate-DCB Run the first time... 0:24:54 Validate-DCB shows the MTU Error 0:26:54 Validate-DCB on all Hosts (Changed the script in Notepad) 0:28:18 Validate-DCB Use the wrong Policy Name for the SMB 0:29:38 Validate-DCB NetQosPolicy Name changed 0:31:54 Validate-DCB ETS Traffic Class missing 0:34:00 Validate-DCB Okay 0:37:05 Found the Subnet error on the Management vNIC 0:40:01 Cluster wizard failed first time 0:41:32 Cluster wizard failed second time 0:42:02 Cluster Wizard failed again 0:42:56 Cluster Wizard started again, this time it works... 0:45:07 Cluster created (Back in the game) 0:46:10 Enable Storage Spaces Direct S2D again (cross the finger that is accept the old pool) 0:48:08 Add the "old" Storage Pool back to the "new" Cluster 0:48:26 Add the "old" Disk back to the "new" Cluster (vDisk 1 to 3) 0:49:21 Rename vDisks after add to the cluster 0:51:14 Add SOFS03 Role 0:55:42 SOFS Role created for "SOFS03" 0:56:09 Delegation of Control to the CNO 0:56:53 Add File Share for vDisk2 0:58:00 Add File Share for vDisk2 0:59:10 Add File Share for vDisk3 1:01:35 SCVMM Remove the old SOFS Provider from SCVMM 1:01:43 SCVMM Add the rebuild SOFS 1:05:04 SCVMM Add the vDisk Access Control 1:06:31 SCVMM Missing the "File Share managed by Virtual Machine Manager" 1:07:02 SCVMM Add the File Share Storage vDisk1 1:08:58 SCVMM Repair the vDisk2 File Share Storage connection 1:09:06 SCVMM Add the File Share Storage vDisk3 1:09:28 SCVMM File Share Storage all vDisks are now green 1:09:50 Explain the last problems (vlan and SOFS Client access) 1:11:00 Start the first VM *** Links: DCB, PFC and ETS Configuration for RDMA/RoCE.

https://www.s2d.dk/2019/12/dcb-configuration.html Validator for RDMA Configuration and Best Practices

https://www.s2d.dk/2019/10/validator-for-rdma-configuration-and.html

Microsoft Docs version of the Blog is now released Validate an Azure Stack HCI cluster https://docs.microsoft.com/en-us/azure-stack/hci/deploy/validate Validate-DCB Disconnected installation

https://www.s2d.dk/2019/10/validate-dcb-disconnected-installation.html

AzSHCI Installation and Troubleshooting

Azure Stack HCI - Public Preview - Installation and Troubleshooting

*** Disclaimer ***
s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment.
For any reference links to other websites we encourages you to read the privacy statements of the third-party websites.
The names of actual companies and products mentioned herein may be the trademarks of their respective owners.
***

Azure Stack HCI - Public Preview - Installation and Troubleshooting You can see the challenges I ran into in my Live Microsoft Team Session today... Installation with dual boot on vhdx, planned and unplanned "Oops..." Troubleshooting with Windows Admin Center Azure Stack HCI launch blog: aka.ms/AzureStackHciPreviewBlog

Windows Admin Center blog: aka.ms/wac2007 Azure Stack HCI Preview Access: azure.com/HCI Lab Setup: aka.ms/wslab Evaluate Azure Stack HCI using Nested Virtualizatio https://github.com/Azure/AzureStackHCI-EvalGuide

***

Part 1

Help navigate the 2 hours session, jump direct to: Download Azure Stack HCI - Public Preview 10:40 Dual boot on vhdx command file changed 11:00 Build the vm/vhdx from the ISO 26:20 vhdx boot - Oops... 34:00 Convert vhdx 37:45 Installation Windows Admin Center 2007 (Support for Azure Stack HCI Preview) 49:28 Choose the cluster type - Cluster wizard 1:07:19 Network problems - planned Troubleshooting (Not using the correct vlan) 1:11:47 The Physical Network Adapter can connect... 1:26:00 Missing vlans - planned Troubleshooting (Physical Switch dont have all the vlan in each Switch) 1:12:01 Virtual Switch 1:13:38 Failed the one vNIC - planned Troubleshooting 1:17:17 Login to the Physical Switch to change the vlan alo 1:28:00 Network passed Virtual Switch - planned Troubleshooting 1:29:45 Validate the cluster 1:30:00 Error in cluster validation 1:32:00 WinRM unplanned Troubleshooting and workaround... 1:39:20 Validate the cluster "Test report isn't available, try to validate again"

Part 2 Fixed the "Test report isn't available, try to validate again" 1. Connect to the Host 2. Get-WinSystemLocale 3. Set-WinSystemLocale -SystemLocale en-US 4. Reboot the Host 5. Validate the cluster again... 6. :-)

Storage Maintenance Mode

Storage Maintenance Mode impact when a Node reboot

*** Disclaimer ***
s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment.
For any reference links to other websites we encourages you to read the privacy statements of the third-party websites.
The names of actual companies and products mentioned herein may be the trademarks of their respective owners.
***


Draft work in progress


Storage Maintenance Mode impact when a Node reboot

What is the Impact or benefit to have Storage Maintenance Mode enabled when a Node reboot

Is there a benefit ? let's test...


Demo Setup with 6 Node S2D Cluster and 6 vDisk/CSVs.
18 x Guest VMs (Windows Server 2012R2, 2016 and 2019)

Part 1 - Preparation

Rebalacing after Node join back to the cluster and disable Storage Maintenance Mode


Video will come

Part 2

Video will come

# ************************************************************************************** #
PowerShell used in the Demo Videos
# ************************************************************************************** #

Get-VirtualDisk | sort FriendlyName

Get-StoragePool -FriendlyName "S2D on S2D10"
Optimize-StoragePool -FriendlyName "S2D on S2D10"

Get-StorageJob

# ************************************************************************************** #
# List the Operational Status of the Extens, for all vDisks
# ************************************************************************************** #

$vDisk=(Get-VirtualDisk).FriendlyName

# Operational Status for the Extens
foreach ($vDisknumber in $vDisk) {
$VirtualDisk1 = Get-VirtualDisk "$vDisknumber"
$Extents = Get-VirtualDisk "$vDisknumber" | Get-PhysicalExtent | Where VirtualDiskUniqueId -Eq $VirtualDisk1.UniqueId
$Extents | Group OperationalStatus -NoElement
$vDisknumber
}

# Copy Number
foreach ($vDisknumber in $vDisk) {
$VirtualDisk1 = Get-VirtualDisk "$vDisknumber"
$Extents = Get-VirtualDisk "$vDisknumber" | Get-PhysicalExtent | Where VirtualDiskUniqueId -Eq $VirtualDisk1.UniqueId
$Extents | Group CopyNumber -NoElement
$vDisknumber
}

# Size
foreach ($vDisknumber in $vDisk) {
$VirtualDisk1 = Get-VirtualDisk "$vDisknumber"
$Extents = Get-VirtualDisk "$vDisknumber" | Get-PhysicalExtent | Where VirtualDiskUniqueId -Eq $VirtualDisk1.UniqueId
$Extents | Group Size -NoElement
$vDisknumber
}

# ************************************************************************************** #

Get-VirtualDisk "vDisk1" | Get-PhysicalExtent
Get-VirtualDisk "vDisk1" | Get-PhysicalExtent | Select-Object -First 24

# ************************************************************************************** #
# MetaData
# ************************************************************************************** #
Get-StoragePool -FriendlyName s2d*   | Get-PhysicalDisk -HasMetadata
Get-VirtualDisk -FriendlyName vDisk1  | Get-PhysicalDisk -HasMetadata 
Get-VirtualDisk -FriendlyName vDisk2  | Get-PhysicalDisk -HasMetadata 
Get-VirtualDisk -FriendlyName vDisk3  | Get-PhysicalDisk -HasMetadata
Get-VirtualDisk -FriendlyName vDisk4  | Get-PhysicalDisk -HasMetadata 
Get-VirtualDisk -FriendlyName vDisk5  | Get-PhysicalDisk -HasMetadata 

Get-VirtualDisk -FriendlyName vDisk6  | Get-PhysicalDisk -HasMetadata

S2D and DiskSpd

Azure Stack HCI / Storage Spaces Direct (S2D) and Performance test with DiskSpd

*** Disclaimer ***
s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment.
For any reference links to other websites we encourages you to read the privacy statements of the third-party websites.
The names of actual companies and products mentioned herein may be the trademarks of their respective owners.
***

Azure Stack HCI / Storage Spaces Direct (S2D) and Performance test with DiskSpd Shows how to copy and run DiskSpd in all VMs at the same time... Easy way to test your Storage Performance for Microsoft Storage Spaces Direct (S2D) Cluster

Links for download DiskSpd, VMFleet and Command help on GitHub

• DiskSpd: is a storage load generator 
• DiskSpd: Command line and parameters
• VMFleet

You can use watch-cluster.ps1 from VMFleet to see the IOPS from the VMs impact...

watch-cluster

Copy the Scripts from the link below and save as watch-cluster.ps1 in "C:\Temp\watch-cluster.ps1"
Then run is with the option "-sets *"
.\watch-cluster.ps1 -sets *

Link:

https://github.com/Microsoft/diskspd/blob/master/Frameworks/VMFleet/watch-cluster.ps1
Script:

*** Note: Before you run the test with DiskSpd set the CSV cache to 0. DiskSpd create synthetic workloads on Windows Server 2019 Clusters or later (including Storage Spaces Direct S2D). The Read only cache causes performance slowness with synthetic workloads, however with real workloads it increases performance.

# ****************************************************************************************************
# Copy the files to the VMs
# ****************************************************************************************************
$clusterName = "S2D10"
$Servers=(Get-ClusterNode -Cluster $ClusterName).Name
$VMs=(Get-VM -ComputerName $Servers).Name
$VMs
$source = "\\TFTP01\data\CopyFileToVMs\*.*"
$destination = "C$\Temp\"
foreach ($computer in $VMs) {
if ((Test-Path -Path \\$computer\$destination)) {
Copy-Item $source -Destination \\$computer\$destination -Recurse
} else {
"\\$computer\$destination is not reachable or does not exist"
}
}
# ****************************************************************************************************
# Run DiskSpd.exe in all VMs for 120s 
# ****************************************************************************************************
Invoke-Command -ComputerName $VMs -AsJob -ScriptBlock {c:\temp\diskspd.exe -d120 -r -w0 -t4 -o8 -b8k -Su -L -c10G c:\temp\io.dat > c:\temp\ResultsDiskSpd-Sb.txt}
# ****************************************************************************************************

Scale-Out File Server (SOFS) Demo Part 2 - Troubleshooting series

Scale-Out File Server (SOFS) Demo Part 2 - Troubleshooting series

*** Disclaimer ***
s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment.
For any reference links to other websites we encourages you to read the privacy statements of the third-party websites.
The names of actual companies and products mentioned herein may be the trademarks of their respective owners.
***

S2D Troubleshooting Series - Scale-Out File Server (SOFS) Demo Part 2 SOFS Demo Part 2: Demo Storage Spaces Direct (S2D), Scale-Out File Server (SOFS) and Hyper-V Cluster


Part 2: The Demo shows the impact of "crashing" one of the S2D Cluster Nodes, owning the Storage Pool and one of the vDisks hosting VHDX for the VMs connecting to the SOFS Role.

Part 2 also include some steps to replace/add a failed Disk to one of the S2D Cluster Nodes
Demo of some Microsoft Tools:

• Windows Admin Center
• NotMyFault
• Scale-Out File Server Improvements in Windows Server 2019