Featured Post

Links library

2019/10/16

List Version and Reboot Time

s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

PowerShell Script to list the Windows Version and Last Reboot Time:

List my three S2D Cluster versions and the last reboot time

“Thanks to Steffen Brandsborg, Sentia Denmark for creating the PowerShell used below”

# Cluster S2D01
 $ComputerList = "S046031","S046032","S046033","S046034","S046035","S046036"
# Cluster S2D02
 $ComputerList += "S047027","S047028"
# Cluster S2D03
 $ComputerList += "S048001","S048002","S048003","S048004"

$InvokeResult = invoke-command -ComputerName $ComputerList -ScriptBlock {
    $BuildBranch = (Get-ItemProperty "HKLM:\Software\Microsoft\Windows NT\CurrentVersion").BuildBranch
    $CurrentVersion = (Get-ItemProperty "HKLM:\Software\Microsoft\Windows NT\CurrentVersion").CurrentVersion
    $CurrentBuildNumber = (Get-ItemProperty "HKLM:\Software\Microsoft\Windows NT\CurrentVersion").CurrentBuildNumber
    $UBR = (Get-ItemProperty "HKLM:\Software\Microsoft\Windows NT\CurrentVersion").UBR
    $BootTime = ((Get-WmiObject Win32_OperatingSystem).ConvertToDateTime((Get-WmiObject Win32_OperatingSystem).LastBootUpTime)).ToString("yyyy-MM-dd HH:mm:ss")
    $ProductName = (Get-ItemProperty "HKLM:\Software\Microsoft\Windows NT\CurrentVersion").ProductName

    $Result =  @{
        Computername = $env:COMPUTERNAME
        ProductName = $ProductName
        WindowsVersion = ("$CurrentVersion.$CurrentBuildNumber.$UBR")
        BootTime = $BootTime
    }
    New-Object -TypeName PSObject -Property $Result
}
$InvokeResult | Sort Computername | Select ComputerName, ProductName, WindowsVersion, BootTime | ft 




















2019/10/15

Validator for RDMA Configuration and Best Practices

s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

Validator for RDMA Configuration and Best Practices

More information on GitHub
How to do the Installation and first run in 5 minutes







Upgrade Storage Spaces Direct (S2D) from Windows Server 2016 to Windows Server 2019

s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

Upgrade Storage Spaces Direct (S2D) from Windows Server 2016 to Windows Server 2019

(DRAFT, work in progress last update 2019.10.17 New Screen Images and videos will be available within the next days)

How to upgrade a Storage Spaces Direct (S2D) cluster to Windows Server 2019. The four approaches to upgrading a S2D cluster from Windows Server 2016 to Windows Server 2019
  • In-place upgrade while VMs are running
  • Clean-OS installation while VMs are running
  • In-place upgrade while VMs are stopped
  • Clean-OS install while VMs are stopped
The Video shows the "Clean-OS installation while VMs are running" with the use of boot Host OS on VHDX (Dual boot lab systems)
The lab system used is a 6 Node S2D Cluster



After the upgrade enable Cluster Performance History



















2019/10/09

RoCE Troubleshooting

s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

(Updated 2019.10.09)

RoCE Troubleshooting

(DRAFT, work in progress last update 2019.10.09 New Screen Images and videos will be available within the next days)

The RDMA Activity Performance Counter can give you some useful information

RDMA Accepted Connections will increase for each new connection. The Picture show a lab system with a RoCE Error. RDMA Accepted Connections will normally only change if you move VMs (SOFS), disable/enable pNIC/vNIC or reboot a Node.

The picture show that I have thousands of Accepted Connections and after just a few seconds I have 4 Accepted Connections. With no change in the Cluster.

Active Connections

Show the Active Connections on each vNIC/pNIC to see the connect use the "netstat -xan"











Monitor RoCE - Windows Server

s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

Performance Monitor

Monitor Priority and Pause Frames for RDMA (RoCE) with Mellanox QoS

Examples of Physical Switch configuration for Mellanox, Cisco, HPE, Fujitsu and Dell

For all the physical switch examples, always get your hardware vendor to verify your settings before use.
  1. Mellanox
  2. Cisco
  3. HPE
  4. Fujitsu
  5. Dell
  6. Lenovo
Lab setup:
Storages Spaces Direct (S2D) clusters with 2,4 or 6 nodes, Windows Server 2016 and Windows Server 2019
    The Host and Switch need to be configured for DCB/PFC/ETS
    Note: Microsoft does not currently support DCBx/LLDP if you use SET. You need to be sure that the Switch is not set to "PFC Auto" on the used ports. But use "PFC Enable" or "PFC On"

    Use the Microsoft Deployment Guide and Validate DCB Scripts for Host validation
    In the Examples we use Priority 3 for Storage and Priority 7 for Cluster Traffic.
    • Storage Traffic = SMB 445 Traffic (Storage, CSV and Live Migration)
      • Storage Traffic need PFC/ETS
    • Cluster Traffic = UDP 3343 Traffic
      • Cluster Traffic only use the ETS
    In the lab I use Jumbo Frames for Ethernet and auto size for RDMA.
    (RDMA/RoCE Max Frame Size is in Auto Mode).

    Sent Pause Frames
    The total number of pause frames sent from this priority to the far-end port. The untagged instance indicates the number of global pause frames that were sent.
    Sent Pause Duration
    The total duration of packets transmission being paused on this priority in microseconds.
    Received Pause Frames
    The number of pause frames that were received to this priority from the far-end port. The untagged instance indicates the number of global pause frames that were received.
    Received Pause Duration
    The total duration that far-end port was requested to pause for the transmission of packets in microseconds.
    Sent Discard Frames 
    The number of packets discarded by the transmitter.  Note: this counter is per Traffic Class (TC) and not per priority.


    Performance Monitor:
    The pictures below shows the Priority 3 (SMB) and also Priority 7 (Cluster)
    (There need to be high load on the Physical Switch Port and/or Network Adapter before you will see that values change for Pause Frames).

    New-NetQosPolicy "Cluster" -Cluster -PriorityValue8021Action 7
    New-NetQosPolicy "SMB" -NetDirectPortMatchCondition 445 -PriorityValue8021Action 3
    New-NetQosPolicy "DEFAULT" -Default -PriorityValue8021Action 0

    Set-VMNetworkAdapter -ManagementOS -Name "SMB1" -IeeePriorityTag on
    Set-VMNetworkAdapter -ManagementOS -Name "SMB2" -IeeePriorityTag on




    Performance Monitor:
    The pictures below shows the Priority 3 (SMB) and Pause Frames (Rcv/Sent) on the pNICs with 10 minuttes between each other. We can see that the Frames increased, so the system sent and recived it correct. (There need to be high load on the Physical Switch Port and/or Network Adapter before you will see that values change).














    Monitor RoCE - Mellanox

    s2d.dk is not responsible for any errors, or for the results obtained from the use of this information on s2d.dk. All information in this site is provided as "draft notes" and "as is", with no guarantee of completeness, accuracy, timeliness or of the results obtained from the use of this information. Always test in a lab setup, before use any of the information in production environment. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

    (Updated 2019.10.09)


    Performance Monitor
    Monitor Priority and Pause Frames for RDMA (RoCE) on Mellanox Switch.

    (DRAFT, work in progress last update 2019.10.09 New Screen Images and videos will be available within the next days)


    Microsoft references:

    For the screen captures I use two different Mellanox Switch.
    • Mellanox SX1012 (SwitchX - Onyx 3.6.8010)
    • Mellanox SN2100 (Spectrum - Onyx 3.6.8010)
    • The SwitchX and Spectrum are configured in different ways when we look on L2 ETS
    • The Spectrum also support to use L3 (DSCP) Note: Microsoft configuration guide use RoCE Layer 2 (L2) for S2D that use SET (NDKm2)

    ******************************************************************************
    Mellanox Show Commands
    ******************************************************************************

    S2D# show dcb priority-flow-control detail
    S2D# show dcb priority-flow-control
    S2D# show dcb priority-flow-control interface ethernet 1/1
    S2D# show dcb ets
    S2D# show dcb ets interface ethernet 1/1


    References

    Microsoft:
    Windows Server 2016 and 2019 RDMA Deployment Guide
    Validate DCB

    Mellanox L2 PCP TC:
    How to Install Windows Server 2016 with RoCEv2 and Switch Embedded Teaming over HA Mellanox Network Solution
    Understanding QoS Classification (Trust) on Spectrum Switche
    Understanding Traffic Class (TC) Scheduling on Mellanox Spectrum Switches (WRR,SP)
    Understanding RoCEv2 Congestion Management
    DCBX Versions and Support on Mellanox  Ethernet Switches
    HowTo Configure PFC on ConnectX-4

    Mellanox L3 - DSCP:
    Recommended Network Configuration Examples for RoCE Deployment
    Lossless RoCE Configuration for MLNX-OS Switches in DSCP-Based QoS Mode
    Lossless RoCE Configuration for MLNX-OS Switches in DSCP-Based QoS Mode (advanced mode)
    Lossless RoCE Configuration for WinOF2 driver in DSCP-based QoS mode
    HowTo Configure Mellanox Spectrum Switch for Lossless RoCE
    HowTo Configure ECN on Mellanox Ethernet Switches (Spectrum)

    Terminology:
    pNIC Physical NIC, the physical hardware that exchanges packets with the TOR
    vNIC    Host vNIC – Virtual NIC from vSwitch exposed in the host partition
    tNIC Host tNIC - Team Interface NIC from LBFO Team 
    vmNIC Virtual Machine NIC – Virtual NIC from vSwitch exposed in a guest partition
    vSwitch Hyper-V virtual switch
    SET Switch Embedded Teaming, Hyper-V virtual switch supported in Windows Server 2016 and 2019

    ToR Top of Rack switch
    RDMA Remote Direct Memory Access
    RoCE    RDMA over Converged Ethernet
    RoCEv2 2nd generation RoCE using UDP/IP for routability (a.k.a. Routable RoCE)

    DCB     Data Center Bridging
    LLDP    Link Layer Data Protocol
    DCBx    Data Center Bridging Capability Exchange protocol (DCBX) is an extension of LLDP.
    PFC     Priority Flow Control
    ETS     Enhanced Transmission Service
    TC      Traffic Class

    ECN     Explicit Congestion Notification
    RED     Random Early Detection
    CNP     Congestion Notification Packet. CNP control frames (congestion ACK)
    SP      Strict Priority
    WRR     Weighted Round Robin


    ******************************************************************************
    Mellanox SX1012 (SwitchX)
    ******************************************************************************
    S2D# show dcb priority-flow-control detail

    PFC enabled
    Priority Enabled List    :3
    Priority Disabled List   :0 1 2 4 5 6 7

    PFC Port Eth1/x            Information
    -----------------------------------------------
    PFC Port Mode       :On
    PFC Oper State      :On
    No Remote Entry is Present
    -----------------------------------------------

    ******************************************************************************
    S2D# show dcb priority-flow-control

    PFC enabled
    Priority Enabled List    :3
    Priority Disabled List   :0 1 2 4 5 6 7

    TC     Lossless
    ---    ----------
    0           N
    1           Y
    2           Y
    3           N

    Interface      PFC admin        PFC oper
    ------------   --------------   -------------
    Eth1/1           On               Enabled
    Eth1/2           On               Enabled
    Eth1/3           On               Enabled
    Eth1/4           On               Enabled
    Eth1/5           On               Enabled
    Eth1/6           On               Enabled
    Eth1/7           On               Enabled
    Eth1/8           On               Enabled
    Eth1/9           On               Enabled
    Eth1/10          On               Enabled
    Eth1/11          On               Enabled
    Eth1/12          On               Enabled


    ******************************************************************************
    S2D# show dcb priority-flow-control interface ethernet 1/1

    PFC enabled
    Priority Enabled List    :3
    Priority Disabled List   :0 1 2 4 5 6 7

    TC     Lossless
    ---    ----------
    0           N
    1           Y
    2           Y
    3           N

    Interface      PFC admin        PFC oper
    ------------   --------------   -------------
    Eth1/1           On               Enabled

    ******************************************************************************
    S2D# show dcb ets
    (Note: Default value before change)

    ETS enabled

    TC        Bandwidth
    --------------------------
    0         25%
    1         25%
    2         25%
    3         25%

    Number of Traffic Class: 4

    ******************************************************************************
    S2D# show dcb ets interface ethernet 1/1
    (Note: Default value before change)

    ETS Port Mode             :AUTO MODE
    ETS Oper State            :INIT STATE
    ETS State Machine Type    :Assymetric
    -----------------------------------------------
    ETS Local Port Info
    -----------------------------------------------
    TC bandwidth table
    -----------------------------------------------
    TC        Bandwidth        RecomBandwidth
    -----------------------------------------------
    0         25%              25%
    1         25%              25%
    2         25%              25%
    3         25%              25%

    priority assignment table
    --------------------------------------
    Priority     TC
    --------------------------------------
    0            0
    1            0
    2            1
    3            1
    4            2
    5            2
    6            3
    7            3

    Number of Traffic Class: 4

    Willing Status:  Disable
    -----------------------------------------------
    ETS Admin Port Info
    -----------------------------------------------
    TC        Bandwidth        RecomBandwidth
    -----------------------------------------------
    0         25%              25%
    1         25%              25%
    2         25%              25%
    3         25%              25%
    -----------------------------------------------
    ETS Remote Port Info
    -----------------------------------------------
    No Remote Entry is Present
    -----------------------------------------------

    ******************************************************************************
    Configuration of SX1012 (SwitchX)
    ******************************************************************************

    Mapping of priority to traffic classes (TC)
    Priority 0 and 1 mapped to TC 0
    Priority 2 and 3 mapped to TC 1
    Priority 4 and 5 mapped to TC 2
    Priority 6 and 7 mapped to TC 3

    TC 0 and TC 3 are lossy TCs.
    TC 1 and TC 2 are lossless TCs.

    For the PFC enabled priorities we need to use the lossless TCs

    So for the Microsoft Example use case we change the default to:
    Priority 0 Default traffic 39%
    Priority 3 SMB traffic 60%
    Priority 5 Cluster traffic 1%

    S2D (config)# dcb ets tc bandwidth 39 60 1 0

    ******************************************************************************

    S2D# show dcb ets interface ethernet 1/1
    (Note: Value after change)

    S2D# sh dcb ets interface ethernet 1/1

    ETS Port Mode             :AUTO MODE
    ETS Oper State            :INIT STATE
    ETS State Machine Type    :Assymetric
    -----------------------------------------------
    ETS Local Port Info
    -----------------------------------------------
    TC bandwidth table
    -----------------------------------------------
    TC        Bandwidth        RecomBandwidth
    -----------------------------------------------
    0         39%              39%
    1         60%              60%
    2          1%               1%
    3          0%               0%

    priority assignment table
    --------------------------------------
    Priority     TC
    --------------------------------------
    0            0
    1            0
    2            1
    3            1
    4            2
    5            2
    6            3
    7            3

    Number of Traffic Class: 4

    Willing Status:  Disable
    -----------------------------------------------
    ETS Admin Port Info
    -----------------------------------------------
    TC        Bandwidth        RecomBandwidth
    -----------------------------------------------
    0         39%              39%
    1         60%              60%
    2          1%               1%
    3          0%               0%
    -----------------------------------------------
    ETS Remote Port Info
    -----------------------------------------------
    No Remote Entry is Present
    -----------------------------------------------

    ******************************************************************************
    Mellanox SN-2100 Spectrum
    ******************************************************************************

    S2D# show dcb ets interface ethernet 1/1
    (Note: Default value before change)

    Eth1/1
     Interface Bandwidth Shape [Mbps]: N/A
     Multicast unaware mapping : disabled

     ETS per TC :
     TC Scheduling Mode Weight Weight (%)
     -- --------------- ------ ----------
     0  WRR             12     12
     1  WRR             13     13
     2  WRR             12     12
     3  WRR             13     13
     4  WRR             12     12
     5  WRR             13     13
     6  WRR             12     12
     7  WRR             13     13

     Bandwidth Shape per TC:
     TC Bandwidth Shape [Mbps]
     -- ----------------------
     0  N/A
     1  N/A
     2  N/A
     3  N/A
     4  N/A
     5  N/A
     6  N/A
     7  N/A

     Bandwidth Guarantee per TC:
     TC Bandwidth Guaranteed [Mbps]
     -- ---------------------------
     0  0
     1  0
     2  0
     3  0
     4  0
     5  0
     6  0
     7  0

     Switch Priority to TC mapping:
     Switch Priority TC
     --------------- --
     0               0
     1               1
     2               2
     3               3
     4               4
     5               5
     6               6
     7               7

    ******************************************************************************







    ******************************************************************************

    Mellanox Switch configuration example (L2):
    ******************************************************************************

    Mellanox SN2100 (Spectrum)

    S2D> enable
    S2D# configure terminal 
    S2D (Config)# interface ethernet 1/1-1/12 traffic-class 3 dcb ets Strict
    S2D (Config)# interface ethernet 1/1-1/12 traffic-class 7 dcb ets Strict

    S2D (Config)# interface ethernet 1/1-1/x flowcontrol send off force
    S2D (Config)# interface ethernet 1/1-1/x flowcontrol receive off force

    S2D (Config)# dcb priority-flow-control enable force
    S2D (Config)# dcb priority-flow-control priority 3 enable
    (Priority 3 is used for Storage (SMB) traffic)
    S2D (Config)# interface ethernet 1/1-1/x dcb priority-flow-control mode on force
    S2D (Config)# interface ethernet 1/x switchport mode hybrid
    (Need to be repeated for each port from 1 to 12/16)
    S2D# interface ethernet 1/x switchport hybrid allowed-vlan all
    Need to be repeated for each port from 1 to 12/16. Only allow the needed vlans, the "allowed-vlan all" is not recommended, this is from a lab/test system).
    S2D (Config)# exit
    S2D# write memory

    ******************************************************************************
    S2D# show dcb ets interface ethernet 1/1
    Eth1/1
     Interface Bandwidth Shape [Mbps]: N/A
     Multicast unaware mapping : disabled

     ETS per TC :
     TC Scheduling Mode Weight Weight (%)
     -- --------------- ------ ----------
     0  WRR             12     12
     1  WRR             13     13
     2  WRR             12     12
     3  Strict          0      0
     4  WRR             12     12
     5  WRR             13     13
     6  WRR             12     12
     7  Strict          0      0

     Bandwidth Shape per TC:
     TC Bandwidth Shape [Mbps]
     -- ----------------------
     0  N/A
     1  N/A
     2  N/A
     3  N/A
     4  N/A
     5  N/A
     6  N/A
     7  N/A

     Bandwidth Guarantee per TC:
     TC Bandwidth Guaranteed [Mbps]
     -- ---------------------------
     0  0
     1  0
     2  0
     3  0
     4  0
     5  0
     6  0
     7  0

     Switch Priority to TC mapping:
     Switch Priority TC
     --------------- --
     0               0
     1               1
     2               2
     3               3
     4               4
     5               5
     6               6
     7               7



    ******************************************************************************
    Mellanox Switch configuration example with DSCP:

    Draft - Test only last update 2019.01.19
    Note:
    #1: The example use both L2/L3. SwitchX use L2 and Spectrum use both L2/L3
    #2: Microsoft configuration guide use RoCE Layer 2 (L2) for S2D.
    #3: DSCP is supported by Windows Server OS. However to use it for S2D is not recommended right now by Microsoft and for test you need to work close with the Hardware vendors. 

    Lossless RoCE Configuration for MLNX-OS Switches in DSCP-Based QoS Mode
    Lossless RoCE Configuration for MLNX-OS Switches in DSCP-Based QoS Mode (Old version)
    ******************************************************************************

    S2D (Config)# interface ethernet 1/1-1/x qos trust ?
           port       based on port default settings
           L2         based on PCP, DEI fields
           L3         based on EXP, DSCP fields
           both       based on PCP, DEI and EXP, DSCP fields

    S2D (Config)# interface ethernet 1/1-1/x qos trust both
    (Note: In my lab I used both SwitchX and Spectrum. The Spectrum Switch need to accepts PCP and DSCP in my example)

    ******************************************************************************

    S2D (config) # interface ethernet 1/1-1/x traffic-class 3 congestion-control ecn minimum-absolute 150 maximum-absolute 1500
    S2D (config) # traffic pool roce type lossless
    S2D (config) # traffic pool roce memory percent 50.00
    S2D (config) # traffic pool roce map switch-priority 3
    S2D (config) # interface ethernet 1/1-1/x traffic-class 3 dcb ets strict
    S2D (config) # interface ethernet 1/1-1/x traffic-class 7 dcb ets strict
    S2D (config) # interface ethernet 1/1-1/x qos trust both

    ******************************************************************************

    S2D (Config)# interface ethernet 1/1-1/x traffic-class 0 bind switch-priority 0
    S2D (Config)# interface ethernet 1/1-1/x traffic-class 3 bind switch-priority 3
    S2D (Config)# interface ethernet 1/1-1/x traffic-class 7 bind switch-priority 7

    ******************************************************************************

    S2D (Config)# interface ethernet 1/1-1/x flowcontrol send off force
    S2D (Config)# interface ethernet 1/1-1/x flowcontrol receive off force

    S2D (Config)# dcb priority-flow-control enable force
    S2D (Config)# dcb priority-flow-control priority 3 enable
    (Priority 3 is used for Storage (SMB) traffic)
    S2D (Config)# interface ethernet 1/1-1/x dcb priority-flow-control mode on force
    S2D (Config)# interface ethernet 1/x switchport mode hybrid
    (Need to be repeated for each port from 1 to x)
    S2D (Config)# interface ethernet 1/x switchport hybrid allowed-vlan all
    Need to be repeated for each port from 1 to x. Only allow the needed vlans, the "allowed-vlan all" is not recommended, this is from a lab/test system).

    ******************************************************************************
    Host pNIC configuration

    Driver version 2.10 or newer







    S2D PS> Mlx5Cmd.exe -QosConfig -SetupRoceQosConfig -Name NIC3 -Configure 2





    Default DSCP to switch-priority mapping:
    0-7 → 0
    8-15 → 1
    16-23 → 2
    24-31 → 3
    32-39 → 4
    40-47 → 5
    48-55 → 6
    56-63 → 7


    ******************************************************************************