Skip to content

Home

Jump to bottom
Basil Salman edited this page Oct 12, 2020 · 35 revisions

Table of Contents

About VirtHCK

VirtHCK is a set of shell scripts for running Microsoft HCK (Hardware Certification Kit) tests for a variety of Windows drivers, in a VM (virtual machines) environment. It creates an isolated virtual network environment for HCK Controller and Clients' virtual machines, running on top of QEMU-KVM and allows to run various HCK tests in an automated manner.

Moreover, the preparation of the new VMs for HCK functionality, and the installation of the HCK Controller and Clients themselves, can be automated using the auxiliary scripts found under guest_tools/KitAutosetup (along with the corresponding documentation).

Overall, such a setup in a virtualized environment allows enormous flexibility regarding the properties of the machines, the operating systems installed on them, and the ease of switching between them - far above and beyond a physical setup. Additionally, fast recovery from OS corruption is possible (just delete the current disk image and use one from a backup!) and the results are expected to be very consistent - no physical hardware that may malfunction is involved (except, possibly the tested device itself, if it is indeed physical).

VirtHCK Initial Setup

  1. Clone VirtHCK and enter its directory:
    git clone https://github.com/daynix/VirtHCK.git
    cd VirtHCK
  2. Create a QEMU VM image for the HCK Controller:
    qemu-img create -f qcow2 images/HCK_Studio_<OS NAME>.qcow2 500G
  3. Create two QEMU VM images for the HCK Clients:
    qemu-img create -f qcow2 images/HCK_CL1_<OS NAME>.qcow2 500G
    qemu-img create -f qcow2 images/HCK_CL2_<OS NAME>.qcow2 500G
  4. Update the STUDIO_IMAGE, CLIENT1_IMAGE, and CLIENT2_IMAGE variables in hck_setup.cfg.
  5. Update the STUDIO_EXTRA, CLIENT1_EXTRA, and CLIENT2_EXTRA variables with the paths to the Windows installation DVD ISO (can be the same path for all).
    • Do not forget to comment these variables after the installation is done!
  6. Turn on the installation options in hck_setup.cfg:
    • Set UNSAFE_CACHE to on for faster VM installation.
    • Set CLIENT_WORLD_ACCESS to on to enable the activation of Windows on the client machines.
  7. Modify additional parameters in hck_setup.cfg, as needed. Refer to their description below.
    • Pay attention that the value of WORLD_BR_NAME corresponds to the bridge name on your machine.
    • Also, if several VirtHCK setups are to be used, make sure that the UNIQUE_ID is truly unique.
  8. Run the VirtHCK Setup by: sudo ./hck.sh.
  9. Connect to the machines' display using VNC or Spice (the correct port will be shown in the summary on screen), and install Windows, as usual.
  10. Prepare the machines for HCK Controller/Studio/Clients installation, and install them.
  11. Turn the installation options (UNSAFE_CACHE, CLIENT_WORLD_ACCESS) to off. This is important, as otherwise the images can get corrupted and the external network traffic can get contaminated!
  12. For additional information on HCK, please refer to Microsoft's documentation.

Directory Tree

  • VirtHCK/ - Directory with all the scripts.
  • VirtHCK/guest_tools/ - Directory with auxiliary scripts, needed for automation and configuration tasks, but not for running VirtHCK.
  • images/ - Directory where the VMs' images should be located.
  • SMB_SHARE/ - Directory that will be used as a Samba share, to transfer files between the Controller, Clients, and the host.

Scripts' Description

hck.sh

  • Usage:
    hck.sh [ st | c1 | c2 ] [ c1 | c2] [ c2 ]
This is the main script for VirtHCK environment start up. It starts the Controller, and the two Client VMs. When started, the current VirtHCK environment configuration will be shown in a summary on screen.

This script can be executed with parameters, that specify which VMs to run:

  • st - Start the HCK Controller VM.
  • c1 - Start the HCK Client 1 VM.
  • c2 - Start the HCK Client 2 VM.
By default, the virtual machines will auto-respawn - they will turn on immediately after being shut down. This is necessary for some of the HCK tests. When you wish to actually shut the machines down (like when you want to terminate the setup) press enter in the terminal window where hck.sh was run, and this will keep the machines off when they are turned off.

hck_setup.cfg

This is the VirtHCK Setup Configuration File, required by scripts for environment configuration and VM startup parameters. It should be configured accordingly to required HCK setup.

Configurable variables in hck_setup.cfg
  1. UNIQUE_ID=[1-80] - Unique ID of the current VirtHCK setup, used for the creation of unique values for: virtual NIC MAC addresses, virtual machine UUIDs, etc. It is very important to set this variable to a unique number for each running VirtHCK setup, in order to avoid system collisions.
  2. QEMU_BIN - QEMU executable path.
  3. TEST_DEV_TYPE, TEST_DEV_NAME, TEST_DEV_EXTRA - Define the VM hardware for testing. Several preconfigured device types are available and commented out. The preconfigured device names are the most frequently used devices for their type. The third variable enables to add extra device parameters, if needed.
  4. IS_PHYSICAL=[false|true] - Sets the test device type to physical - for testing physical devices. Currently supports network devices only.
    In case it is set to true, the following options take effect:
    1. ASSIGNMENT=[vfio-pci|pci-assign] - Sets the device assignment type.
      • pci-assign is the old approach to device assignment in Qemu, and it generally requires no configuration on the host.
      • vfio-pci is the newer approach, however it generally requires some configuration on the host. You can read more about that topic here.
    2. CLIENT1_HOST_ADDRESS, CLIENT2_HOST_ADDRESS - The physical PCI address of the device on the host for clients 1 & 2. Obtained from the lspci output.
  5. MACHINE_TYPE - For example: pc, q35, or their variants. The full list of machine types that are available in Qemu can be obtained by running:
    <The current Qemu binary> -machine help.
  6. BOOT_ORDER - Sets boot order preferences. For more info on Qemu's <-boot> parameter, you can checkout the Qemu documentation here.
  7. VIDEO_TYPE=[VNC|SPICE] - Sets RDC protocol for VirtHCK VMs.
  8. STUDIO_IMAGE, CLIENT1_IMAGE, CLIENT2_IMAGE - Virtual Machines' image names.
  9. CLIENT_CPUS - Amount of vCPU cores for each client VM. Some HCK tests require at least 4 cores.
  10. CLIENT_MEMORY - Sets client's VM memory size. At least 4G is required for some HCK tests, but other tests may fail if the value is over 3G.
  11. WORLD_NET_DEVICE=[e1000|rtl8139|virtio-net-pci] - Set the type of NIC for external Internet connection.
  12. CTRL_NET_DEVICE=[e1000|rtl8139|virtio-net-pci] - Set the type of dedicated NIC for internal network (Controller-Client) interconnection.
  13. CLIENT1_USB_DEV, CLIENT2_USB_DEV - Attaches an image to the client's machine as a usb device. The image can be in any supported Qemu format.
  14. STUDIO_EXTRA, CLIENT1_EXTRA, CLIENT2_EXTRA - Variables used for extra VM settings, like CD-ROM attachment, alternate BIOS image path, etc.
  15. SHARE_ON_HOST - A directory on the host machine, that will be available as a Samba share on the HCK Controller and can be made available on the HCK Clients as well, by placing a file named USE_SHARE in it. The default location is: VirtHCK/SMB_SHARE. Setting SHARE_ON_HOST to false will disable the SMB share on both the Controller and the Clients.
    • Before testing a network device, please make sure that the SMB share is disabled on the Clients: if a file named USE_SHARE was present in this directory, delete it, and shut down the Clients (do not reboot - actually shut them down). When the clients will boot again (by auto spawning or manually) the share will not be available on them (but will be on the Controller).
  16. vIOMMU - enable vIOMMU device.
    • vIOMMU is supported for Virtio devices with Q35 machines only.

run_hck_studio.sh

This script is executed from hck.sh, and starts up the Controller (Studio) VM, which is configured according to the settings from hck_setup.cfg.

run_hck_client.sh

This script is executed from hck.sh, and starts up Client VMs, which are configured according to the settings from hck_setup.cfg.

hck_ctrl_bridge_ifup.sh

This script is executed by QEMU, it creates a virtual network bridge on the host, for the Controller-Client interconnection.

hck_test_bridge_ifup.sh

This script is executed by QEMU, it creates a virtual network bridge on the host, for the NIC under test.

hck_world_bridge_ifup.sh

This script is executed by QEMU, it creates a virtual network bridge on the host, for the VMs to connect to the Internet.

guest_tools/run_hck_studio.bat

This script should be executed on the HCK Controller, to start the HCK Studio. It disconnects the Controller from the Internet before the HCK Studio is run, and connects it again after the Studio closed. Update the adapter name parameter accordingly to the interface name of the outbound NIC on the Controller.

guest_tools/enable_world_network.bat

This script could be executed on the Controller in order to enable Internet connection during the HCK Studio run. Update the adapter name parameter accordingly to the interface name of the outbound NIC on the Controller.

guest_tools/install_cert.bat

  • Usage:
    install_cert.bat TestCert.cer
This script helps to install a test certificate on a Client VM. It should be run with Administrator privileges. It requires a certificate file, and the certmgr.exe console application from WinDDK/WDK in script's directory. certmgr.exe is usually located in:
  • C:\Program Files (x86)\Windows Kits\〈ver. number〉\bin\x86 - for a 32-bit OS
  • C:\Program Files (x86)\Windows Kits\〈ver. number〉\bin\x64 - for a 64-bit OS

Checklist for a New Controller VM

  • This step can be automated using the scripts available in guest_tools/KitAutosetup - see documentation there. Manual instructions are provided below:
  1. Install a VM with Windows 2012 OS (with min. 2 GB of RAM).
  2. Set a static IP address for the control adapter - 192.168.100.1.
  3. Disable the Windows Firewall.
  4. Make all the network connections work/private.
  5. Disable Windows Update.
  6. Disable the screen saver, auto-locking, and auto-sleep.
  7. Rename the control network adapter to Control.
  8. Rename the computer to HCK-STUDIO.
  9. Activate OS.
  10. Install the HCK Controller according to Microsoft's recommendations.
  11. Download the HCK Filters from here, extract the UpdateFilters.sql file from the downloaded .cab archive, put it under the C:\Program Files (x86)\Windows Kits\〈ver. number〉\Hardware Certification Kit\Controller directory (〈ver. number〉 = HCK version) and run the updatefilters.exe file there.
  12. Map the Samba share (\\192.168.101.1\qemu) as a network drive X:.
  13. Shut down the machine (verify it didn't respawn) and then compress the image and put it to the archive directory.

Checklist for a New Client VM

  • This step can be automated using the scripts available in guest_tools/KitAutosetup - see documentation there. Manual instructions are provided below:
  1. Install a pair of VMs with the target OS (the VMs should be installed separately, not copied). Min. amount of RAM per VM: 4 GB. Note: for some tests max. 3 GB is needed.
  2. Activate the computers. (External network access of Clients is required here and ONLY here. It is enabled automatically only when the images are being installed).
  3. Set static IP addresses for the control adapters (Client 1: 192.168.100.2, Client 2: 192.168.100.3).
  4. Rename control network adapter to MessageDevice (this specific name is important for LAN device tests!).
  5. Disable the Windows Firewall.
  6. Make all the network connections work/private.
  7. Disable Windows Update.
  8. Disable the screen saver, auto-locking, and auto-sleep.
  9. Enable automatic reboot and kernel memory dump on bugcheck.
  10. Rename the computers (Client 1: CL1-〈OS〉, Client 2: CL2-〈OS〉).
  11. Install the HCK Client from \\HCK-STUDIO\HCKInstall\Client\setup.exe (the Controller VM needs to be on at this stage).
  12. Optional: Install the test signing certificates (64-bit OSes) by the script guest_tools/install_cert.bat.
  13. Install the drivers for the device being tested.
  14. Run at least one test and verify it passes (Important: at this stage the HCK finalizes the configuration - it is necessary).
  15. Map the Samba share (\\192.168.101.1\qemu) as a network drive X:.
  16. Create a device manager icon on the desktop.
  17. Shut down the machines (verify they didn't respawn), compress their images, and put them to the archive directory.

VirtHCK Automation and Remote Testing

It is possible to automate the tests and to run them remotely, or from the host machine. For more information, please read the documentation of RemoteHCK.

Test Requirements and Recommendations

ivshmem

Run ivshmem device only on QEMU version 2.12.0 or newer, on earlier versions a bug in ivshmem caused QEMU to crash

To run ivshmem tests, you need to install ivshmem-server on the host side:

on fedora or other RedHat distros run "sudo dnf install ivshmem-tools"

on other distros compile QEMU, on build directory, executable name ivshmem-server will be created, on the file hck_setup.cfg update IVSHMEM_SERVER_BIN value to ivshmem-server new build path.

File System Testing Prerequisites

In storage devices tests requires to create additional partitions: File System Testing Prerequisites. You can create these partitions on the fly with StorageAutoPartition script or provide a pre made image and configure it on hck_setup.cfg on FILESYSTEM_TESTS_IMAGE

Storage Tests

  • The HCK Client images should be at least 30 GB in size.
  • The HCK "Crashdump Support Test" requires Client VM Internet connection in order to load the debug symbols from Microsoft's Symbol Server. It can be enabled manually, with the CLIENT_WORLD_ACCESS parameter in hck_setup.cfg. Do not forget to turn it off when done!
  • If "Disk Stress" and/or "Disk Verification" tests fail with performance counters warnings on the Client, update the registry performance counters' values: run the following in the command prompt, as the Administrator:
    lodctr /r
    lodctr /q
  • If the "DP WLK - Hot-Add - Device Test" fails with "Interface GUID_DEVINTERFACE_BUSENUM_TOASTER is not registered" error, see this link for the resolution.

Troubleshooting

  • If the Controller machine freezes when starting the HCK Studio using the RunStudio.bat script (which is the only recommended way to run it) try using Qemu v2.6.0 or later.
  • The test logs can be found on the Client machines, in C:\WLK\JobsWorkingDir\WLKSvc.txt.
  • In the case that a test fails with the message: "Waiting for computer to be assigned", make sure that the client is running with at least 4G RAM, and 4 cores. Also check the logs on the clients, for other possible causes.
  • If Windows crashes during tests that involve hibernation (S4), try to install MS updates KB2823506 and KB2822241. Or, run these tests with 3G or RAM.
  • In a case that a networking device test fails with the message:
    WDTF_SIMPLE_IO : NetworkSimpleIO.Network.1 SetTarget() ERROR : NetworkPlugin: SetTarget() - The network interface does not have an IPv6 address
    Follow these steps:
    1. Set an IPv6 address to the NIC under test on the Client machine (a random address, e.g.: fd00::0:0:0:0:0:2).
    2. Set the test parameter WDTFREMOTESYSTEM (you'll be prompted for it when scheduling the tests that require it) to the Link-local IPv6 Address of the support client (you can see this address on the support client's Networking Interfaces panel, under Status → Details of the support device NIC). Alternatively, set the default gateway of the device under test to that address.
  • In a case of the following error:
    ... had too many consecutive timeouts while pinging the gateway address or the remote system IP address ...
    Just the second step of the above will suffice to solve the problem.
  • In case that the test "Multiple processor group device" is stuck for a long period of time, check the target Client. In case that the Client's MessageDevice NIC has no network connection, or is connecting and disconnecting repeatedly, then run the following commands, as the Administrator, and reboot afterwards:
    bcdedit.exe /set groupaware off
    bcdedit.exe /deletevalue groupsize
    After that, this test should pass after applying the latest HCK filters.
  • In case no device shows up on 'device manager' in HCK kit studio, uninstall both HCK clients (Windows Hardware Certification Kit Client) and reinstall from \\HCK-STUDIO\HCKInstall\Client\setup.exe
  • If you encounter the following error it is normal (netfilter is already disabled by default):
Disabling bridge-netfilter...
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-arptables: No such file or directory
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory
Clone this wiki locally