From: Jouni Malinen <j@w1.fi>
Date: Sun, 18 Jan 2015 16:06:47 +0000 (+0200)
Subject: tests: Add step-by-step guide for setting up test framework
X-Git-Tag: hostap_2_4~368
X-Git-Url: http://git.ipfire.org/?a=commitdiff_plain;h=9eadcfbf06efdb72fc8bf9bf38338378112ee02e;p=thirdparty%2Fhostap.git

tests: Add step-by-step guide for setting up test framework

This set of notes provides information on how virtual guess OS can be
used to run the mac80211_hwsim test cases under any host OS. The
specific example here uses Ubuntu 14.04.1 server as the starting point
and lists the additional packages that need to be installed and commands
that can be used to fetch and build the test programs.

Signed-off-by: Jouni Malinen <j@w1.fi>
---

diff --git a/tests/hwsim/README b/tests/hwsim/README
index 8e8267050..e68b0dcc3 100644
--- a/tests/hwsim/README
+++ b/tests/hwsim/README
@@ -65,6 +65,9 @@ Debian/Ubuntu packages are likely to be needed:
 - libsqlite3-dev
 - libpcap-dev
 
+example-setup.txt provides more complete step-by-step example on how a
+test setup can be built.
+
 
 wpaspy
 ------
diff --git a/tests/hwsim/example-setup.txt b/tests/hwsim/example-setup.txt
new file mode 100644
index 000000000..ee55a4258
--- /dev/null
+++ b/tests/hwsim/example-setup.txt
@@ -0,0 +1,177 @@
+Step-by-step guide for setting up hostapd/wpa_supplicant test framework
+-----------------------------------------------------------------------
+
+This document can be used as a quick guide for getting started with
+hostapd/wpa_supplicant test framework with mac80211_hwsim. While the
+example here uses Ubuntu 14.04.1 server to have a list of exact steps,
+there are no requirements for using that specific distribution in the
+testing setup.
+
+The steps here describe how to run a full Linux installation in a
+virtual machine with any host system (e.g., Linux, Windows, or OS X as
+the host and using kvm, VirtualBox, etc. for running the virtual guest
+system). For more advanced (and significantly faster and with more
+testing coverage) configuration on a Linux host system, parallel virtual
+machines can be used as an alternative setup. See tests/hwsim/vm/README
+for more details on that.
+
+
+Install Ubuntu Server 14.04.1 in the virtual machine
+
+- download installation image, e.g.,
+  http://releases.ubuntu.com/14.04.1/ubuntu-14.04.1-server-amd64.iso
+- use virtualization software specific steps to create a new VM and
+  install the the guest system with default settings (i.e., no need to
+  select any extra packages during initial installation)
+- if the host system has multiple CPU cores, it is likely a good idea to
+  enabled at least two CPUs in the guest; 1024 MB of RAM should be enough
+  for testing purposes
+- 8 GB of virtual hard driver should be fine for this purpose
+- boot to the installed operating system
+
+
+Install the prerequisite packages that may not have been installed by default
+
+sudo apt-get install build-essential git libpcap-dev libsqlite3-dev binutils-dev libnl-3-dev libnl-genl-3-dev libnl-route-3-dev libssl-dev libiberty-dev libdbus-1-dev iw bridge-utils python-pyrad
+
+
+Install a recent kernel wireless components (mac80211_hwsim, mac80211,
+cfg80211)
+
+For this step, the kernel version may be updated, but the simpler option
+is to install the latest version of Backports package. For example:
+
+wget http://www.kernel.org/pub/linux/kernel/projects/backports/stable/v3.19-rc1/backports-3.19-rc1-1.tar.xz
+tar xJf backports-3.19-rc1-1.tar.xz
+cd backports-3.19-rc1-1
+
+cat > defconfigs/mac80211_hwsim <<EOF
+CPTCFG_CFG80211=m
+CPTCFG_CFG80211_WEXT=y
+CPTCFG_MAC80211=m
+CPTCFG_MAC80211_LEDS=y
+CPTCFG_MAC80211_MESH=y
+CPTCFG_WLAN=y
+CPTCFG_MAC80211_HWSIM=m
+EOF
+
+make defconfig-mac80211_hwsim
+make
+sudo make install
+cd ..
+
+
+Update iw based on custom iw.git build
+
+Couple of the test cases expect iw to have support for requesting
+cfg80211 scan results to be flushed. That functionality is not included
+in the version that Ubuntu 14.04.1 includes (iw v3.4). Following steps
+can be used to replace that version with a custom build. This is
+optional, i.e., most test cases will work with the old iw version, but
+some test cases are skipped and some are more likely to fail if iw does
+not get updated.
+
+wget https://www.kernel.org/pub/software/network/iw/iw-3.17.tar.gz
+tar xf iw-3.17.tar.gz
+cd iw-3.17
+make
+sudo mv /sbin/iw{,-distro}
+sudo cp iw /sbin/iw
+cd ..
+
+
+Update wireless-regdb
+
+Number of VHT and DFS test cases are skipped if the old wireless-regdb
+version from Ubuntu 14.04 (2013.02.13) is used. Following steps can
+optionally be used to update wireless-regdb to a newer snapshot to
+enable additional test cases:
+
+wget http://kernel.org/pub/software/network/wireless-regdb/wireless-regdb-2014.10.07.tar.xz
+tar xJf wireless-regdb-2014.10.07.tar.xz
+sudo mv /lib/crda/regulatory.bin{,-distro}
+sudo cp wireless-regdb-2014.10.07/regulatory.bin /lib/crda/regulatory.bin
+
+# following command can be used to verify that the new version is trusted
+regdbdump /lib/crda/regulatory.bin
+
+
+Download a snapshot of the hostap.git repository and build the programs
+
+git clone git://w1.fi/hostap.git
+cd hostap/tests/hwsim
+./build.sh
+
+
+Setup is now ready for testing. You can run a quick test to confirm that
+things work as expected:
+
+# load mac80211_hwsim and start test software
+./start.sh
+
+# run a single test case ap_open
+sudo ./run-tests.py ap_open
+
+This should print out following style results:
+
+DEV: wlan0: 02:00:00:00:00:00
+DEV: wlan1: 02:00:00:00:01:00
+DEV: wlan2: 02:00:00:00:02:00
+APDEV: wlan3
+APDEV: wlan4
+START ap_open 1/1
+Test: AP with open mode (no security) configuration
+Starting AP wlan3
+Connect STA wlan0 to AP
+PASS ap_open 0.175895 2015-01-17 20:12:07.486006
+passed all 1 test case(s)
+
+(If that "PASS ap_open" line does not show up, something unexpected has
+happened and the setup is not in working condition.)
+
+# to stop test software and unload mac80211_hwsim
+./stop.sh
+
+
+To run all available test cases (about thousand or so), you can run following:
+
+./run-all.sh
+
+This will take about half an hour to hour to run (if that sounds long, see
+vm/README for information on how parallel VMs can be used to speed this
+up; e.g., a 4-core i7-4770K can run these in under 10 minutes with 7
+parallel VMs).
+
+The results may look something like this:
+
+START grpform_goneg_fail_with_group_iface 1/981
+PASS grpform_goneg_fail_with_group_iface 0.371424 2015-01-17 22:17:16.659803
+START grpform2 2/981
+PASS grpform2 1.476142 2015-01-17 22:17:18.136539
+...
+START ext_password_psk_not_found 981/981
+PASS ext_password_psk_not_found 1.544709 2015-01-17 22:46:56.489764
+failed tests: wext_wpa2_psk wext_wep_open_auth wext_open wext_rfkill wext_scan_hidden wext_pmksa_cache wext_wep_shared_key_auth
+
+
+In this example, about 860 test cases passed and about 100 were skipped.
+
+Most of the skipped test cases are in following categories:
+- D-Bus (requires kvm-based test run, see vm/README)
+- VHT 80 and 160 MHz channels (requires wireless-regdb update)
+- DFS (requires wireless-regdb updates)
+
+The following test failed every time (i.e., other failed cases could be
+passed on second attempt):
+
+wext_pmf wext_wpa2_psk wext_wep_open_auth wext_open wext_rfkill wext_scan_hidden wext_pmksa_cache wext_wep_shared_key_auth
+
+WEXT failures are due to the specific cfg80211/mac80211 version from
+Backports not allowing WEXT support to be enabled. A newer build
+addresses that and these WEXT test cases pass, e.g., with this snapshot
+build:
+http://buildbot.w1.fi/backports-wireless-testing/backports-wireless-testing-2015-01-18-ba3f765.tar.bz2
+
+With that version, ibss_rsn is failing due to a known cfg80211
+regression in the specific snapshot build. All other test cases passed
+at least on retry or were skipped due to missing testing capability.