[oi-dev] Automated test suite for OpenIndiana

[Michal Nowak]

Hello,

last couple of months I have been working on automated test suite for 
OpenIndiana. You can find it's dashboard at

   https://openqa.oi.mnowak.cz

It makes use of openQA (http://open.qa/), an open source automated test 
tool that makes possible to test the whole installation process of 
operating system, OS configuration, regression and sanity testing of 
individual components. openQA is currently used by Fedora 
(https://openqa.fedoraproject.org/) and openSUSE 
(https://openqa.opensuse.org/) for package updates and snapshot release 
testing, respectively.

Test suite sources and guidance on how to setup your local openQA 
instance are on GitHub

   https://github.com/Mno-hime/os-autoinst-distri-openindiana/

Currently Live, Text, and Minimal variants of both ISO and USB images 
are tested as VM guests on KVM, VirtualBox, and Xen.

Some of the test scenarios implemented:

* General installation via text- as well as GUI-installer to disk, 
reboot to the installed system, OS setup, shutdown.
   - installation_mate at 64bit: https://openqa.oi.mnowak.cz/tests/1820 (on 
KVM host)
   - installation_textmode_raidz at 64bit: 
https://openqa.oi.mnowak.cz/tests/1841 (on KVM host)
   - installation_textmode_mbr at vbox64_ide_piix3: 
https://openqa.oi.mnowak.cz/tests/1805 (on VirtualBox host)
   - installation_mate at xen-hvm: https://openqa.oi.mnowak.cz/tests/1846 
(on Xen host; note: illumos guest Xen drivers are in bad shape atm)

* Upgrade scenarios for latest OpenIndiana snapshots to the latest 
incorporation, e.g. upgrade_textmode_20161030: 
https://openqa.oi.mnowak.cz/tests/1808

* illumos KVM virtualization test: https://openqa.oi.mnowak.cz/tests/1796

* Extended `pkg` console tests: https://openqa.oi.mnowak.cz/tests/1816

* Toolchain (pkg_src & oi-userland build) tests: 
https://openqa.oi.mnowak.cz/tests/1815

* Build of OpenIndiana snapshot images from latest published packages: 
https://openqa.oi.mnowak.cz/group_overview/2

* When things go wrong: installation_textmode at vbox64_sas_LSILogicSAS: 
https://openqa.oi.mnowak.cz/tests/1795

My test-host/worker - a 5 year old 4 core Ivy Bridge laptop with 12 GB 
of RAM, and SSD - is able to run 3 jobs in parallel reliably. Full build 
validation takes about 8 hours. Bare metal server is certainly desired.

I believe that this test suite could help us produce OpenIndiana hipster 
snapshots more validated, with higher level of confidence to particular 
updates, periodically run package's upstream testsuite in real-life 
environment as opposed to a bare-bone build environment, or release 
snapshots more often (if desired).

How to get involved: Deploy your local openQA instance in accordance to 
the documentation at http://open.qa on a Linux host (sic; I never 
deployed openQA on illumos as I don't have illumos capable hardware), 
follow OpenIndiana-specific guide at 
https://github.com/Mno-hime/os-autoinst-distri-openindiana/blob/master/README.md, 
clone particular job from openqa.oi.mnowak.cz, see how your changes 
affect the test run, and contribute the code back as a pull request.

Similar test suite can be implemented for other illumos distributions 
like OmniOS CE and SmartOS.

Feel free to contact me (mnowak_) on #illumos or #oi-dev.

Michal