[oi-dev] Automated test suite for OpenIndiana

[Michal Nowak]

We discussed the topics with Adam over IRC so just a short summary:

Ad (1): It should be doable, first I need to understand how Vagrant does 
stuff, will have a look at oi-packer.

Ad (2):

The dashboard, scheduler, and DB parts of openQA can run in a VM 
instance (and they do now), however worker (the part where the 
system-under-test aka VM actually runs) needs hypervizor to spawn VMs.

My laptop is the current KVM, VirtualBox, and Xen host. First two 
hypervizors can be switched on-the-fly, Xen environment is accessible 
only via reboot.

For jobs which require KVM the hypervizor host is openQA worker as well. 
For jobs which require VirtualBox or Xen hypervizor, the openQA worker 
and the hypervizor host may be different systems which can talk to one 
another via HTTP and SSH, but for HW reasons they are united in the 
current setup.

openQA has a good CLI which leverages REST-like API: e.g. 
https://openqa.oi.mnowak.cz/xyz.

Michal

On 11/25/2017 10:16 AM, Adam Števko wrote:
> Hello,
> 
> nice job!
> 
> Few questions:
> 
> 1) Would you be able to incorporate vagrant box testing? 2) We could
> try and get an instance for running this. Can this run inside the VM
> reliably? I see KVM/Xen/VirtualBox host. Can you confirm these are
> not your laptop, but dedicated hosts?
> 
> With this there is also a much more incentive to automate the
> creation of images as tests can run. Is there a good CLI client/API
> for managing openqa instance?
> 
> Cheers Adam
> 
>> On Nov 24, 2017, at 9:28 PM, Michal Nowak <mnowak at startmail.com>
>> wrote:
>> 
>> Hello,
>> 
>> last couple of months I have been working on automated test suite
>> for OpenIndiana. You can find it's dashboard at
>> 
>> https://openqa.oi.mnowak.cz
>> 
>> It makes use of openQA (http://open.qa/), an open source automated
>> test tool that makes possible to test the whole installation
>> process of operating system, OS configuration, regression and
>> sanity testing of individual components. openQA is currently used
>> by Fedora (https://openqa.fedoraproject.org/) and openSUSE
>> (https://openqa.opensuse.org/) for package updates and snapshot
>> release testing, respectively.
>> 
>> Test suite sources and guidance on how to setup your local openQA
>> instance are on GitHub
>> 
>> https://github.com/Mno-hime/os-autoinst-distri-openindiana/
>> 
>> Currently Live, Text, and Minimal variants of both ISO and USB
>> images are tested as VM guests on KVM, VirtualBox, and Xen.
>> 
>> Some of the test scenarios implemented:
>> 
>> * General installation via text- as well as GUI-installer to disk,
>> reboot to the installed system, OS setup, shutdown. -
>> installation_mate at 64bit: https://openqa.oi.mnowak.cz/tests/1820 (on
>> KVM host) - installation_textmode_raidz at 64bit:
>> https://openqa.oi.mnowak.cz/tests/1841 (on KVM host) -
>> installation_textmode_mbr at vbox64_ide_piix3:
>> https://openqa.oi.mnowak.cz/tests/1805 (on VirtualBox host) -
>> installation_mate at xen-hvm: https://openqa.oi.mnowak.cz/tests/1846
>> (on Xen host; note: illumos guest Xen drivers are in bad shape
>> atm)
>> 
>> * Upgrade scenarios for latest OpenIndiana snapshots to the latest
>> incorporation, e.g. upgrade_textmode_20161030:
>> https://openqa.oi.mnowak.cz/tests/1808
>> 
>> * illumos KVM virtualization test:
>> https://openqa.oi.mnowak.cz/tests/1796
>> 
>> * Extended `pkg` console tests:
>> https://openqa.oi.mnowak.cz/tests/1816
>> 
>> * Toolchain (pkg_src & oi-userland build) tests:
>> https://openqa.oi.mnowak.cz/tests/1815
>> 
>> * Build of OpenIndiana snapshot images from latest published
>> packages: https://openqa.oi.mnowak.cz/group_overview/2
>> 
>> * When things go wrong:
>> installation_textmode at vbox64_sas_LSILogicSAS:
>> https://openqa.oi.mnowak.cz/tests/1795
>> 
>> My test-host/worker - a 5 year old 4 core Ivy Bridge laptop with 12
>> GB of RAM, and SSD - is able to run 3 jobs in parallel reliably.
>> Full build validation takes about 8 hours. Bare metal server is
>> certainly desired.
>> 
>> I believe that this test suite could help us produce OpenIndiana
>> hipster snapshots more validated, with higher level of confidence
>> to particular updates, periodically run package's upstream
>> testsuite in real-life environment as opposed to a bare-bone build
>> environment, or release snapshots more often (if desired).
>> 
>> How to get involved: Deploy your local openQA instance in
>> accordance to the documentation at http://open.qa on a Linux host
>> (sic; I never deployed openQA on illumos as I don't have illumos
>> capable hardware), follow OpenIndiana-specific guide at
>> https://github.com/Mno-hime/os-autoinst-distri-openindiana/blob/master/README.md,
>> clone particular job from openqa.oi.mnowak.cz, see how your changes
>> affect the test run, and contribute the code back as a pull
>> request.
>> 
>> Similar test suite can be implemented for other illumos
>> distributions like OmniOS CE and SmartOS.
>> 
>> Feel free to contact me (mnowak_) on #illumos or #oi-dev.
>> 
>> Michal
>> 
>> _______________________________________________ oi-dev mailing
>> list oi-dev at openindiana.org 
>> https://openindiana.org/mailman/listinfo/oi-dev
> 
> 
> 
> _______________________________________________ oi-dev mailing list 
> oi-dev at openindiana.org 
> https://openindiana.org/mailman/listinfo/oi-dev
>