[oi-dev] Automated test suite for OpenIndiana

[Adam Števko]

Hello,

nice job!

Few questions:

1) Would you be able to incorporate vagrant box testing?
2) We could try and get an instance for running this. Can this run inside the VM reliably? I see KVM/Xen/VirtualBox host. Can you confirm these are not your laptop, but dedicated hosts?

With this there is also a much more incentive to automate the creation of images as tests can run. Is there a good CLI client/API for managing openqa instance?

Cheers
Adam

> On Nov 24, 2017, at 9:28 PM, Michal Nowak <mnowak at startmail.com> wrote:
> 
> Hello,
> 
> last couple of months I have been working on automated test suite for OpenIndiana. You can find it's dashboard at
> 
>  https://openqa.oi.mnowak.cz
> 
> It makes use of openQA (http://open.qa/), an open source automated test tool that makes possible to test the whole installation process of operating system, OS configuration, regression and sanity testing of individual components. openQA is currently used by Fedora (https://openqa.fedoraproject.org/) and openSUSE (https://openqa.opensuse.org/) for package updates and snapshot release testing, respectively.
> 
> Test suite sources and guidance on how to setup your local openQA instance are on GitHub
> 
>  https://github.com/Mno-hime/os-autoinst-distri-openindiana/
> 
> Currently Live, Text, and Minimal variants of both ISO and USB images are tested as VM guests on KVM, VirtualBox, and Xen.
> 
> Some of the test scenarios implemented:
> 
> * General installation via text- as well as GUI-installer to disk, reboot to the installed system, OS setup, shutdown.
>  - installation_mate at 64bit: https://openqa.oi.mnowak.cz/tests/1820 (on KVM host)
>  - installation_textmode_raidz at 64bit: https://openqa.oi.mnowak.cz/tests/1841 (on KVM host)
>  - installation_textmode_mbr at vbox64_ide_piix3: https://openqa.oi.mnowak.cz/tests/1805 (on VirtualBox host)
>  - installation_mate at xen-hvm: https://openqa.oi.mnowak.cz/tests/1846 (on Xen host; note: illumos guest Xen drivers are in bad shape atm)
> 
> * Upgrade scenarios for latest OpenIndiana snapshots to the latest incorporation, e.g. upgrade_textmode_20161030: https://openqa.oi.mnowak.cz/tests/1808
> 
> * illumos KVM virtualization test: https://openqa.oi.mnowak.cz/tests/1796
> 
> * Extended `pkg` console tests: https://openqa.oi.mnowak.cz/tests/1816
> 
> * Toolchain (pkg_src & oi-userland build) tests: https://openqa.oi.mnowak.cz/tests/1815
> 
> * Build of OpenIndiana snapshot images from latest published packages: https://openqa.oi.mnowak.cz/group_overview/2
> 
> * When things go wrong: installation_textmode at vbox64_sas_LSILogicSAS: https://openqa.oi.mnowak.cz/tests/1795
> 
> My test-host/worker - a 5 year old 4 core Ivy Bridge laptop with 12 GB of RAM, and SSD - is able to run 3 jobs in parallel reliably. Full build validation takes about 8 hours. Bare metal server is certainly desired.
> 
> I believe that this test suite could help us produce OpenIndiana hipster snapshots more validated, with higher level of confidence to particular updates, periodically run package's upstream testsuite in real-life environment as opposed to a bare-bone build environment, or release snapshots more often (if desired).
> 
> How to get involved: Deploy your local openQA instance in accordance to the documentation at http://open.qa on a Linux host (sic; I never deployed openQA on illumos as I don't have illumos capable hardware), follow OpenIndiana-specific guide at https://github.com/Mno-hime/os-autoinst-distri-openindiana/blob/master/README.md, clone particular job from openqa.oi.mnowak.cz, see how your changes affect the test run, and contribute the code back as a pull request.
> 
> Similar test suite can be implemented for other illumos distributions like OmniOS CE and SmartOS.
> 
> Feel free to contact me (mnowak_) on #illumos or #oi-dev.
> 
> Michal
> 
> _______________________________________________
> oi-dev mailing list
> oi-dev at openindiana.org
> https://openindiana.org/mailman/listinfo/oi-dev

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://openindiana.org/pipermail/oi-dev/attachments/20171125/2a8b5757/attachment-0005.bin>