testing.rst 52 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757758759760761762763764765766767768769770771772773774775776777778779780781782783784785786787788789790791792793794795796797798799800801802803804805806807808809810811812813814815816817818819820821822823824825826827828829830831832833834835836837838839840841842843844845846847848849850851852853854855856857858859860861862863864865866867868869870871872873874875876877878879880881882883884885886887888889890891892893894895896897898899900901902903904905906907908909910911912913914915916917918919920921922923924925926927928929930931932933934935936937938939940941942943944945946947948949950951952953954955956957958959960961962963964965966967968969970971972973974975976977978979980981982983984985986987988989990991992993994995996997998999100010011002100310041005100610071008100910101011101210131014101510161017101810191020102110221023102410251026102710281029103010311032103310341035103610371038103910401041104210431044104510461047104810491050105110521053105410551056105710581059106010611062106310641065106610671068106910701071107210731074107510761077107810791080108110821083108410851086108710881089109010911092109310941095109610971098109911001101110211031104110511061107110811091110111111121113111411151116111711181119112011211122112311241125112611271128112911301131113211331134113511361137113811391140114111421143114411451146114711481149115011511152115311541155115611571158115911601161116211631164116511661167116811691170117111721173117411751176117711781179118011811182118311841185118611871188118911901191119211931194119511961197119811991200120112021203120412051206120712081209121012111212121312141215121612171218121912201221122212231224122512261227122812291230123112321233123412351236123712381239124012411242124312441245124612471248124912501251125212531254125512561257125812591260126112621263126412651266126712681269127012711272127312741275127612771278127912801281128212831284128512861287128812891290129112921293129412951296129712981299130013011302130313041305130613071308130913101311131213131314131513161317131813191320132113221323132413251326132713281329133013311332133313341335133613371338133913401341134213431344134513461347134813491350135113521353135413551356135713581359136013611362136313641365136613671368136913701371137213731374137513761377137813791380138113821383138413851386138713881389139013911392139313941395139613971398139914001401140214031404140514061407140814091410141114121413141414151416141714181419142014211422142314241425142614271428142914301431143214331434143514361437143814391440144114421443144414451446144714481449145014511452145314541455145614571458145914601461146214631464146514661467146814691470147114721473147414751476147714781479148014811482148314841485148614871488148914901491
  1. .. _testing:
  2. Testing in QEMU
  3. ===============
  4. This document describes the testing infrastructure in QEMU.
  5. Testing with "make check"
  6. -------------------------
  7. The "make check" testing family includes most of the C based tests in QEMU. For
  8. a quick help, run ``make check-help`` from the source tree.
  9. The usual way to run these tests is:
  10. .. code::
  11. make check
  12. which includes QAPI schema tests, unit tests, QTests and some iotests.
  13. Different sub-types of "make check" tests will be explained below.
  14. Before running tests, it is best to build QEMU programs first. Some tests
  15. expect the executables to exist and will fail with obscure messages if they
  16. cannot find them.
  17. Unit tests
  18. ~~~~~~~~~~
  19. Unit tests, which can be invoked with ``make check-unit``, are simple C tests
  20. that typically link to individual QEMU object files and exercise them by
  21. calling exported functions.
  22. If you are writing new code in QEMU, consider adding a unit test, especially
  23. for utility modules that are relatively stateless or have few dependencies. To
  24. add a new unit test:
  25. 1. Create a new source file. For example, ``tests/unit/foo-test.c``.
  26. 2. Write the test. Normally you would include the header file which exports
  27. the module API, then verify the interface behaves as expected from your
  28. test. The test code should be organized with the glib testing framework.
  29. Copying and modifying an existing test is usually a good idea.
  30. 3. Add the test to ``tests/unit/meson.build``. The unit tests are listed in a
  31. dictionary called ``tests``. The values are any additional sources and
  32. dependencies to be linked with the test. For a simple test whose source
  33. is in ``tests/unit/foo-test.c``, it is enough to add an entry like::
  34. {
  35. ...
  36. 'foo-test': [],
  37. ...
  38. }
  39. Since unit tests don't require environment variables, the simplest way to debug
  40. a unit test failure is often directly invoking it or even running it under
  41. ``gdb``. However there can still be differences in behavior between ``make``
  42. invocations and your manual run, due to ``$MALLOC_PERTURB_`` environment
  43. variable (which affects memory reclamation and catches invalid pointers better)
  44. and gtester options. If necessary, you can run
  45. .. code::
  46. make check-unit V=1
  47. and copy the actual command line which executes the unit test, then run
  48. it from the command line.
  49. QTest
  50. ~~~~~
  51. QTest is a device emulation testing framework. It can be very useful to test
  52. device models; it could also control certain aspects of QEMU (such as virtual
  53. clock stepping), with a special purpose "qtest" protocol. Refer to
  54. :doc:`qtest` for more details.
  55. QTest cases can be executed with
  56. .. code::
  57. make check-qtest
  58. Writing portable test cases
  59. ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  60. Both unit tests and qtests can run on POSIX hosts as well as Windows hosts.
  61. Care must be taken when writing portable test cases that can be built and run
  62. successfully on various hosts. The following list shows some best practices:
  63. * Use portable APIs from glib whenever necessary, e.g.: g_setenv(),
  64. g_mkdtemp(), g_mkdir().
  65. * Avoid using hardcoded /tmp for temporary file directory.
  66. Use g_get_tmp_dir() instead.
  67. * Bear in mind that Windows has different special string representation for
  68. stdin/stdout/stderr and null devices. For example if your test case uses
  69. "/dev/fd/2" and "/dev/null" on Linux, remember to use "2" and "nul" on
  70. Windows instead. Also IO redirection does not work on Windows, so avoid
  71. using "2>nul" whenever necessary.
  72. * If your test cases uses the blkdebug feature, use relative path to pass
  73. the config and image file paths in the command line as Windows absolute
  74. path contains the delimiter ":" which will confuse the blkdebug parser.
  75. * Use double quotes in your extra QEMU command line in your test cases
  76. instead of single quotes, as Windows does not drop single quotes when
  77. passing the command line to QEMU.
  78. * Windows opens a file in text mode by default, while a POSIX compliant
  79. implementation treats text files and binary files the same. So if your
  80. test cases opens a file to write some data and later wants to compare the
  81. written data with the original one, be sure to pass the letter 'b' as
  82. part of the mode string to fopen(), or O_BINARY flag for the open() call.
  83. * If a certain test case can only run on POSIX or Linux hosts, use a proper
  84. #ifdef in the codes. If the whole test suite cannot run on Windows, disable
  85. the build in the meson.build file.
  86. QAPI schema tests
  87. ~~~~~~~~~~~~~~~~~
  88. The QAPI schema tests validate the QAPI parser used by QMP, by feeding
  89. predefined input to the parser and comparing the result with the reference
  90. output.
  91. The input/output data is managed under the ``tests/qapi-schema`` directory.
  92. Each test case includes four files that have a common base name:
  93. * ``${casename}.json`` - the file contains the JSON input for feeding the
  94. parser
  95. * ``${casename}.out`` - the file contains the expected stdout from the parser
  96. * ``${casename}.err`` - the file contains the expected stderr from the parser
  97. * ``${casename}.exit`` - the expected error code
  98. Consider adding a new QAPI schema test when you are making a change on the QAPI
  99. parser (either fixing a bug or extending/modifying the syntax). To do this:
  100. 1. Add four files for the new case as explained above. For example:
  101. ``$EDITOR tests/qapi-schema/foo.{json,out,err,exit}``.
  102. 2. Add the new test in ``tests/Makefile.include``. For example:
  103. ``qapi-schema += foo.json``
  104. check-block
  105. ~~~~~~~~~~~
  106. ``make check-block`` runs a subset of the block layer iotests (the tests that
  107. are in the "auto" group).
  108. See the "QEMU iotests" section below for more information.
  109. QEMU iotests
  110. ------------
  111. QEMU iotests, under the directory ``tests/qemu-iotests``, is the testing
  112. framework widely used to test block layer related features. It is higher level
  113. than "make check" tests and 99% of the code is written in bash or Python
  114. scripts. The testing success criteria is golden output comparison, and the
  115. test files are named with numbers.
  116. To run iotests, make sure QEMU is built successfully, then switch to the
  117. ``tests/qemu-iotests`` directory under the build directory, and run ``./check``
  118. with desired arguments from there.
  119. By default, "raw" format and "file" protocol is used; all tests will be
  120. executed, except the unsupported ones. You can override the format and protocol
  121. with arguments:
  122. .. code::
  123. # test with qcow2 format
  124. ./check -qcow2
  125. # or test a different protocol
  126. ./check -nbd
  127. It's also possible to list test numbers explicitly:
  128. .. code::
  129. # run selected cases with qcow2 format
  130. ./check -qcow2 001 030 153
  131. Cache mode can be selected with the "-c" option, which may help reveal bugs
  132. that are specific to certain cache mode.
  133. More options are supported by the ``./check`` script, run ``./check -h`` for
  134. help.
  135. Writing a new test case
  136. ~~~~~~~~~~~~~~~~~~~~~~~
  137. Consider writing a tests case when you are making any changes to the block
  138. layer. An iotest case is usually the choice for that. There are already many
  139. test cases, so it is possible that extending one of them may achieve the goal
  140. and save the boilerplate to create one. (Unfortunately, there isn't a 100%
  141. reliable way to find a related one out of hundreds of tests. One approach is
  142. using ``git grep``.)
  143. Usually an iotest case consists of two files. One is an executable that
  144. produces output to stdout and stderr, the other is the expected reference
  145. output. They are given the same number in file names. E.g. Test script ``055``
  146. and reference output ``055.out``.
  147. In rare cases, when outputs differ between cache mode ``none`` and others, a
  148. ``.out.nocache`` file is added. In other cases, when outputs differ between
  149. image formats, more than one ``.out`` files are created ending with the
  150. respective format names, e.g. ``178.out.qcow2`` and ``178.out.raw``.
  151. There isn't a hard rule about how to write a test script, but a new test is
  152. usually a (copy and) modification of an existing case. There are a few
  153. commonly used ways to create a test:
  154. * A Bash script. It will make use of several environmental variables related
  155. to the testing procedure, and could source a group of ``common.*`` libraries
  156. for some common helper routines.
  157. * A Python unittest script. Import ``iotests`` and create a subclass of
  158. ``iotests.QMPTestCase``, then call ``iotests.main`` method. The downside of
  159. this approach is that the output is too scarce, and the script is considered
  160. harder to debug.
  161. * A simple Python script without using unittest module. This could also import
  162. ``iotests`` for launching QEMU and utilities etc, but it doesn't inherit
  163. from ``iotests.QMPTestCase`` therefore doesn't use the Python unittest
  164. execution. This is a combination of 1 and 2.
  165. Pick the language per your preference since both Bash and Python have
  166. comparable library support for invoking and interacting with QEMU programs. If
  167. you opt for Python, it is strongly recommended to write Python 3 compatible
  168. code.
  169. Both Python and Bash frameworks in iotests provide helpers to manage test
  170. images. They can be used to create and clean up images under the test
  171. directory. If no I/O or any protocol specific feature is needed, it is often
  172. more convenient to use the pseudo block driver, ``null-co://``, as the test
  173. image, which doesn't require image creation or cleaning up. Avoid system-wide
  174. devices or files whenever possible, such as ``/dev/null`` or ``/dev/zero``.
  175. Otherwise, image locking implications have to be considered. For example,
  176. another application on the host may have locked the file, possibly leading to a
  177. test failure. If using such devices are explicitly desired, consider adding
  178. ``locking=off`` option to disable image locking.
  179. Debugging a test case
  180. ~~~~~~~~~~~~~~~~~~~~~
  181. The following options to the ``check`` script can be useful when debugging
  182. a failing test:
  183. * ``-gdb`` wraps every QEMU invocation in a ``gdbserver``, which waits for a
  184. connection from a gdb client. The options given to ``gdbserver`` (e.g. the
  185. address on which to listen for connections) are taken from the ``$GDB_OPTIONS``
  186. environment variable. By default (if ``$GDB_OPTIONS`` is empty), it listens on
  187. ``localhost:12345``.
  188. It is possible to connect to it for example with
  189. ``gdb -iex "target remote $addr"``, where ``$addr`` is the address
  190. ``gdbserver`` listens on.
  191. If the ``-gdb`` option is not used, ``$GDB_OPTIONS`` is ignored,
  192. regardless of whether it is set or not.
  193. * ``-valgrind`` attaches a valgrind instance to QEMU. If it detects
  194. warnings, it will print and save the log in
  195. ``$TEST_DIR/<valgrind_pid>.valgrind``.
  196. The final command line will be ``valgrind --log-file=$TEST_DIR/
  197. <valgrind_pid>.valgrind --error-exitcode=99 $QEMU ...``
  198. * ``-d`` (debug) just increases the logging verbosity, showing
  199. for example the QMP commands and answers.
  200. * ``-p`` (print) redirects QEMU’s stdout and stderr to the test output,
  201. instead of saving it into a log file in
  202. ``$TEST_DIR/qemu-machine-<random_string>``.
  203. Test case groups
  204. ~~~~~~~~~~~~~~~~
  205. "Tests may belong to one or more test groups, which are defined in the form
  206. of a comment in the test source file. By convention, test groups are listed
  207. in the second line of the test file, after the "#!/..." line, like this:
  208. .. code::
  209. #!/usr/bin/env python3
  210. # group: auto quick
  211. #
  212. ...
  213. Another way of defining groups is creating the tests/qemu-iotests/group.local
  214. file. This should be used only for downstream (this file should never appear
  215. in upstream). This file may be used for defining some downstream test groups
  216. or for temporarily disabling tests, like this:
  217. .. code::
  218. # groups for some company downstream process
  219. #
  220. # ci - tests to run on build
  221. # down - our downstream tests, not for upstream
  222. #
  223. # Format of each line is:
  224. # TEST_NAME TEST_GROUP [TEST_GROUP ]...
  225. 013 ci
  226. 210 disabled
  227. 215 disabled
  228. our-ugly-workaround-test down ci
  229. Note that the following group names have a special meaning:
  230. - quick: Tests in this group should finish within a few seconds.
  231. - auto: Tests in this group are used during "make check" and should be
  232. runnable in any case. That means they should run with every QEMU binary
  233. (also non-x86), with every QEMU configuration (i.e. must not fail if
  234. an optional feature is not compiled in - but reporting a "skip" is ok),
  235. work at least with the qcow2 file format, work with all kind of host
  236. filesystems and users (e.g. "nobody" or "root") and must not take too
  237. much memory and disk space (since CI pipelines tend to fail otherwise).
  238. - disabled: Tests in this group are disabled and ignored by check.
  239. .. _container-ref:
  240. Container based tests
  241. ---------------------
  242. Introduction
  243. ~~~~~~~~~~~~
  244. The container testing framework in QEMU utilizes public images to
  245. build and test QEMU in predefined and widely accessible Linux
  246. environments. This makes it possible to expand the test coverage
  247. across distros, toolchain flavors and library versions. The support
  248. was originally written for Docker although we also support Podman as
  249. an alternative container runtime. Although many of the target
  250. names and scripts are prefixed with "docker" the system will
  251. automatically run on whichever is configured.
  252. The container images are also used to augment the generation of tests
  253. for testing TCG. See :ref:`checktcg-ref` for more details.
  254. Docker Prerequisites
  255. ~~~~~~~~~~~~~~~~~~~~
  256. Install "docker" with the system package manager and start the Docker service
  257. on your development machine, then make sure you have the privilege to run
  258. Docker commands. Typically it means setting up passwordless ``sudo docker``
  259. command or login as root. For example:
  260. .. code::
  261. $ sudo yum install docker
  262. $ # or `apt-get install docker` for Ubuntu, etc.
  263. $ sudo systemctl start docker
  264. $ sudo docker ps
  265. The last command should print an empty table, to verify the system is ready.
  266. An alternative method to set up permissions is by adding the current user to
  267. "docker" group and making the docker daemon socket file (by default
  268. ``/var/run/docker.sock``) accessible to the group:
  269. .. code::
  270. $ sudo groupadd docker
  271. $ sudo usermod $USER -a -G docker
  272. $ sudo chown :docker /var/run/docker.sock
  273. Note that any one of above configurations makes it possible for the user to
  274. exploit the whole host with Docker bind mounting or other privileged
  275. operations. So only do it on development machines.
  276. Podman Prerequisites
  277. ~~~~~~~~~~~~~~~~~~~~
  278. Install "podman" with the system package manager.
  279. .. code::
  280. $ sudo dnf install podman
  281. $ podman ps
  282. The last command should print an empty table, to verify the system is ready.
  283. Quickstart
  284. ~~~~~~~~~~
  285. From source tree, type ``make docker-help`` to see the help. Testing
  286. can be started without configuring or building QEMU (``configure`` and
  287. ``make`` are done in the container, with parameters defined by the
  288. make target):
  289. .. code::
  290. make docker-test-build@centos8
  291. This will create a container instance using the ``centos8`` image (the image
  292. is downloaded and initialized automatically), in which the ``test-build`` job
  293. is executed.
  294. Registry
  295. ~~~~~~~~
  296. The QEMU project has a container registry hosted by GitLab at
  297. ``registry.gitlab.com/qemu-project/qemu`` which will automatically be
  298. used to pull in pre-built layers. This avoids unnecessary strain on
  299. the distro archives created by multiple developers running the same
  300. container build steps over and over again. This can be overridden
  301. locally by using the ``NOCACHE`` build option:
  302. .. code::
  303. make docker-image-debian-arm64-cross NOCACHE=1
  304. Images
  305. ~~~~~~
  306. Along with many other images, the ``centos8`` image is defined in a Dockerfile
  307. in ``tests/docker/dockerfiles/``, called ``centos8.docker``. ``make docker-help``
  308. command will list all the available images.
  309. A ``.pre`` script can be added beside the ``.docker`` file, which will be
  310. executed before building the image under the build context directory. This is
  311. mainly used to do necessary host side setup. One such setup is ``binfmt_misc``,
  312. for example, to make qemu-user powered cross build containers work.
  313. Most of the existing Dockerfiles were written by hand, simply by creating a
  314. a new ``.docker`` file under the ``tests/docker/dockerfiles/`` directory.
  315. This has led to an inconsistent set of packages being present across the
  316. different containers.
  317. Thus going forward, QEMU is aiming to automatically generate the Dockerfiles
  318. using the ``lcitool`` program provided by the ``libvirt-ci`` project:
  319. https://gitlab.com/libvirt/libvirt-ci
  320. ``libvirt-ci`` contains an ``lcitool`` program as well as a list of
  321. mappings to distribution package names for a wide variety of third
  322. party projects. ``lcitool`` applies the mappings to a list of build
  323. pre-requisites in ``tests/lcitool/projects/qemu.yml``, determines the
  324. list of native packages to install on each distribution, and uses them
  325. to generate build environments (dockerfiles and Cirrus CI variable files)
  326. that are consistent across OS distribution.
  327. Adding new build pre-requisites
  328. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  329. When preparing a patch series that adds a new build
  330. pre-requisite to QEMU, the prerequisites should to be added to
  331. ``tests/lcitool/projects/qemu.yml`` in order to make the dependency
  332. available in the CI build environments.
  333. In the simple case where the pre-requisite is already known to ``libvirt-ci``
  334. the following steps are needed:
  335. * Edit ``tests/lcitool/projects/qemu.yml`` and add the pre-requisite
  336. * Run ``make lcitool-refresh`` to re-generate all relevant build environment
  337. manifests
  338. It may be that ``libvirt-ci`` does not know about the new pre-requisite.
  339. If that is the case, some extra preparation steps will be required
  340. first to contribute the mapping to the ``libvirt-ci`` project:
  341. * Fork the ``libvirt-ci`` project on gitlab
  342. * Add an entry for the new build prerequisite to
  343. ``lcitool/facts/mappings.yml``, listing its native package name on as
  344. many OS distros as practical. Run ``python -m pytest --regenerate-output``
  345. and check that the changes are correct.
  346. * Commit the ``mappings.yml`` change together with the regenerated test
  347. files, and submit a merge request to the ``libvirt-ci`` project.
  348. Please note in the description that this is a new build pre-requisite
  349. desired for use with QEMU.
  350. * CI pipeline will run to validate that the changes to ``mappings.yml``
  351. are correct, by attempting to install the newly listed package on
  352. all OS distributions supported by ``libvirt-ci``.
  353. * Once the merge request is accepted, go back to QEMU and update
  354. the ``tests/lcitool/libvirt-ci`` submodule to point to a commit that
  355. contains the ``mappings.yml`` update. Then add the prerequisite and
  356. run ``make lcitool-refresh``.
  357. For enterprise distros that default to old, end-of-life versions of the
  358. Python runtime, QEMU uses a separate set of mappings that work with more
  359. recent versions. These can be found in ``tests/lcitool/mappings.yml``.
  360. Modifying this file should not be necessary unless the new pre-requisite
  361. is a Python library or tool.
  362. Adding new OS distros
  363. ^^^^^^^^^^^^^^^^^^^^^
  364. In some cases ``libvirt-ci`` will not know about the OS distro that is
  365. desired to be tested. Before adding a new OS distro, discuss the proposed
  366. addition:
  367. * Send a mail to qemu-devel, copying people listed in the
  368. MAINTAINERS file for ``Build and test automation``.
  369. There are limited CI compute resources available to QEMU, so the
  370. cost/benefit tradeoff of adding new OS distros needs to be considered.
  371. * File an issue at https://gitlab.com/libvirt/libvirt-ci/-/issues
  372. pointing to the qemu-devel mail thread in the archives.
  373. This alerts other people who might be interested in the work
  374. to avoid duplication, as well as to get feedback from libvirt-ci
  375. maintainers on any tips to ease the addition
  376. Assuming there is agreement to add a new OS distro then
  377. * Fork the ``libvirt-ci`` project on gitlab
  378. * Add metadata under ``lcitool/facts/targets/`` for the new OS
  379. distro. There might be code changes required if the OS distro
  380. uses a package format not currently known. The ``libvirt-ci``
  381. maintainers can advise on this when the issue is filed.
  382. * Edit the ``lcitool/facts/mappings.yml`` change to add entries for
  383. the new OS, listing the native package names for as many packages
  384. as practical. Run ``python -m pytest --regenerate-output`` and
  385. check that the changes are correct.
  386. * Commit the changes to ``lcitool/facts`` and the regenerated test
  387. files, and submit a merge request to the ``libvirt-ci`` project.
  388. Please note in the description that this is a new build pre-requisite
  389. desired for use with QEMU
  390. * CI pipeline will run to validate that the changes to ``mappings.yml``
  391. are correct, by attempting to install the newly listed package on
  392. all OS distributions supported by ``libvirt-ci``.
  393. * Once the merge request is accepted, go back to QEMU and update
  394. the ``libvirt-ci`` submodule to point to a commit that contains
  395. the ``mappings.yml`` update.
  396. Tests
  397. ~~~~~
  398. Different tests are added to cover various configurations to build and test
  399. QEMU. Docker tests are the executables under ``tests/docker`` named
  400. ``test-*``. They are typically shell scripts and are built on top of a shell
  401. library, ``tests/docker/common.rc``, which provides helpers to find the QEMU
  402. source and build it.
  403. The full list of tests is printed in the ``make docker-help`` help.
  404. Debugging a Docker test failure
  405. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  406. When CI tasks, maintainers or yourself report a Docker test failure, follow the
  407. below steps to debug it:
  408. 1. Locally reproduce the failure with the reported command line. E.g. run
  409. ``make docker-test-mingw@fedora J=8``.
  410. 2. Add "V=1" to the command line, try again, to see the verbose output.
  411. 3. Further add "DEBUG=1" to the command line. This will pause in a shell prompt
  412. in the container right before testing starts. You could either manually
  413. build QEMU and run tests from there, or press Ctrl-D to let the Docker
  414. testing continue.
  415. 4. If you press Ctrl-D, the same building and testing procedure will begin, and
  416. will hopefully run into the error again. After that, you will be dropped to
  417. the prompt for debug.
  418. Options
  419. ~~~~~~~
  420. Various options can be used to affect how Docker tests are done. The full
  421. list is in the ``make docker`` help text. The frequently used ones are:
  422. * ``V=1``: the same as in top level ``make``. It will be propagated to the
  423. container and enable verbose output.
  424. * ``J=$N``: the number of parallel tasks in make commands in the container,
  425. similar to the ``-j $N`` option in top level ``make``. (The ``-j`` option in
  426. top level ``make`` will not be propagated into the container.)
  427. * ``DEBUG=1``: enables debug. See the previous "Debugging a Docker test
  428. failure" section.
  429. Thread Sanitizer
  430. ----------------
  431. Thread Sanitizer (TSan) is a tool which can detect data races. QEMU supports
  432. building and testing with this tool.
  433. For more information on TSan:
  434. https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual
  435. Thread Sanitizer in Docker
  436. ~~~~~~~~~~~~~~~~~~~~~~~~~~
  437. TSan is currently supported in the ubuntu2204 docker.
  438. The test-tsan test will build using TSan and then run make check.
  439. .. code::
  440. make docker-test-tsan@ubuntu2204
  441. TSan warnings under docker are placed in files located at build/tsan/.
  442. We recommend using DEBUG=1 to allow launching the test from inside the docker,
  443. and to allow review of the warnings generated by TSan.
  444. Building and Testing with TSan
  445. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  446. It is possible to build and test with TSan, with a few additional steps.
  447. These steps are normally done automatically in the docker.
  448. There is a one time patch needed in clang-9 or clang-10 at this time:
  449. .. code::
  450. sed -i 's/^const/static const/g' \
  451. /usr/lib/llvm-10/lib/clang/10.0.0/include/sanitizer/tsan_interface.h
  452. To configure the build for TSan:
  453. .. code::
  454. ../configure --enable-tsan --cc=clang-10 --cxx=clang++-10 \
  455. --disable-werror --extra-cflags="-O0"
  456. The runtime behavior of TSAN is controlled by the TSAN_OPTIONS environment
  457. variable.
  458. More information on the TSAN_OPTIONS can be found here:
  459. https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags
  460. For example:
  461. .. code::
  462. export TSAN_OPTIONS=suppressions=<path to qemu>/tests/tsan/suppressions.tsan \
  463. detect_deadlocks=false history_size=7 exitcode=0 \
  464. log_path=<build path>/tsan/tsan_warning
  465. The above exitcode=0 has TSan continue without error if any warnings are found.
  466. This allows for running the test and then checking the warnings afterwards.
  467. If you want TSan to stop and exit with error on warnings, use exitcode=66.
  468. TSan Suppressions
  469. ~~~~~~~~~~~~~~~~~
  470. Keep in mind that for any data race warning, although there might be a data race
  471. detected by TSan, there might be no actual bug here. TSan provides several
  472. different mechanisms for suppressing warnings. In general it is recommended
  473. to fix the code if possible to eliminate the data race rather than suppress
  474. the warning.
  475. A few important files for suppressing warnings are:
  476. tests/tsan/suppressions.tsan - Has TSan warnings we wish to suppress at runtime.
  477. The comment on each suppression will typically indicate why we are
  478. suppressing it. More information on the file format can be found here:
  479. https://github.com/google/sanitizers/wiki/ThreadSanitizerSuppressions
  480. tests/tsan/blacklist.tsan - Has TSan warnings we wish to disable
  481. at compile time for test or debug.
  482. Add flags to configure to enable:
  483. "--extra-cflags=-fsanitize-blacklist=<src path>/tests/tsan/blacklist.tsan"
  484. More information on the file format can be found here under "Blacklist Format":
  485. https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags
  486. TSan Annotations
  487. ~~~~~~~~~~~~~~~~
  488. include/qemu/tsan.h defines annotations. See this file for more descriptions
  489. of the annotations themselves. Annotations can be used to suppress
  490. TSan warnings or give TSan more information so that it can detect proper
  491. relationships between accesses of data.
  492. Annotation examples can be found here:
  493. https://github.com/llvm/llvm-project/tree/master/compiler-rt/test/tsan/
  494. Good files to start with are: annotate_happens_before.cpp and ignore_race.cpp
  495. The full set of annotations can be found here:
  496. https://github.com/llvm/llvm-project/blob/master/compiler-rt/lib/tsan/rtl/tsan_interface_ann.cpp
  497. docker-binfmt-image-debian-% targets
  498. ------------------------------------
  499. It is possible to combine Debian's bootstrap scripts with a configured
  500. ``binfmt_misc`` to bootstrap a number of Debian's distros including
  501. experimental ports not yet supported by a released OS. This can
  502. simplify setting up a rootfs by using docker to contain the foreign
  503. rootfs rather than manually invoking chroot.
  504. Setting up ``binfmt_misc``
  505. ~~~~~~~~~~~~~~~~~~~~~~~~~~
  506. You can use the script ``qemu-binfmt-conf.sh`` to configure a QEMU
  507. user binary to automatically run binaries for the foreign
  508. architecture. While the scripts will try their best to work with
  509. dynamically linked QEMU's a statically linked one will present less
  510. potential complications when copying into the docker image. Modern
  511. kernels support the ``F`` (fix binary) flag which will open the QEMU
  512. executable on setup and avoids the need to find and re-open in the
  513. chroot environment. This is triggered with the ``--persistent`` flag.
  514. Example invocation
  515. ~~~~~~~~~~~~~~~~~~
  516. For example to setup the HPPA ports builds of Debian::
  517. make docker-binfmt-image-debian-sid-hppa \
  518. DEB_TYPE=sid DEB_ARCH=hppa \
  519. DEB_URL=http://ftp.ports.debian.org/debian-ports/ \
  520. DEB_KEYRING=/usr/share/keyrings/debian-ports-archive-keyring.gpg \
  521. EXECUTABLE=(pwd)/qemu-hppa V=1
  522. The ``DEB_`` variables are substitutions used by
  523. ``debian-boostrap.pre`` which is called to do the initial debootstrap
  524. of the rootfs before it is copied into the container. The second stage
  525. is run as part of the build. The final image will be tagged as
  526. ``qemu/debian-sid-hppa``.
  527. VM testing
  528. ----------
  529. This test suite contains scripts that bootstrap various guest images that have
  530. necessary packages to build QEMU. The basic usage is documented in ``Makefile``
  531. help which is displayed with ``make vm-help``.
  532. Quickstart
  533. ~~~~~~~~~~
  534. Run ``make vm-help`` to list available make targets. Invoke a specific make
  535. command to run build test in an image. For example, ``make vm-build-freebsd``
  536. will build the source tree in the FreeBSD image. The command can be executed
  537. from either the source tree or the build dir; if the former, ``./configure`` is
  538. not needed. The command will then generate the test image in ``./tests/vm/``
  539. under the working directory.
  540. Note: images created by the scripts accept a well-known RSA key pair for SSH
  541. access, so they SHOULD NOT be exposed to external interfaces if you are
  542. concerned about attackers taking control of the guest and potentially
  543. exploiting a QEMU security bug to compromise the host.
  544. QEMU binaries
  545. ~~~~~~~~~~~~~
  546. By default, ``qemu-system-x86_64`` is searched in $PATH to run the guest. If
  547. there isn't one, or if it is older than 2.10, the test won't work. In this case,
  548. provide the QEMU binary in env var: ``QEMU=/path/to/qemu-2.10+``.
  549. Likewise the path to ``qemu-img`` can be set in QEMU_IMG environment variable.
  550. Make jobs
  551. ~~~~~~~~~
  552. The ``-j$X`` option in the make command line is not propagated into the VM,
  553. specify ``J=$X`` to control the make jobs in the guest.
  554. Debugging
  555. ~~~~~~~~~
  556. Add ``DEBUG=1`` and/or ``V=1`` to the make command to allow interactive
  557. debugging and verbose output. If this is not enough, see the next section.
  558. ``V=1`` will be propagated down into the make jobs in the guest.
  559. Manual invocation
  560. ~~~~~~~~~~~~~~~~~
  561. Each guest script is an executable script with the same command line options.
  562. For example to work with the netbsd guest, use ``$QEMU_SRC/tests/vm/netbsd``:
  563. .. code::
  564. $ cd $QEMU_SRC/tests/vm
  565. # To bootstrap the image
  566. $ ./netbsd --build-image --image /var/tmp/netbsd.img
  567. <...>
  568. # To run an arbitrary command in guest (the output will not be echoed unless
  569. # --debug is added)
  570. $ ./netbsd --debug --image /var/tmp/netbsd.img uname -a
  571. # To build QEMU in guest
  572. $ ./netbsd --debug --image /var/tmp/netbsd.img --build-qemu $QEMU_SRC
  573. # To get to an interactive shell
  574. $ ./netbsd --interactive --image /var/tmp/netbsd.img sh
  575. Adding new guests
  576. ~~~~~~~~~~~~~~~~~
  577. Please look at existing guest scripts for how to add new guests.
  578. Most importantly, create a subclass of BaseVM and implement ``build_image()``
  579. method and define ``BUILD_SCRIPT``, then finally call ``basevm.main()`` from
  580. the script's ``main()``.
  581. * Usually in ``build_image()``, a template image is downloaded from a
  582. predefined URL. ``BaseVM._download_with_cache()`` takes care of the cache and
  583. the checksum, so consider using it.
  584. * Once the image is downloaded, users, SSH server and QEMU build deps should
  585. be set up:
  586. - Root password set to ``BaseVM.ROOT_PASS``
  587. - User ``BaseVM.GUEST_USER`` is created, and password set to
  588. ``BaseVM.GUEST_PASS``
  589. - SSH service is enabled and started on boot,
  590. ``$QEMU_SRC/tests/keys/id_rsa.pub`` is added to ssh's ``authorized_keys``
  591. file of both root and the normal user
  592. - DHCP client service is enabled and started on boot, so that it can
  593. automatically configure the virtio-net-pci NIC and communicate with QEMU
  594. user net (10.0.2.2)
  595. - Necessary packages are installed to untar the source tarball and build
  596. QEMU
  597. * Write a proper ``BUILD_SCRIPT`` template, which should be a shell script that
  598. untars a raw virtio-blk block device, which is the tarball data blob of the
  599. QEMU source tree, then configure/build it. Running "make check" is also
  600. recommended.
  601. Image fuzzer testing
  602. --------------------
  603. An image fuzzer was added to exercise format drivers. Currently only qcow2 is
  604. supported. To start the fuzzer, run
  605. .. code::
  606. tests/image-fuzzer/runner.py -c '[["qemu-img", "info", "$test_img"]]' /tmp/test qcow2
  607. Alternatively, some command different from ``qemu-img info`` can be tested, by
  608. changing the ``-c`` option.
  609. Integration tests using the Avocado Framework
  610. ---------------------------------------------
  611. The ``tests/avocado`` directory hosts integration tests. They're usually
  612. higher level tests, and may interact with external resources and with
  613. various guest operating systems.
  614. These tests are written using the Avocado Testing Framework (which must
  615. be installed separately) in conjunction with a the ``avocado_qemu.Test``
  616. class, implemented at ``tests/avocado/avocado_qemu``.
  617. Tests based on ``avocado_qemu.Test`` can easily:
  618. * Customize the command line arguments given to the convenience
  619. ``self.vm`` attribute (a QEMUMachine instance)
  620. * Interact with the QEMU monitor, send QMP commands and check
  621. their results
  622. * Interact with the guest OS, using the convenience console device
  623. (which may be useful to assert the effectiveness and correctness of
  624. command line arguments or QMP commands)
  625. * Interact with external data files that accompany the test itself
  626. (see ``self.get_data()``)
  627. * Download (and cache) remote data files, such as firmware and kernel
  628. images
  629. * Have access to a library of guest OS images (by means of the
  630. ``avocado.utils.vmimage`` library)
  631. * Make use of various other test related utilities available at the
  632. test class itself and at the utility library:
  633. - http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test
  634. - http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html
  635. Running tests
  636. ~~~~~~~~~~~~~
  637. You can run the avocado tests simply by executing:
  638. .. code::
  639. make check-avocado
  640. This involves the automatic creation of Python virtual environment
  641. within the build tree (at ``tests/venv``) which will have all the
  642. right dependencies, and will save tests results also within the
  643. build tree (at ``tests/results``).
  644. Note: the build environment must be using a Python 3 stack, and have
  645. the ``venv`` and ``pip`` packages installed. If necessary, make sure
  646. ``configure`` is called with ``--python=`` and that those modules are
  647. available. On Debian and Ubuntu based systems, depending on the
  648. specific version, they may be on packages named ``python3-venv`` and
  649. ``python3-pip``.
  650. It is also possible to run tests based on tags using the
  651. ``make check-avocado`` command and the ``AVOCADO_TAGS`` environment
  652. variable:
  653. .. code::
  654. make check-avocado AVOCADO_TAGS=quick
  655. Note that tags separated with commas have an AND behavior, while tags
  656. separated by spaces have an OR behavior. For more information on Avocado
  657. tags, see:
  658. https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/tags.html
  659. To run a single test file, a couple of them, or a test within a file
  660. using the ``make check-avocado`` command, set the ``AVOCADO_TESTS``
  661. environment variable with the test files or test names. To run all
  662. tests from a single file, use:
  663. .. code::
  664. make check-avocado AVOCADO_TESTS=$FILEPATH
  665. The same is valid to run tests from multiple test files:
  666. .. code::
  667. make check-avocado AVOCADO_TESTS='$FILEPATH1 $FILEPATH2'
  668. To run a single test within a file, use:
  669. .. code::
  670. make check-avocado AVOCADO_TESTS=$FILEPATH:$TESTCLASS.$TESTNAME
  671. The same is valid to run single tests from multiple test files:
  672. .. code::
  673. make check-avocado AVOCADO_TESTS='$FILEPATH1:$TESTCLASS1.$TESTNAME1 $FILEPATH2:$TESTCLASS2.$TESTNAME2'
  674. The scripts installed inside the virtual environment may be used
  675. without an "activation". For instance, the Avocado test runner
  676. may be invoked by running:
  677. .. code::
  678. tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/avocado/
  679. Note that if ``make check-avocado`` was not executed before, it is
  680. possible to create the Python virtual environment with the dependencies
  681. needed running:
  682. .. code::
  683. make check-venv
  684. It is also possible to run tests from a single file or a single test within
  685. a test file. To run tests from a single file within the build tree, use:
  686. .. code::
  687. tests/venv/bin/avocado run tests/avocado/$TESTFILE
  688. To run a single test within a test file, use:
  689. .. code::
  690. tests/venv/bin/avocado run tests/avocado/$TESTFILE:$TESTCLASS.$TESTNAME
  691. Valid test names are visible in the output from any previous execution
  692. of Avocado or ``make check-avocado``, and can also be queried using:
  693. .. code::
  694. tests/venv/bin/avocado list tests/avocado
  695. Manual Installation
  696. ~~~~~~~~~~~~~~~~~~~
  697. To manually install Avocado and its dependencies, run:
  698. .. code::
  699. pip install --user avocado-framework
  700. Alternatively, follow the instructions on this link:
  701. https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/installing.html
  702. Overview
  703. ~~~~~~~~
  704. The ``tests/avocado/avocado_qemu`` directory provides the
  705. ``avocado_qemu`` Python module, containing the ``avocado_qemu.Test``
  706. class. Here's a simple usage example:
  707. .. code::
  708. from avocado_qemu import QemuSystemTest
  709. class Version(QemuSystemTest):
  710. """
  711. :avocado: tags=quick
  712. """
  713. def test_qmp_human_info_version(self):
  714. self.vm.launch()
  715. res = self.vm.command('human-monitor-command',
  716. command_line='info version')
  717. self.assertRegexpMatches(res, r'^(\d+\.\d+\.\d)')
  718. To execute your test, run:
  719. .. code::
  720. avocado run version.py
  721. Tests may be classified according to a convention by using docstring
  722. directives such as ``:avocado: tags=TAG1,TAG2``. To run all tests
  723. in the current directory, tagged as "quick", run:
  724. .. code::
  725. avocado run -t quick .
  726. The ``avocado_qemu.Test`` base test class
  727. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  728. The ``avocado_qemu.Test`` class has a number of characteristics that
  729. are worth being mentioned right away.
  730. First of all, it attempts to give each test a ready to use QEMUMachine
  731. instance, available at ``self.vm``. Because many tests will tweak the
  732. QEMU command line, launching the QEMUMachine (by using ``self.vm.launch()``)
  733. is left to the test writer.
  734. The base test class has also support for tests with more than one
  735. QEMUMachine. The way to get machines is through the ``self.get_vm()``
  736. method which will return a QEMUMachine instance. The ``self.get_vm()``
  737. method accepts arguments that will be passed to the QEMUMachine creation
  738. and also an optional ``name`` attribute so you can identify a specific
  739. machine and get it more than once through the tests methods. A simple
  740. and hypothetical example follows:
  741. .. code::
  742. from avocado_qemu import QemuSystemTest
  743. class MultipleMachines(QemuSystemTest):
  744. def test_multiple_machines(self):
  745. first_machine = self.get_vm()
  746. second_machine = self.get_vm()
  747. self.get_vm(name='third_machine').launch()
  748. first_machine.launch()
  749. second_machine.launch()
  750. first_res = first_machine.command(
  751. 'human-monitor-command',
  752. command_line='info version')
  753. second_res = second_machine.command(
  754. 'human-monitor-command',
  755. command_line='info version')
  756. third_res = self.get_vm(name='third_machine').command(
  757. 'human-monitor-command',
  758. command_line='info version')
  759. self.assertEquals(first_res, second_res, third_res)
  760. At test "tear down", ``avocado_qemu.Test`` handles all the QEMUMachines
  761. shutdown.
  762. The ``avocado_qemu.LinuxTest`` base test class
  763. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  764. The ``avocado_qemu.LinuxTest`` is further specialization of the
  765. ``avocado_qemu.Test`` class, so it contains all the characteristics of
  766. the later plus some extra features.
  767. First of all, this base class is intended for tests that need to
  768. interact with a fully booted and operational Linux guest. At this
  769. time, it uses a Fedora 31 guest image. The most basic example looks
  770. like this:
  771. .. code::
  772. from avocado_qemu import LinuxTest
  773. class SomeTest(LinuxTest):
  774. def test(self):
  775. self.launch_and_wait()
  776. self.ssh_command('some_command_to_be_run_in_the_guest')
  777. Please refer to tests that use ``avocado_qemu.LinuxTest`` under
  778. ``tests/avocado`` for more examples.
  779. QEMUMachine
  780. ~~~~~~~~~~~
  781. The QEMUMachine API is already widely used in the Python iotests,
  782. device-crash-test and other Python scripts. It's a wrapper around the
  783. execution of a QEMU binary, giving its users:
  784. * the ability to set command line arguments to be given to the QEMU
  785. binary
  786. * a ready to use QMP connection and interface, which can be used to
  787. send commands and inspect its results, as well as asynchronous
  788. events
  789. * convenience methods to set commonly used command line arguments in
  790. a more succinct and intuitive way
  791. QEMU binary selection
  792. ^^^^^^^^^^^^^^^^^^^^^
  793. The QEMU binary used for the ``self.vm`` QEMUMachine instance will
  794. primarily depend on the value of the ``qemu_bin`` parameter. If it's
  795. not explicitly set, its default value will be the result of a dynamic
  796. probe in the same source tree. A suitable binary will be one that
  797. targets the architecture matching host machine.
  798. Based on this description, test writers will usually rely on one of
  799. the following approaches:
  800. 1) Set ``qemu_bin``, and use the given binary
  801. 2) Do not set ``qemu_bin``, and use a QEMU binary named like
  802. "qemu-system-${arch}", either in the current
  803. working directory, or in the current source tree.
  804. The resulting ``qemu_bin`` value will be preserved in the
  805. ``avocado_qemu.Test`` as an attribute with the same name.
  806. Attribute reference
  807. ~~~~~~~~~~~~~~~~~~~
  808. Test
  809. ^^^^
  810. Besides the attributes and methods that are part of the base
  811. ``avocado.Test`` class, the following attributes are available on any
  812. ``avocado_qemu.Test`` instance.
  813. vm
  814. ''
  815. A QEMUMachine instance, initially configured according to the given
  816. ``qemu_bin`` parameter.
  817. arch
  818. ''''
  819. The architecture can be used on different levels of the stack, e.g. by
  820. the framework or by the test itself. At the framework level, it will
  821. currently influence the selection of a QEMU binary (when one is not
  822. explicitly given).
  823. Tests are also free to use this attribute value, for their own needs.
  824. A test may, for instance, use the same value when selecting the
  825. architecture of a kernel or disk image to boot a VM with.
  826. The ``arch`` attribute will be set to the test parameter of the same
  827. name. If one is not given explicitly, it will either be set to
  828. ``None``, or, if the test is tagged with one (and only one)
  829. ``:avocado: tags=arch:VALUE`` tag, it will be set to ``VALUE``.
  830. cpu
  831. '''
  832. The cpu model that will be set to all QEMUMachine instances created
  833. by the test.
  834. The ``cpu`` attribute will be set to the test parameter of the same
  835. name. If one is not given explicitly, it will either be set to
  836. ``None ``, or, if the test is tagged with one (and only one)
  837. ``:avocado: tags=cpu:VALUE`` tag, it will be set to ``VALUE``.
  838. machine
  839. '''''''
  840. The machine type that will be set to all QEMUMachine instances created
  841. by the test.
  842. The ``machine`` attribute will be set to the test parameter of the same
  843. name. If one is not given explicitly, it will either be set to
  844. ``None``, or, if the test is tagged with one (and only one)
  845. ``:avocado: tags=machine:VALUE`` tag, it will be set to ``VALUE``.
  846. qemu_bin
  847. ''''''''
  848. The preserved value of the ``qemu_bin`` parameter or the result of the
  849. dynamic probe for a QEMU binary in the current working directory or
  850. source tree.
  851. LinuxTest
  852. ^^^^^^^^^
  853. Besides the attributes present on the ``avocado_qemu.Test`` base
  854. class, the ``avocado_qemu.LinuxTest`` adds the following attributes:
  855. distro
  856. ''''''
  857. The name of the Linux distribution used as the guest image for the
  858. test. The name should match the **Provider** column on the list
  859. of images supported by the avocado.utils.vmimage library:
  860. https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
  861. distro_version
  862. ''''''''''''''
  863. The version of the Linux distribution as the guest image for the
  864. test. The name should match the **Version** column on the list
  865. of images supported by the avocado.utils.vmimage library:
  866. https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
  867. distro_checksum
  868. '''''''''''''''
  869. The sha256 hash of the guest image file used for the test.
  870. If this value is not set in the code or by a test parameter (with the
  871. same name), no validation on the integrity of the image will be
  872. performed.
  873. Parameter reference
  874. ~~~~~~~~~~~~~~~~~~~
  875. To understand how Avocado parameters are accessed by tests, and how
  876. they can be passed to tests, please refer to::
  877. https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#accessing-test-parameters
  878. Parameter values can be easily seen in the log files, and will look
  879. like the following:
  880. .. code::
  881. PARAMS (key=qemu_bin, path=*, default=./qemu-system-x86_64) => './qemu-system-x86_64
  882. Test
  883. ^^^^
  884. arch
  885. ''''
  886. The architecture that will influence the selection of a QEMU binary
  887. (when one is not explicitly given).
  888. Tests are also free to use this parameter value, for their own needs.
  889. A test may, for instance, use the same value when selecting the
  890. architecture of a kernel or disk image to boot a VM with.
  891. This parameter has a direct relation with the ``arch`` attribute. If
  892. not given, it will default to None.
  893. cpu
  894. '''
  895. The cpu model that will be set to all QEMUMachine instances created
  896. by the test.
  897. machine
  898. '''''''
  899. The machine type that will be set to all QEMUMachine instances created
  900. by the test.
  901. qemu_bin
  902. ''''''''
  903. The exact QEMU binary to be used on QEMUMachine.
  904. LinuxTest
  905. ^^^^^^^^^
  906. Besides the parameters present on the ``avocado_qemu.Test`` base
  907. class, the ``avocado_qemu.LinuxTest`` adds the following parameters:
  908. distro
  909. ''''''
  910. The name of the Linux distribution used as the guest image for the
  911. test. The name should match the **Provider** column on the list
  912. of images supported by the avocado.utils.vmimage library:
  913. https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
  914. distro_version
  915. ''''''''''''''
  916. The version of the Linux distribution as the guest image for the
  917. test. The name should match the **Version** column on the list
  918. of images supported by the avocado.utils.vmimage library:
  919. https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
  920. distro_checksum
  921. '''''''''''''''
  922. The sha256 hash of the guest image file used for the test.
  923. If this value is not set in the code or by this parameter no
  924. validation on the integrity of the image will be performed.
  925. Skipping tests
  926. ~~~~~~~~~~~~~~
  927. The Avocado framework provides Python decorators which allow for easily skip
  928. tests running under certain conditions. For example, on the lack of a binary
  929. on the test system or when the running environment is a CI system. For further
  930. information about those decorators, please refer to::
  931. https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#skipping-tests
  932. While the conditions for skipping tests are often specifics of each one, there
  933. are recurring scenarios identified by the QEMU developers and the use of
  934. environment variables became a kind of standard way to enable/disable tests.
  935. Here is a list of the most used variables:
  936. AVOCADO_ALLOW_LARGE_STORAGE
  937. ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  938. Tests which are going to fetch or produce assets considered *large* are not
  939. going to run unless that ``AVOCADO_ALLOW_LARGE_STORAGE=1`` is exported on
  940. the environment.
  941. The definition of *large* is a bit arbitrary here, but it usually means an
  942. asset which occupies at least 1GB of size on disk when uncompressed.
  943. AVOCADO_ALLOW_UNTRUSTED_CODE
  944. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  945. There are tests which will boot a kernel image or firmware that can be
  946. considered not safe to run on the developer's workstation, thus they are
  947. skipped by default. The definition of *not safe* is also arbitrary but
  948. usually it means a blob which either its source or build process aren't
  949. public available.
  950. You should export ``AVOCADO_ALLOW_UNTRUSTED_CODE=1`` on the environment in
  951. order to allow tests which make use of those kind of assets.
  952. AVOCADO_TIMEOUT_EXPECTED
  953. ^^^^^^^^^^^^^^^^^^^^^^^^
  954. The Avocado framework has a timeout mechanism which interrupts tests to avoid the
  955. test suite of getting stuck. The timeout value can be set via test parameter or
  956. property defined in the test class, for further details::
  957. https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#setting-a-test-timeout
  958. Even though the timeout can be set by the test developer, there are some tests
  959. that may not have a well-defined limit of time to finish under certain
  960. conditions. For example, tests that take longer to execute when QEMU is
  961. compiled with debug flags. Therefore, the ``AVOCADO_TIMEOUT_EXPECTED`` variable
  962. has been used to determine whether those tests should run or not.
  963. GITLAB_CI
  964. ^^^^^^^^^
  965. A number of tests are flagged to not run on the GitLab CI. Usually because
  966. they proved to the flaky or there are constraints on the CI environment which
  967. would make them fail. If you encounter a similar situation then use that
  968. variable as shown on the code snippet below to skip the test:
  969. .. code::
  970. @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
  971. def test(self):
  972. do_something()
  973. Uninstalling Avocado
  974. ~~~~~~~~~~~~~~~~~~~~
  975. If you've followed the manual installation instructions above, you can
  976. easily uninstall Avocado. Start by listing the packages you have
  977. installed::
  978. pip list --user
  979. And remove any package you want with::
  980. pip uninstall <package_name>
  981. If you've used ``make check-avocado``, the Python virtual environment where
  982. Avocado is installed will be cleaned up as part of ``make check-clean``.
  983. .. _checktcg-ref:
  984. Testing with "make check-tcg"
  985. -----------------------------
  986. The check-tcg tests are intended for simple smoke tests of both
  987. linux-user and softmmu TCG functionality. However to build test
  988. programs for guest targets you need to have cross compilers available.
  989. If your distribution supports cross compilers you can do something as
  990. simple as::
  991. apt install gcc-aarch64-linux-gnu
  992. The configure script will automatically pick up their presence.
  993. Sometimes compilers have slightly odd names so the availability of
  994. them can be prompted by passing in the appropriate configure option
  995. for the architecture in question, for example::
  996. $(configure) --cross-cc-aarch64=aarch64-cc
  997. There is also a ``--cross-cc-cflags-ARCH`` flag in case additional
  998. compiler flags are needed to build for a given target.
  999. If you have the ability to run containers as the user the build system
  1000. will automatically use them where no system compiler is available. For
  1001. architectures where we also support building QEMU we will generally
  1002. use the same container to build tests. However there are a number of
  1003. additional containers defined that have a minimal cross-build
  1004. environment that is only suitable for building test cases. Sometimes
  1005. we may use a bleeding edge distribution for compiler features needed
  1006. for test cases that aren't yet in the LTS distros we support for QEMU
  1007. itself.
  1008. See :ref:`container-ref` for more details.
  1009. Running subset of tests
  1010. ~~~~~~~~~~~~~~~~~~~~~~~
  1011. You can build the tests for one architecture::
  1012. make build-tcg-tests-$TARGET
  1013. And run with::
  1014. make run-tcg-tests-$TARGET
  1015. Adding ``V=1`` to the invocation will show the details of how to
  1016. invoke QEMU for the test which is useful for debugging tests.
  1017. TCG test dependencies
  1018. ~~~~~~~~~~~~~~~~~~~~~
  1019. The TCG tests are deliberately very light on dependencies and are
  1020. either totally bare with minimal gcc lib support (for softmmu tests)
  1021. or just glibc (for linux-user tests). This is because getting a cross
  1022. compiler to work with additional libraries can be challenging.
  1023. Other TCG Tests
  1024. ---------------
  1025. There are a number of out-of-tree test suites that are used for more
  1026. extensive testing of processor features.
  1027. KVM Unit Tests
  1028. ~~~~~~~~~~~~~~
  1029. The KVM unit tests are designed to run as a Guest OS under KVM but
  1030. there is no reason why they can't exercise the TCG as well. It
  1031. provides a minimal OS kernel with hooks for enabling the MMU as well
  1032. as reporting test results via a special device::
  1033. https://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git
  1034. Linux Test Project
  1035. ~~~~~~~~~~~~~~~~~~
  1036. The LTP is focused on exercising the syscall interface of a Linux
  1037. kernel. It checks that syscalls behave as documented and strives to
  1038. exercise as many corner cases as possible. It is a useful test suite
  1039. to run to exercise QEMU's linux-user code::
  1040. https://linux-test-project.github.io/
  1041. GCC gcov support
  1042. ----------------
  1043. ``gcov`` is a GCC tool to analyze the testing coverage by
  1044. instrumenting the tested code. To use it, configure QEMU with
  1045. ``--enable-gcov`` option and build. Then run the tests as usual.
  1046. If you want to gather coverage information on a single test the ``make
  1047. clean-gcda`` target can be used to delete any existing coverage
  1048. information before running a single test.
  1049. You can generate a HTML coverage report by executing ``make
  1050. coverage-html`` which will create
  1051. ``meson-logs/coveragereport/index.html``.
  1052. Further analysis can be conducted by running the ``gcov`` command
  1053. directly on the various .gcda output files. Please read the ``gcov``
  1054. documentation for more information.