TestingLibcxx.rst 8.0 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268
  1. ==============
  2. Testing libc++
  3. ==============
  4. .. contents::
  5. :local:
  6. Getting Started
  7. ===============
  8. libc++ uses LIT to configure and run its tests.
  9. The primary way to run the libc++ tests is by using `make check-libcxx`.
  10. However since libc++ can be used in any number of possible
  11. configurations it is important to customize the way LIT builds and runs
  12. the tests. This guide provides information on how to use LIT directly to
  13. test libc++.
  14. Please see the `Lit Command Guide`_ for more information about LIT.
  15. .. _LIT Command Guide: http://llvm.org/docs/CommandGuide/lit.html
  16. Setting up the Environment
  17. --------------------------
  18. After building libc++ you must setup your environment to test libc++ using
  19. LIT.
  20. #. Create a shortcut to the actual lit executable so that you can invoke it
  21. easily from the command line.
  22. .. code-block:: bash
  23. $ alias lit='python path/to/llvm/utils/lit/lit.py'
  24. #. Tell LIT where to find your build configuration.
  25. .. code-block:: bash
  26. $ export LIBCXX_SITE_CONFIG=path/to/build-libcxx/test/lit.site.cfg
  27. Example Usage
  28. -------------
  29. Once you have your environment set up and you have built libc++ you can run
  30. parts of the libc++ test suite by simply running `lit` on a specified test or
  31. directory. For example:
  32. .. code-block:: bash
  33. $ cd path/to/src/libcxx
  34. $ lit -sv test/std/re # Run all of the std::regex tests
  35. $ lit -sv test/std/depr/depr.c.headers/stdlib_h.pass.cpp # Run a single test
  36. $ lit -sv test/std/atomics test/std/threads # Test std::thread and std::atomic
  37. Sometimes you'll want to change the way LIT is running the tests. Custom options
  38. can be specified using the `--param=<name>=<val>` flag. The most common option
  39. you'll want to change is the standard dialect (ie -std=c++XX). By default the
  40. test suite will select the newest C++ dialect supported by the compiler and use
  41. that. However if you want to manually specify the option like so:
  42. .. code-block:: bash
  43. $ lit -sv test/std/containers # Run the tests with the newest -std
  44. $ lit -sv --param=std=c++03 test/std/containers # Run the tests in C++03
  45. Occasionally you'll want to add extra compile or link flags when testing.
  46. You can do this as follows:
  47. .. code-block:: bash
  48. $ lit -sv --param=compile_flags='-Wcustom-warning'
  49. $ lit -sv --param=link_flags='-L/custom/library/path'
  50. Some other common examples include:
  51. .. code-block:: bash
  52. # Specify a custom compiler.
  53. $ lit -sv --param=cxx_under_test=/opt/bin/g++ test/std
  54. # Enable warnings in the test suite
  55. $ lit -sv --param=enable_warnings=true test/std
  56. # Use UBSAN when running the tests.
  57. $ lit -sv --param=use_sanitizer=Undefined
  58. LIT Options
  59. ===========
  60. :program:`lit` [*options*...] [*filenames*...]
  61. Command Line Options
  62. --------------------
  63. To use these options you pass them on the LIT command line as --param NAME or
  64. --param NAME=VALUE. Some options have default values specified during CMake's
  65. configuration. Passing the option on the command line will override the default.
  66. .. program:: lit
  67. .. option:: cxx_under_test=<path/to/compiler>
  68. Specify the compiler used to build the tests.
  69. .. option:: cxx_stdlib_under_test=<stdlib name>
  70. **Values**: libc++, libstdc++
  71. Specify the C++ standard library being tested. Unless otherwise specified
  72. libc++ is used. This option is intended to allow running the libc++ test
  73. suite against other standard library implementations.
  74. .. option:: std=<standard version>
  75. **Values**: c++98, c++03, c++11, c++14, c++17, c++2a
  76. Change the standard version used when building the tests.
  77. .. option:: libcxx_site_config=<path/to/lit.site.cfg>
  78. Specify the site configuration to use when running the tests. This option
  79. overrides the environment variable LIBCXX_SITE_CONFIG.
  80. .. option:: cxx_headers=<path/to/headers>
  81. Specify the c++ standard library headers that are tested. By default the
  82. headers in the source tree are used.
  83. .. option:: cxx_library_root=<path/to/lib/>
  84. Specify the directory of the libc++ library to be tested. By default the
  85. library folder of the build directory is used. This option cannot be used
  86. when use_system_cxx_lib is provided.
  87. .. option:: cxx_runtime_root=<path/to/lib/>
  88. Specify the directory of the libc++ library to use at runtime. This directory
  89. is not added to the linkers search path. This can be used to compile tests
  90. against one version of libc++ and run them using another. The default value
  91. for this option is `cxx_library_root`.
  92. .. option:: use_system_cxx_lib=<bool>
  93. **Default**: False
  94. Enable or disable testing against the installed version of libc++ library.
  95. Note: This does not use the installed headers.
  96. .. option:: use_lit_shell=<bool>
  97. Enable or disable the use of LIT's internal shell in ShTests. If the
  98. environment variable LIT_USE_INTERNAL_SHELL is present then that is used as
  99. the default value. Otherwise the default value is True on Windows and False
  100. on every other platform.
  101. .. option:: compile_flags="<list-of-args>"
  102. Specify additional compile flags as a space delimited string.
  103. Note: This options should not be used to change the standard version used.
  104. .. option:: link_flags="<list-of-args>"
  105. Specify additional link flags as a space delimited string.
  106. .. option:: debug_level=<level>
  107. **Values**: 0, 1
  108. Enable the use of debug mode. Level 0 enables assertions and level 1 enables
  109. assertions and debugging of iterator misuse.
  110. .. option:: use_sanitizer=<sanitizer name>
  111. **Values**: Memory, MemoryWithOrigins, Address, Undefined
  112. Run the tests using the given sanitizer. If LLVM_USE_SANITIZER was given when
  113. building libc++ then that sanitizer will be used by default.
  114. .. option:: color_diagnostics
  115. Enable the use of colorized compile diagnostics. If the color_diagnostics
  116. option is specified or the environment variable LIBCXX_COLOR_DIAGNOSTICS is
  117. present then color diagnostics will be enabled.
  118. .. option:: llvm_unwinder
  119. Enable the use of LLVM unwinder instead of libgcc.
  120. .. option:: builtins_library
  121. Path to the builtins library to use instead of libgcc.
  122. Environment Variables
  123. ---------------------
  124. .. envvar:: LIBCXX_SITE_CONFIG=<path/to/lit.site.cfg>
  125. Specify the site configuration to use when running the tests.
  126. Also see `libcxx_site_config`.
  127. .. envvar:: LIBCXX_COLOR_DIAGNOSTICS
  128. If ``LIBCXX_COLOR_DIAGNOSTICS`` is defined then the test suite will attempt
  129. to use color diagnostic outputs from the compiler.
  130. Also see `color_diagnostics`.
  131. Benchmarks
  132. ==========
  133. Libc++ contains benchmark tests separately from the test of the test suite.
  134. The benchmarks are written using the `Google Benchmark`_ library, a copy of which
  135. is stored in the libc++ repository.
  136. For more information about using the Google Benchmark library see the
  137. `official documentation <https://github.com/google/benchmark>`_.
  138. .. _`Google Benchmark`: https://github.com/google/benchmark
  139. Building Benchmarks
  140. -------------------
  141. The benchmark tests are not built by default. The benchmarks can be built using
  142. the ``cxx-benchmarks`` target.
  143. An example build would look like:
  144. .. code-block:: bash
  145. $ cd build
  146. $ cmake [options] <path to libcxx sources>
  147. $ make cxx-benchmarks
  148. This will build all of the benchmarks under ``<libcxx-src>/benchmarks`` to be
  149. built against the just-built libc++. The compiled tests are output into
  150. ``build/benchmarks``.
  151. The benchmarks can also be built against the platforms native standard library
  152. using the ``-DLIBCXX_BUILD_BENCHMARKS_NATIVE_STDLIB=ON`` CMake option. This
  153. is useful for comparing the performance of libc++ to other standard libraries.
  154. The compiled benchmarks are named ``<test>.libcxx.out`` if they test libc++ and
  155. ``<test>.native.out`` otherwise.
  156. Also See:
  157. * :ref:`Building Libc++ <build instructions>`
  158. * :ref:`CMake Options`
  159. Running Benchmarks
  160. ------------------
  161. The benchmarks must be run manually by the user. Currently there is no way
  162. to run them as part of the build.
  163. For example:
  164. .. code-block:: bash
  165. $ cd build/benchmarks
  166. $ make cxx-benchmarks
  167. $ ./algorithms.libcxx.out # Runs all the benchmarks
  168. $ ./algorithms.libcxx.out --benchmark_filter=BM_Sort.* # Only runs the sort benchmarks
  169. For more information about running benchmarks see `Google Benchmark`_.