TestingLibcxx.rst 8.1 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266
  1. ==============
  2. Testing libc++
  3. ==============
  4. .. contents::
  5. :local:
  6. Getting Started
  7. ===============
  8. libc++ uses LIT to configure and run its tests. The primary way to run the
  9. libc++ tests is by using make check-libcxx. However since libc++ can be used
  10. in any number of possible configurations it is important to customize the way
  11. LIT builds and runs the tests. This guide provides information on how to use
  12. LIT directly to test libc++.
  13. Please see the `Lit Command Guide`_ for more information about LIT.
  14. .. _LIT Command Guide: http://llvm.org/docs/CommandGuide/lit.html
  15. Setting up the Environment
  16. --------------------------
  17. After building libc++ you must setup your environment to test libc++ using
  18. LIT.
  19. #. Create a shortcut to the actual lit executable so that you can invoke it
  20. easily from the command line.
  21. .. code-block:: bash
  22. $ alias lit='python path/to/llvm/utils/lit/lit.py'
  23. #. Tell LIT where to find your build configuration.
  24. .. code-block:: bash
  25. $ export LIBCXX_SITE_CONFIG=path/to/build-libcxx/test/lit.site.cfg
  26. Example Usage
  27. -------------
  28. Once you have your environment set up and you have built libc++ you can run
  29. parts of the libc++ test suite by simply running `lit` on a specified test or
  30. directory. For example:
  31. .. code-block:: bash
  32. $ cd path/to/src/libcxx
  33. $ lit -sv test/std/re # Run all of the std::regex tests
  34. $ lit -sv test/std/depr/depr.c.headers/stdlib_h.pass.cpp # Run a single test
  35. $ lit -sv test/std/atomics test/std/threads # Test std::thread and std::atomic
  36. Sometimes you'll want to change the way LIT is running the tests. Custom options
  37. can be specified using the `--param=<name>=<val>` flag. The most common option
  38. you'll want to change is the standard dialect (ie -std=c++XX). By default the
  39. test suite will select the newest C++ dialect supported by the compiler and use
  40. that. However if you want to manually specify the option like so:
  41. .. code-block:: bash
  42. $ lit -sv test/std/containers # Run the tests with the newest -std
  43. $ lit -sv --param=std=c++03 test/std/containers # Run the tests in C++03
  44. Occasionally you'll want to add extra compile or link flags when testing.
  45. You can do this as follows:
  46. .. code-block:: bash
  47. $ lit -sv --param=compile_flags='-Wcustom-warning'
  48. $ lit -sv --param=link_flags='-L/custom/library/path'
  49. Some other common examples include:
  50. .. code-block:: bash
  51. # Specify a custom compiler.
  52. $ lit -sv --param=cxx_under_test=/opt/bin/g++ test/std
  53. # Enable warnings in the test suite
  54. $ lit -sv --param=enable_warnings=true test/std
  55. # Use UBSAN when running the tests.
  56. $ lit -sv --param=use_sanitizer=Undefined
  57. LIT Options
  58. ===========
  59. :program:`lit` [*options*...] [*filenames*...]
  60. Command Line Options
  61. --------------------
  62. To use these options you pass them on the LIT command line as --param NAME or
  63. --param NAME=VALUE. Some options have default values specified during CMake's
  64. configuration. Passing the option on the command line will override the default.
  65. .. program:: lit
  66. .. option:: cxx_under_test=<path/to/compiler>
  67. Specify the compiler used to build the tests.
  68. .. option:: cxx_stdlib_under_test=<stdlib name>
  69. **Values**: libc++, libstdc++
  70. Specify the C++ standard library being tested. Unless otherwise specified
  71. libc++ is used. This option is intended to allow running the libc++ test
  72. suite against other standard library implementations.
  73. .. option:: std=<standard version>
  74. **Values**: c++98, c++03, c++11, c++14, c++1z
  75. Change the standard version used when building the tests.
  76. .. option:: libcxx_site_config=<path/to/lit.site.cfg>
  77. Specify the site configuration to use when running the tests. This option
  78. overrides the environment variable LIBCXX_SITE_CONFIG.
  79. .. option:: cxx_headers=<path/to/headers>
  80. Specify the c++ standard library headers that are tested. By default the
  81. headers in the source tree are used.
  82. .. option:: cxx_library_root=<path/to/lib/>
  83. Specify the directory of the libc++ library to be tested. By default the
  84. library folder of the build directory is used. This option cannot be used
  85. when use_system_cxx_lib is provided.
  86. .. option:: cxx_runtime_root=<path/to/lib/>
  87. Specify the directory of the libc++ library to use at runtime. This directory
  88. is not added to the linkers search path. This can be used to compile tests
  89. against one version of libc++ and run them using another. The default value
  90. for this option is `cxx_library_root`. This option cannot be used
  91. when use_system_cxx_lib is provided.
  92. .. option:: use_system_cxx_lib=<bool>
  93. **Default**: False
  94. Enable or disable testing against the installed version of libc++ library.
  95. Note: This does not use the installed headers.
  96. .. option:: use_lit_shell=<bool>
  97. Enable or disable the use of LIT's internal shell in ShTests. If the
  98. environment variable LIT_USE_INTERNAL_SHELL is present then that is used as
  99. the default value. Otherwise the default value is True on Windows and False
  100. on every other platform.
  101. .. option:: no_default_flags=<bool>
  102. **Default**: False
  103. Disable all default compile and link flags from being added. When this
  104. option is used only flags specified using the compile_flags and link_flags
  105. will be used.
  106. .. option:: compile_flags="<list-of-args>"
  107. Specify additional compile flags as a space delimited string.
  108. Note: This options should not be used to change the standard version used.
  109. .. option:: link_flags="<list-of-args>"
  110. Specify additional link flags as a space delimited string.
  111. .. option:: debug_level=<level>
  112. **Values**: 0, 1
  113. Enable the use of debug mode. Level 0 enables assertions and level 1 enables
  114. assertions and debugging of iterator misuse.
  115. .. option:: use_sanitizer=<sanitizer name>
  116. **Values**: Memory, MemoryWithOrigins, Address, Undefined
  117. Run the tests using the given sanitizer. If LLVM_USE_SANITIZER was given when
  118. building libc++ then that sanitizer will be used by default.
  119. .. option:: color_diagnostics
  120. Enable the use of colorized compile diagnostics. If the color_diagnostics
  121. option is specified or the environment variable LIBCXX_COLOR_DIAGNOSTICS is
  122. present then color diagnostics will be enabled.
  123. Environment Variables
  124. ---------------------
  125. .. envvar:: LIBCXX_SITE_CONFIG=<path/to/lit.site.cfg>
  126. Specify the site configuration to use when running the tests.
  127. Also see `libcxx_site_config`.
  128. .. envvar:: LIBCXX_COLOR_DIAGNOSTICS
  129. If ``LIBCXX_COLOR_DIAGNOSTICS`` is defined then the test suite will attempt
  130. to use color diagnostic outputs from the compiler.
  131. Also see `color_diagnostics`.
  132. Benchmarks
  133. ==========
  134. Libc++ contains benchmark tests separately from the test of the test suite.
  135. The benchmarks are written using the `Google Benchmark`_ library, a copy of which
  136. is stored in the libc++ repository.
  137. For more information about using the Google Benchmark library see the
  138. `official documentation <https://github.com/google/benchmark>`_.
  139. .. _`Google Benchmark`: https://github.com/google/benchmark
  140. Building Benchmarks
  141. -------------------
  142. The benchmark tests are not built by default. The benchmarks can be built using
  143. the ``cxx-benchmarks`` target.
  144. An example build would look like:
  145. .. code-block:: bash
  146. $ cd build
  147. $ cmake [options] <path to libcxx sources>
  148. $ make cxx-benchmarks
  149. This will build all of the benchmarks under ``<libcxx-src>/benchmarks`` to be
  150. built against the just-built libc++. The compiled tests are output into
  151. ``build/benchmarks``.
  152. The benchmarks can also be built against the platforms native standard library
  153. using the ``-DLIBCXX_BUILD_BENCHMARKS_NATIVE_STDLIB=ON`` CMake option. This
  154. is useful for comparing the performance of libc++ to other standard libraries.
  155. The compiled benchmarks are named ``<test>.libcxx.out`` if they test libc++ and
  156. ``<test>.native.out`` otherwise.
  157. Also See:
  158. * :ref:`Building Libc++ <build instructions>`
  159. * :ref:`CMake Options`
  160. Running Benchmarks
  161. ------------------
  162. The benchmarks must be run manually by the user. Currently there is no way
  163. to run them as part of the build.
  164. For example:
  165. .. code-block:: bash
  166. $ cd build/benchmarks
  167. $ make cxx-benchmarks
  168. $ ./algorithms.libcxx.out # Runs all the benchmarks
  169. $ ./algorithms.libcxx.out --benchmark_filter=BM_Sort.* # Only runs the sort benchmarks
  170. For more information about running benchmarks see `Google Benchmark`_.