qemu/docs/devel/migration/qpl-compression.rst

===============
QPL Compression
===============
The Intel Query Processing Library (Intel ``QPL``) is an open-source library to
provide compression and decompression features and it is based on deflate
compression algorithm (RFC 1951).

The ``QPL`` compression relies on Intel In-Memory Analytics Accelerator(``IAA``)
and Shared Virtual Memory(``SVM``) technology, they are new features supported
from Intel 4th Gen Intel Xeon Scalable processors, codenamed Sapphire Rapids
processor(``SPR``).

For more ``QPL`` introduction, please refer to `QPL Introduction
<https://intel.github.io/qpl/documentation/introduction_docs/introduction.html>`_

QPL Compression Framework
=========================

::

  +----------------+       +------------------+
  | MultiFD Thread |       |accel-config tool |
  +-------+--------+       +--------+---------+
          |                         |
          |                         |
          |compress/decompress      |
  +-------+--------+                | Setup IAA
  |  QPL library   |                | Resources
  +-------+---+----+                |
          |   |                     |
          |   +-------------+-------+
          |   Open IAA      |
          |   Devices +-----+-----+
          |           |idxd driver|
          |           +-----+-----+
          |                 |
          |                 |
          |           +-----+-----+
          +-----------+IAA Devices|
      Submit jobs     +-----------+
      via enqcmd


QPL Build And Installation
--------------------------

.. code-block:: shell

  $git clone --recursive https://github.com/intel/qpl.git qpl
  $mkdir qpl/build
  $cd qpl/build
  $cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr -DQPL_LIBRARY_TYPE=SHARED ..
  $sudo cmake --build . --target install

For more details about ``QPL`` installation, please refer to `QPL Installation
<https://intel.github.io/qpl/documentation/get_started_docs/installation.html>`_

IAA Device Management
---------------------

The number of ``IAA`` devices will vary depending on the Xeon product model.
On a ``SPR`` server, there can be a maximum of 8 ``IAA`` devices, with up to
4 devices per socket.

By default, all ``IAA`` devices are disabled and need to be configured and
enabled by users manually.

Check the number of devices through the following command

.. code-block:: shell

  #lspci -d 8086:0cfe
  6a:02.0 System peripheral: Intel Corporation Device 0cfe
  6f:02.0 System peripheral: Intel Corporation Device 0cfe
  74:02.0 System peripheral: Intel Corporation Device 0cfe
  79:02.0 System peripheral: Intel Corporation Device 0cfe
  e7:02.0 System peripheral: Intel Corporation Device 0cfe
  ec:02.0 System peripheral: Intel Corporation Device 0cfe
  f1:02.0 System peripheral: Intel Corporation Device 0cfe
  f6:02.0 System peripheral: Intel Corporation Device 0cfe

IAA Device Configuration And Enabling
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The ``accel-config`` tool is used to enable ``IAA`` devices and configure
``IAA`` hardware resources(work queues and engines). One ``IAA`` device
has 8 work queues and 8 processing engines, multiple engines can be assigned
to a work queue via ``group`` attribute.

For ``accel-config`` installation, please refer to `accel-config installation
<https://github.com/intel/idxd-config>`_

One example of configuring and enabling an ``IAA`` device.

.. code-block:: shell

  #accel-config config-engine iax1/engine1.0 -g 0
  #accel-config config-engine iax1/engine1.1 -g 0
  #accel-config config-engine iax1/engine1.2 -g 0
  #accel-config config-engine iax1/engine1.3 -g 0
  #accel-config config-engine iax1/engine1.4 -g 0
  #accel-config config-engine iax1/engine1.5 -g 0
  #accel-config config-engine iax1/engine1.6 -g 0
  #accel-config config-engine iax1/engine1.7 -g 0
  #accel-config config-wq iax1/wq1.0 -g 0 -s 128 -p 10 -b 1 -t 128 -m shared -y user -n app1 -d user
  #accel-config enable-device iax1
  #accel-config enable-wq iax1/wq1.0

.. note::
   IAX is an early name for IAA

- The ``IAA`` device index is 1, use ``ls -lh /sys/bus/dsa/devices/iax*``
  command to query the ``IAA`` device index.

- 8 engines and 1 work queue are configured in group 0, so all compression jobs
  submitted to this work queue can be processed by all engines at the same time.

- Set work queue attributes including the work mode, work queue size and so on.

- Enable the ``IAA1`` device and work queue 1.0

.. note::

  Set work queue mode to shared mode, since ``QPL`` library only supports
  shared mode

For more detailed configuration, please refer to `IAA Configuration Samples
<https://github.com/intel/idxd-config/tree/stable/Documentation/accfg>`_

IAA Unit Test
^^^^^^^^^^^^^

- Enabling ``IAA`` devices for Xeon platform, please refer to `IAA User Guide
  <https://www.intel.com/content/www/us/en/content-details/780887/intel-in-memory-analytics-accelerator-intel-iaa.html>`_

- ``IAA`` device driver is Intel Data Accelerator Driver (idxd), it is
  recommended that the minimum version of Linux kernel is 5.18.

- Add ``"intel_iommu=on,sm_on"`` parameter to kernel command line
  for ``SVM`` feature enabling.

Here is an easy way to verify ``IAA`` device driver and ``SVM`` with `iaa_test
<https://github.com/intel/idxd-config/tree/stable/test>`_

.. code-block:: shell

  #./test/iaa_test
   [ info] alloc wq 0 shared size 128 addr 0x7f26cebe5000 batch sz 0xfffffffe xfer sz 0x80000000
   [ info] test noop: tflags 0x1 num_desc 1
   [ info] preparing descriptor for noop
   [ info] Submitted all noop jobs
   [ info] verifying task result for 0x16f7e20
   [ info] test with op 0 passed


IAA Resources Allocation For Migration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

There is no ``IAA`` resource configuration parameters for migration and
``accel-config`` tool configuration cannot directly specify the ``IAA``
resources used for migration.

The multifd migration with ``QPL`` compression method  will use all work
queues that are enabled and shared mode.

.. note::

  Accessing IAA resources requires ``sudo`` command or ``root`` privileges
  by default. Administrators can modify the IAA device node ownership
  so that QEMU can use IAA with specified user permissions.

  For example

  #chown -R qemu /dev/iax

Shared Virtual Memory(SVM) Introduction
=======================================

An ability for an accelerator I/O device to operate in the same virtual
memory space of applications on host processors. It also implies the
ability to operate from pageable memory, avoiding functional requirements
to pin memory for DMA operations.

When using ``SVM`` technology, users do not need to reserve memory for the
``IAA`` device and perform pin memory operation. The ``IAA`` device can
directly access data using the virtual address of the process.

For more ``SVM`` technology, please refer to
`Shared Virtual Addressing (SVA) with ENQCMD
<https://docs.kernel.org/next/x86/sva.html>`_


How To Use QPL Compression In Migration
=======================================

1 - Installation of ``QPL`` library and ``accel-config`` library if using IAA

2 - Configure and enable ``IAA`` devices and work queues via ``accel-config``

3 - Build ``QEMU`` with ``--enable-qpl`` parameter

  E.g. configure --target-list=x86_64-softmmu --enable-kvm ``--enable-qpl``

4 - Enable ``QPL`` compression during migration

  Set ``migrate_set_parameter multifd-compression qpl`` when migrating, the
  ``QPL`` compression does not support configuring the compression level, it
  only supports one compression level.

The Difference Between QPL And ZLIB
===================================

Although both ``QPL`` and ``ZLIB`` are based on the deflate compression
algorithm, and ``QPL`` can support the header and tail of ``ZLIB``, ``QPL``
is still not fully compatible with the ``ZLIB`` compression in the migration.

``QPL`` only supports 4K history buffer, and ``ZLIB`` is 32K by default.
``ZLIB`` compresses data that ``QPL`` may not decompress correctly and
vice versa.

``QPL`` does not support the ``Z_SYNC_FLUSH`` operation in ``ZLIB`` streaming
compression, current ``ZLIB`` implementation uses ``Z_SYNC_FLUSH``, so each
``multifd`` thread has a ``ZLIB`` streaming context, and all page compression
and decompression are based on this stream. ``QPL`` cannot decompress such data
and vice versa.

The introduction for ``Z_SYNC_FLUSH``, please refer to `Zlib Manual
<https://www.zlib.net/manual.html>`_

The Best Practices
==================
When user enables the IAA device for ``QPL`` compression, it is recommended
to add ``-mem-prealloc`` parameter to the destination boot parameters. This
parameter can avoid the occurrence of I/O page fault and reduce the overhead
of IAA compression and decompression.

The example of booting with ``-mem-prealloc`` parameter

.. code-block:: shell

   $qemu-system-x86_64 --enable-kvm -cpu host --mem-prealloc ...


An example about I/O page fault measurement of destination without
``-mem-prealloc``, the ``svm_prq`` indicates the number of I/O page fault
occurrences and processing time.

.. code-block:: shell

  #echo 1 > /sys/kernel/debug/iommu/intel/dmar_perf_latency
  #echo 2 > /sys/kernel/debug/iommu/intel/dmar_perf_latency
  #echo 3 > /sys/kernel/debug/iommu/intel/dmar_perf_latency
  #echo 4 > /sys/kernel/debug/iommu/intel/dmar_perf_latency
  #cat /sys/kernel/debug/iommu/intel/dmar_perf_latency
  IOMMU: dmar18 Register Base Address: c87fc000
                  <0.1us   0.1us-1us    1us-10us  10us-100us   100us-1ms    1ms-10ms      >=10ms     min(us)     max(us) average(us)
   inv_iotlb           0         286         123           0           0           0           0           0           1           0
  inv_devtlb           0         276         133           0           0           0           0           0           2           0
     inv_iec           0           0           0           0           0           0           0           0           0           0
     svm_prq           0           0       25206         364         395           0           0           1         556           9