hyperv.txt 9.3 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214
  1. Hyper-V Enlightenments
  2. ======================
  3. 1. Description
  4. ===============
  5. In some cases when implementing a hardware interface in software is slow, KVM
  6. implements its own paravirtualized interfaces. This works well for Linux as
  7. guest support for such features is added simultaneously with the feature itself.
  8. It may, however, be hard-to-impossible to add support for these interfaces to
  9. proprietary OSes, namely, Microsoft Windows.
  10. KVM on x86 implements Hyper-V Enlightenments for Windows guests. These features
  11. make Windows and Hyper-V guests think they're running on top of a Hyper-V
  12. compatible hypervisor and use Hyper-V specific features.
  13. 2. Setup
  14. =========
  15. No Hyper-V enlightenments are enabled by default by either KVM or QEMU. In
  16. QEMU, individual enlightenments can be enabled through CPU flags, e.g:
  17. qemu-system-x86_64 --enable-kvm --cpu host,hv_relaxed,hv_vpindex,hv_time, ...
  18. Sometimes there are dependencies between enlightenments, QEMU is supposed to
  19. check that the supplied configuration is sane.
  20. When any set of the Hyper-V enlightenments is enabled, QEMU changes hypervisor
  21. identification (CPUID 0x40000000..0x4000000A) to Hyper-V. KVM identification
  22. and features are kept in leaves 0x40000100..0x40000101.
  23. 3. Existing enlightenments
  24. ===========================
  25. 3.1. hv-relaxed
  26. ================
  27. This feature tells guest OS to disable watchdog timeouts as it is running on a
  28. hypervisor. It is known that some Windows versions will do this even when they
  29. see 'hypervisor' CPU flag.
  30. 3.2. hv-vapic
  31. ==============
  32. Provides so-called VP Assist page MSR to guest allowing it to work with APIC
  33. more efficiently. In particular, this enlightenment allows paravirtualized
  34. (exit-less) EOI processing.
  35. 3.3. hv-spinlocks=xxx
  36. ======================
  37. Enables paravirtualized spinlocks. The parameter indicates how many times
  38. spinlock acquisition should be attempted before indicating the situation to the
  39. hypervisor. A special value 0xffffffff indicates "never to retry".
  40. 3.4. hv-vpindex
  41. ================
  42. Provides HV_X64_MSR_VP_INDEX (0x40000002) MSR to the guest which has Virtual
  43. processor index information. This enlightenment makes sense in conjunction with
  44. hv-synic, hv-stimer and other enlightenments which require the guest to know its
  45. Virtual Processor indices (e.g. when VP index needs to be passed in a
  46. hypercall).
  47. 3.5. hv-runtime
  48. ================
  49. Provides HV_X64_MSR_VP_RUNTIME (0x40000010) MSR to the guest. The MSR keeps the
  50. virtual processor run time in 100ns units. This gives guest operating system an
  51. idea of how much time was 'stolen' from it (when the virtual CPU was preempted
  52. to perform some other work).
  53. 3.6. hv-crash
  54. ==============
  55. Provides HV_X64_MSR_CRASH_P0..HV_X64_MSR_CRASH_P5 (0x40000100..0x40000105) and
  56. HV_X64_MSR_CRASH_CTL (0x40000105) MSRs to the guest. These MSRs are written to
  57. by the guest when it crashes, HV_X64_MSR_CRASH_P0..HV_X64_MSR_CRASH_P5 MSRs
  58. contain additional crash information. This information is outputted in QEMU log
  59. and through QAPI.
  60. Note: unlike under genuine Hyper-V, write to HV_X64_MSR_CRASH_CTL causes guest
  61. to shutdown. This effectively blocks crash dump generation by Windows.
  62. 3.7. hv-time
  63. =============
  64. Enables two Hyper-V-specific clocksources available to the guest: MSR-based
  65. Hyper-V clocksource (HV_X64_MSR_TIME_REF_COUNT, 0x40000020) and Reference TSC
  66. page (enabled via MSR HV_X64_MSR_REFERENCE_TSC, 0x40000021). Both clocksources
  67. are per-guest, Reference TSC page clocksource allows for exit-less time stamp
  68. readings. Using this enlightenment leads to significant speedup of all timestamp
  69. related operations.
  70. 3.8. hv-synic
  71. ==============
  72. Enables Hyper-V Synthetic interrupt controller - an extension of a local APIC.
  73. When enabled, this enlightenment provides additional communication facilities
  74. to the guest: SynIC messages and Events. This is a pre-requisite for
  75. implementing VMBus devices (not yet in QEMU). Additionally, this enlightenment
  76. is needed to enable Hyper-V synthetic timers. SynIC is controlled through MSRs
  77. HV_X64_MSR_SCONTROL..HV_X64_MSR_EOM (0x40000080..0x40000084) and
  78. HV_X64_MSR_SINT0..HV_X64_MSR_SINT15 (0x40000090..0x4000009F)
  79. Requires: hv-vpindex
  80. 3.9. hv-stimer
  81. ===============
  82. Enables Hyper-V synthetic timers. There are four synthetic timers per virtual
  83. CPU controlled through HV_X64_MSR_STIMER0_CONFIG..HV_X64_MSR_STIMER3_COUNT
  84. (0x400000B0..0x400000B7) MSRs. These timers can work either in single-shot or
  85. periodic mode. It is known that certain Windows versions revert to using HPET
  86. (or even RTC when HPET is unavailable) extensively when this enlightenment is
  87. not provided; this can lead to significant CPU consumption, even when virtual
  88. CPU is idle.
  89. Requires: hv-vpindex, hv-synic, hv-time
  90. 3.10. hv-tlbflush
  91. ==================
  92. Enables paravirtualized TLB shoot-down mechanism. On x86 architecture, remote
  93. TLB flush procedure requires sending IPIs and waiting for other CPUs to perform
  94. local TLB flush. In virtualized environment some virtual CPUs may not even be
  95. scheduled at the time of the call and may not require flushing (or, flushing
  96. may be postponed until the virtual CPU is scheduled). hv-tlbflush enlightenment
  97. implements TLB shoot-down through hypervisor enabling the optimization.
  98. Requires: hv-vpindex
  99. 3.11. hv-ipi
  100. =============
  101. Enables paravirtualized IPI send mechanism. HvCallSendSyntheticClusterIpi
  102. hypercall may target more than 64 virtual CPUs simultaneously, doing the same
  103. through APIC requires more than one access (and thus exit to the hypervisor).
  104. Requires: hv-vpindex
  105. 3.12. hv-vendor-id=xxx
  106. =======================
  107. This changes Hyper-V identification in CPUID 0x40000000.EBX-EDX from the default
  108. "Microsoft Hv". The parameter should be no longer than 12 characters. According
  109. to the specification, guests shouldn't use this information and it is unknown
  110. if there is a Windows version which acts differently.
  111. Note: hv-vendor-id is not an enlightenment and thus doesn't enable Hyper-V
  112. identification when specified without some other enlightenment.
  113. 3.13. hv-reset
  114. ===============
  115. Provides HV_X64_MSR_RESET (0x40000003) MSR to the guest allowing it to reset
  116. itself by writing to it. Even when this MSR is enabled, it is not a recommended
  117. way for Windows to perform system reboot and thus it may not be used.
  118. 3.14. hv-frequencies
  119. ============================================
  120. Provides HV_X64_MSR_TSC_FREQUENCY (0x40000022) and HV_X64_MSR_APIC_FREQUENCY
  121. (0x40000023) allowing the guest to get its TSC/APIC frequencies without doing
  122. measurements.
  123. 3.15 hv-reenlightenment
  124. ========================
  125. The enlightenment is nested specific, it targets Hyper-V on KVM guests. When
  126. enabled, it provides HV_X64_MSR_REENLIGHTENMENT_CONTROL (0x40000106),
  127. HV_X64_MSR_TSC_EMULATION_CONTROL (0x40000107)and HV_X64_MSR_TSC_EMULATION_STATUS
  128. (0x40000108) MSRs allowing the guest to get notified when TSC frequency changes
  129. (only happens on migration) and keep using old frequency (through emulation in
  130. the hypervisor) until it is ready to switch to the new one. This, in conjunction
  131. with hv-frequencies, allows Hyper-V on KVM to pass stable clocksource (Reference
  132. TSC page) to its own guests.
  133. Recommended: hv-frequencies
  134. 3.16. hv-evmcs
  135. ===============
  136. The enlightenment is nested specific, it targets Hyper-V on KVM guests. When
  137. enabled, it provides Enlightened VMCS feature to the guest. The feature
  138. implements paravirtualized protocol between L0 (KVM) and L1 (Hyper-V)
  139. hypervisors making L2 exits to the hypervisor faster. The feature is Intel-only.
  140. Note: some virtualization features (e.g. Posted Interrupts) are disabled when
  141. hv-evmcs is enabled. It may make sense to measure your nested workload with and
  142. without the feature to find out if enabling it is beneficial.
  143. Requires: hv-vapic
  144. 3.17. hv-stimer-direct
  145. =======================
  146. Hyper-V specification allows synthetic timer operation in two modes: "classic",
  147. when expiration event is delivered as SynIC message and "direct", when the event
  148. is delivered via normal interrupt. It is known that nested Hyper-V can only
  149. use synthetic timers in direct mode and thus 'hv-stimer-direct' needs to be
  150. enabled.
  151. Requires: hv-vpindex, hv-synic, hv-time, hv-stimer
  152. 3.17. hv-no-nonarch-coresharing=on/off/auto
  153. ===========================================
  154. This enlightenment tells guest OS that virtual processors will never share a
  155. physical core unless they are reported as sibling SMT threads. This information
  156. is required by Windows and Hyper-V guests to properly mitigate SMT related CPU
  157. vulnerabilities.
  158. When the option is set to 'auto' QEMU will enable the feature only when KVM
  159. reports that non-architectural coresharing is impossible, this means that
  160. hyper-threading is not supported or completely disabled on the host. This
  161. setting also prevents migration as SMT settings on the destination may differ.
  162. When the option is set to 'on' QEMU will always enable the feature, regardless
  163. of host setup. To keep guests secure, this can only be used in conjunction with
  164. exposing correct vCPU topology and vCPU pinning.
  165. 4. Development features
  166. ========================
  167. In some cases (e.g. during development) it may make sense to use QEMU in
  168. 'pass-through' mode and give Windows guests all enlightenments currently
  169. supported by KVM. This pass-through mode is enabled by "hv-passthrough" CPU
  170. flag.
  171. Note: enabling this flag effectively prevents migration as supported features
  172. may differ between target and destination.
  173. 4. Useful links
  174. ================
  175. Hyper-V Top Level Functional specification and other information:
  176. https://github.com/MicrosoftDocs/Virtualization-Documentation