<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div><base href="https://www.kernel.org/doc/html/latest/admin-guide/l1tf.html"><style id="print">
@media print {
body {
margin: 2mm 9mm;
}
.original-url {
display: none;
}
#article .float.left {
float: left !important;
}
#article .float.right {
float: right !important;
}
#article .float {
margin-top: 0 !important;
margin-bottom: 0 !important;
}
}
</style><title>L1TF - L1 Terminal Fault — The Linux Kernel documentation</title><div class="original-url"><br></div><div class="original-url">For test systems, we probably want to mark the guests as trusted to prevent further slowdowns.</div><div class="original-url"><br></div><div class="original-url">Paul</div><div class="original-url"><br></div><div class="original-url"><br><a href="https://www.kernel.org/doc/html/latest/admin-guide/l1tf.html">https://www.kernel.org/doc/html/latest/admin-guide/l1tf.html</a><br><br></div><div id="article" role="article" style="text-rendering: optimizeLegibility; font-family: -apple-system-font; font-size: 1.2em; line-height: 1.5em; margin: 0px; padding: 0px;" class="system exported">
<!-- This node will contain a number of div.page. -->
<div class="page" style="text-align: start; word-wrap: break-word; max-width: 100%;"><h1 class="title" style="font-weight: bold; font-size: 1.95552em; line-height: 1.2141em; margin-top: 0px; margin-bottom: 0.5em; text-align: start; -webkit-hyphens: manual; display: block; max-width: 100%;">L1TF - L1 Terminal Fault</h1>
<p style="max-width: 100%;">L1 Terminal Fault is a hardware vulnerability which allows unprivileged
speculative access to data which is available in the Level 1 Data Cache
when the page table entry controlling the virtual address, which is used
for the access, has the Present bit cleared or other reserved bits set.</p>
<div style="max-width: 100%;">
<h2 style="font-weight: bold; font-size: 1.43em; max-width: 100%;">Affected processors</h2>
<p style="max-width: 100%;">This vulnerability affects a wide range of Intel processors. The
vulnerability is not present on:</p>
<blockquote style="max-width: 100%; color: rgba(0, 0, 0, 0.65098); margin-left: 2px; margin-right: 6px; padding-left: 16px; border-left-width: 3px; border-left-style: solid; border-left-color: rgba(0, 0, 0, 0.0980392);">
<div style="max-width: 100%;"><ul style="max-width: 100%;">
<li style="max-width: 100%;">Processors from AMD, Centaur and other non Intel vendors</li>
<li style="max-width: 100%;">Older processor models, where the CPU family is < 6</li>
<li style="max-width: 100%;">A range of Intel ATOM processors (Cedarview, Cloverview, Lincroft,
Penwell, Pineview, Silvermont, Airmont, Merrifield)</li>
<li style="max-width: 100%;">The Intel XEON PHI family</li>
<li style="max-width: 100%;">Intel processors which have the ARCH_CAP_RDCL_NO bit set in the
IA32_ARCH_CAPABILITIES MSR. If the bit is set the CPU is not affected
by the Meltdown vulnerability either. These CPUs should become
available by end of 2018.</li>
</ul>
</div></blockquote>
<p style="max-width: 100%;">Whether a processor is affected or not can be read out from the L1TF
vulnerability file in sysfs. See <span class="converted-anchor" style="max-width: 100%;"><span style="max-width: 100%;">L1TF system information</span></span>.</p>
</div>
<div style="max-width: 100%;">
<h2 style="font-weight: bold; font-size: 1.43em; max-width: 100%;">Problem</h2>
<p style="max-width: 100%;">If an instruction accesses a virtual address for which the relevant page
table entry (PTE) has the Present bit cleared or other reserved bits set,
then speculative execution ignores the invalid PTE and loads the referenced
data if it is present in the Level 1 Data Cache, as if the page referenced
by the address bits in the PTE was still present and accessible.</p>
<p style="max-width: 100%;">While this is a purely speculative mechanism and the instruction will raise
a page fault when it is retired eventually, the pure act of loading the
data and making it available to other speculative instructions opens up the
opportunity for side channel attacks to unprivileged malicious code,
similar to the Meltdown attack.</p>
<p style="max-width: 100%;">While Meltdown breaks the user space to kernel space protection, L1TF
allows to attack any physical memory address in the system and the attack
works across all protection domains. It allows an attack of SGX and also
works from inside virtual machines because the speculation bypasses the
extended page table (EPT) protection mechanism.</p>
</div>
<div style="max-width: 100%;">
<h2 style="font-weight: bold; font-size: 1.43em; max-width: 100%;">Attack scenarios</h2>
<div style="max-width: 100%;">
<h3 style="font-weight: bold; font-size: 1.25em; max-width: 100%;">1. Malicious user space</h3>
<blockquote style="max-width: 100%; color: rgba(0, 0, 0, 0.65098); margin-left: 2px; margin-right: 6px; padding-left: 16px; border-left-width: 3px; border-left-style: solid; border-left-color: rgba(0, 0, 0, 0.0980392);">
<div style="max-width: 100%;"><p style="max-width: 100%;">Operating Systems store arbitrary information in the address bits of a
PTE which is marked non present. This allows a malicious user space
application to attack the physical memory to which these PTEs resolve.
In some cases user-space can maliciously influence the information
encoded in the address bits of the PTE, thus making attacks more
deterministic and more practical.</p>
<p style="max-width: 100%;">The Linux kernel contains a mitigation for this attack vector, PTE
inversion, which is permanently enabled and has no performance
impact. The kernel ensures that the address bits of PTEs, which are not
marked present, never point to cacheable physical memory space.</p>
<p style="max-width: 100%;">A system with an up to date kernel is protected against attacks from
malicious user space applications.</p>
</div></blockquote>
</div>
<div style="max-width: 100%;">
<h3 style="font-weight: bold; font-size: 1.25em; max-width: 100%;">2. Malicious guest in a virtual machine</h3>
<blockquote style="max-width: 100%; color: rgba(0, 0, 0, 0.65098); margin-left: 2px; margin-right: 6px; padding-left: 16px; border-left-width: 3px; border-left-style: solid; border-left-color: rgba(0, 0, 0, 0.0980392);">
<div style="max-width: 100%;"><p style="max-width: 100%;">The fact that L1TF breaks all domain protections allows malicious guest
OSes, which can control the PTEs directly, and malicious guest user
space applications, which run on an unprotected guest kernel lacking the
PTE inversion mitigation for L1TF, to attack physical host memory.</p>
<p style="max-width: 100%;">A special aspect of L1TF in the context of virtualization is symmetric
multi threading (SMT). The Intel implementation of SMT is called
HyperThreading. The fact that Hyperthreads on the affected processors
share the L1 Data Cache (L1D) is important for this. As the flaw allows
only to attack data which is present in L1D, a malicious guest running
on one Hyperthread can attack the data which is brought into the L1D by
the context which runs on the sibling Hyperthread of the same physical
core. This context can be host OS, host user space or a different guest.</p>
<p style="max-width: 100%;">If the processor does not support Extended Page Tables, the attack is
only possible, when the hypervisor does not sanitize the content of the
effective (shadow) page tables.</p>
<p style="max-width: 100%;">While solutions exist to mitigate these attack vectors fully, these
mitigations are not enabled by default in the Linux kernel because they
can affect performance significantly. The kernel provides several
mechanisms which can be utilized to address the problem depending on the
deployment scenario. The mitigations, their protection scope and impact
are described in the next sections.</p>
<p style="max-width: 100%;">The default mitigations and the rationale for choosing them are explained
at the end of this document. See <span class="converted-anchor" style="max-width: 100%;"><span style="max-width: 100%;">Default mitigations</span></span>.</p>
</div></blockquote>
</div>
</div>
<div style="max-width: 100%;">
<span style="max-width: 100%;"></span><h2 style="font-weight: bold; font-size: 1.43em; max-width: 100%;">L1TF system information</h2>
<p style="max-width: 100%;">The Linux kernel provides a sysfs interface to enumerate the current L1TF
status of the system: whether the system is vulnerable, and which
mitigations are active. The relevant sysfs file is:</p>
<p style="max-width: 100%;">/sys/devices/system/cpu/vulnerabilities/l1tf</p>
<p style="max-width: 100%;">The possible values in this file are:</p>
<blockquote class="simple" style="max-width: 100%; color: rgba(0, 0, 0, 0.65098); margin-left: 2px; margin-right: 6px; padding-left: 16px;">
<div style="max-width: 100%;"><div style="max-width: 100%;"><div class="scrollable" style="max-width: 100%; overflow-x: scroll; word-wrap: normal;"><table border="1" style="max-width: none; font-size: 0.9em; text-align: start; word-wrap: break-word; border-collapse: collapse;">
<colgroup style="max-width: 100%;">
<col width="47%" style="max-width: 100%;">
<col width="53%" style="max-width: 100%;">
</colgroup>
<tbody valign="top" style="max-width: 100%;">
<tr style="max-width: 100%;"><td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">‘Not affected’</td>
<td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">The processor is not vulnerable</td>
</tr>
<tr style="max-width: 100%;"><td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">‘Mitigation: PTE Inversion’</td>
<td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">The host protection is active</td>
</tr>
</tbody>
</table></div></div>
</div></blockquote>
<p style="max-width: 100%;">If KVM/VMX is enabled and the processor is vulnerable then the following
information is appended to the ‘Mitigation: PTE Inversion’ part:</p>
<blockquote class="simple" style="max-width: 100%; color: rgba(0, 0, 0, 0.65098); margin-left: 2px; margin-right: 6px; padding-left: 16px;">
<div style="max-width: 100%;"><ul style="max-width: 100%;">
<li style="max-width: 100%;"><p style="max-width: 100%;">SMT status:</p>
<div style="max-width: 100%;"><div class="scrollable" style="max-width: 100%; overflow-x: scroll; word-wrap: normal;"><table border="1" style="max-width: none; font-size: 0.9em; text-align: start; word-wrap: break-word; border-collapse: collapse;">
<colgroup style="max-width: 100%;">
<col width="57%" style="max-width: 100%;">
<col width="43%" style="max-width: 100%;">
</colgroup>
<tbody valign="top" style="max-width: 100%;">
<tr style="max-width: 100%;"><td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">‘VMX: SMT vulnerable’</td>
<td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">SMT is enabled</td>
</tr>
<tr style="max-width: 100%;"><td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">‘VMX: SMT disabled’</td>
<td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">SMT is disabled</td>
</tr>
</tbody>
</table></div></div>
</li>
<li style="max-width: 100%;"><p style="max-width: 100%;">L1D Flush mode:</p>
<div style="max-width: 100%;"><div class="scrollable" style="max-width: 100%; overflow-x: scroll; word-wrap: normal;"><table border="1" style="max-width: none; font-size: 0.9em; text-align: start; word-wrap: break-word; border-collapse: collapse;">
<colgroup style="max-width: 100%;">
<col width="47%" style="max-width: 100%;">
<col width="53%" style="max-width: 100%;">
</colgroup>
<tbody valign="top" style="max-width: 100%;">
<tr style="max-width: 100%;"><td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">‘L1D vulnerable’</td>
<td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">L1D flushing is disabled</td>
</tr>
<tr style="max-width: 100%;"><td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">‘L1D conditional cache flushes’</td>
<td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">L1D flush is conditionally enabled</td>
</tr>
<tr style="max-width: 100%;"><td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">‘L1D cache flushes’</td>
<td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">L1D flush is unconditionally enabled</td>
</tr>
</tbody>
</table></div></div>
</li>
</ul>
</div></blockquote>
<p style="max-width: 100%;">The resulting grade of protection is discussed in the following sections.</p>
</div>
<div style="max-width: 100%;">
<h2 style="font-weight: bold; font-size: 1.43em; max-width: 100%;">Host mitigation mechanism</h2>
<p style="max-width: 100%;">The kernel is unconditionally protected against L1TF attacks from malicious
user space running on the host.</p>
</div>
<div style="max-width: 100%;">
<h2 style="font-weight: bold; font-size: 1.43em; max-width: 100%;">Guest mitigation mechanisms</h2>
<div style="max-width: 100%;">
<span style="max-width: 100%;"></span><h3 style="font-weight: bold; font-size: 1.25em; max-width: 100%;">1. L1D flush on VMENTER</h3>
<blockquote style="max-width: 100%; color: rgba(0, 0, 0, 0.65098); margin-left: 2px; margin-right: 6px; padding-left: 16px; border-left-width: 3px; border-left-style: solid; border-left-color: rgba(0, 0, 0, 0.0980392);">
<div style="max-width: 100%;"><p style="max-width: 100%;">To make sure that a guest cannot attack data which is present in the L1D
the hypervisor flushes the L1D before entering the guest.</p>
<p style="max-width: 100%;">Flushing the L1D evicts not only the data which should not be accessed
by a potentially malicious guest, it also flushes the guest
data. Flushing the L1D has a performance impact as the processor has to
bring the flushed guest data back into the L1D. Depending on the
frequency of VMEXIT/VMENTER and the type of computations in the guest
performance degradation in the range of 1% to 50% has been observed. For
scenarios where guest VMEXIT/VMENTER are rare the performance impact is
minimal. Virtio and mechanisms like posted interrupts are designed to
confine the VMEXITs to a bare minimum, but specific configurations and
application scenarios might still suffer from a high VMEXIT rate.</p>
<dl style="max-width: 100%;">
<dt style="font-weight: bold; max-width: 100%;">The kernel provides two L1D flush modes:</dt>
<dd style="max-width: 100%;"><ul style="max-width: 100%;">
<li style="max-width: 100%;">conditional (‘cond’)</li>
<li style="max-width: 100%;">unconditional (‘always’)</li>
</ul>
</dd>
</dl>
<p style="max-width: 100%;">The conditional mode avoids L1D flushing after VMEXITs which execute
only audited code paths before the corresponding VMENTER. These code
paths have been verified that they cannot expose secrets or other
interesting data to an attacker, but they can leak information about the
address space layout of the hypervisor.</p>
<p style="max-width: 100%;">Unconditional mode flushes L1D on all VMENTER invocations and provides
maximum protection. It has a higher overhead than the conditional
mode. The overhead cannot be quantified correctly as it depends on the
workload scenario and the resulting number of VMEXITs.</p>
<p style="max-width: 100%;">The general recommendation is to enable L1D flush on VMENTER. The kernel
defaults to conditional mode on affected processors.</p>
<p style="max-width: 100%;"><strong style="max-width: 100%;">Note</strong>, that L1D flush does not prevent the SMT problem because the
sibling thread will also bring back its data into the L1D which makes it
attackable again.</p>
<p style="max-width: 100%;">L1D flush can be controlled by the administrator via the kernel command
line and sysfs control files. See <span class="converted-anchor" style="max-width: 100%;"><span style="max-width: 100%;">Mitigation control on the kernel command line</span></span>
and <span class="converted-anchor" style="max-width: 100%;"><span style="max-width: 100%;">Mitigation control for KVM - module parameter</span></span>.</p>
</div></blockquote>
</div>
<div style="max-width: 100%;">
<span style="max-width: 100%;"></span><h3 style="font-weight: bold; font-size: 1.25em; max-width: 100%;">2. Guest VCPU confinement to dedicated physical cores</h3>
<blockquote style="max-width: 100%; color: rgba(0, 0, 0, 0.65098); margin-left: 2px; margin-right: 6px; padding-left: 16px; border-left-width: 3px; border-left-style: solid; border-left-color: rgba(0, 0, 0, 0.0980392);">
<div style="max-width: 100%;"><p style="max-width: 100%;">To address the SMT problem, it is possible to make a guest or a group of
guests affine to one or more physical cores. The proper mechanism for
that is to utilize exclusive cpusets to ensure that no other guest or
host tasks can run on these cores.</p>
<p style="max-width: 100%;">If only a single guest or related guests run on sibling SMT threads on
the same physical core then they can only attack their own memory and
restricted parts of the host memory.</p>
<p style="max-width: 100%;">Host memory is attackable, when one of the sibling SMT threads runs in
host OS (hypervisor) context and the other in guest context. The amount
of valuable information from the host OS context depends on the context
which the host OS executes, i.e. interrupts, soft interrupts and kernel
threads. The amount of valuable data from these contexts cannot be
declared as non-interesting for an attacker without deep inspection of
the code.</p>
<p style="max-width: 100%;"><strong style="max-width: 100%;">Note</strong>, that assigning guests to a fixed set of physical cores affects
the ability of the scheduler to do load balancing and might have
negative effects on CPU utilization depending on the hosting
scenario. Disabling SMT might be a viable alternative for particular
scenarios.</p>
<p style="max-width: 100%;">For further information about confining guests to a single or to a group
of cores consult the cpusets documentation:</p>
<p style="max-width: 100%;"><a href="https://www.kernel.org/doc/Documentation/cgroup-v1/cpusets.txt" style="color: rgb(65, 110, 210); max-width: 100%; text-decoration: underline;">https://www.kernel.org/doc/Documentation/cgroup-v1/cpusets.txt</a></p>
</div></blockquote>
</div>
<div style="max-width: 100%;">
<span style="max-width: 100%;"></span><h3 style="font-weight: bold; font-size: 1.25em; max-width: 100%;">3. Interrupt affinity</h3>
<blockquote style="max-width: 100%; color: rgba(0, 0, 0, 0.65098); margin-left: 2px; margin-right: 6px; padding-left: 16px; border-left-width: 3px; border-left-style: solid; border-left-color: rgba(0, 0, 0, 0.0980392);">
<div style="max-width: 100%;"><p style="max-width: 100%;">Interrupts can be made affine to logical CPUs. This is not universally
true because there are types of interrupts which are truly per CPU
interrupts, e.g. the local timer interrupt. Aside of that multi queue
devices affine their interrupts to single CPUs or groups of CPUs per
queue without allowing the administrator to control the affinities.</p>
<p style="max-width: 100%;">Moving the interrupts, which can be affinity controlled, away from CPUs
which run untrusted guests, reduces the attack vector space.</p>
<p style="max-width: 100%;">Whether the interrupts with are affine to CPUs, which run untrusted
guests, provide interesting data for an attacker depends on the system
configuration and the scenarios which run on the system. While for some
of the interrupts it can be assumed that they won’t expose interesting
information beyond exposing hints about the host OS memory layout, there
is no way to make general assumptions.</p>
<p style="max-width: 100%;">Interrupt affinity can be controlled by the administrator via the
/proc/irq/$NR/smp_affinity[_list] files. Limited documentation is
available at:</p>
<p style="max-width: 100%;"><a href="https://www.kernel.org/doc/Documentation/IRQ-affinity.txt" style="color: rgb(65, 110, 210); max-width: 100%; text-decoration: underline;">https://www.kernel.org/doc/Documentation/IRQ-affinity.txt</a></p>
</div></blockquote>
</div>
<div style="max-width: 100%;">
<span style="max-width: 100%;"></span><h3 style="font-weight: bold; font-size: 1.25em; max-width: 100%;">4. SMT control</h3>
<blockquote class="simple" style="max-width: 100%; color: rgba(0, 0, 0, 0.65098); margin-left: 2px; margin-right: 6px; padding-left: 16px;">
<div style="max-width: 100%;"><p style="max-width: 100%;">To prevent the SMT issues of L1TF it might be necessary to disable SMT
completely. Disabling SMT can have a significant performance impact, but
the impact depends on the hosting scenario and the type of workloads.
The impact of disabling SMT needs also to be weighted against the impact
of other mitigation solutions like confining guests to dedicated cores.</p>
<p style="max-width: 100%;">The kernel provides a sysfs interface to retrieve the status of SMT and
to control it. It also provides a kernel command line interface to
control SMT.</p>
<p style="max-width: 100%;">The kernel command line interface consists of the following options:</p>
<blockquote class="simple" style="max-width: 100%; color: rgba(0, 0, 0, 0.65098); margin-left: 2px; margin-right: 6px; padding-left: 16px;">
<div style="max-width: 100%;"><div style="max-width: 100%;"><div class="scrollable" style="max-width: 100%; overflow-x: scroll; word-wrap: normal;"><table border="1" style="max-width: none; font-size: 0.9em; text-align: start; word-wrap: break-word; border-collapse: collapse;">
<colgroup style="max-width: 100%;">
<col width="16%" style="max-width: 100%;">
<col width="84%" style="max-width: 100%;">
</colgroup>
<tbody valign="top" style="max-width: 100%;">
<tr style="max-width: 100%;"><td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">nosmt</td>
<td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">Affects the bring up of the secondary CPUs during boot. The
kernel tries to bring all present CPUs online during the
boot process. “nosmt” makes sure that from each physical
core only one - the so called primary (hyper) thread is
activated. Due to a design flaw of Intel processors related
to Machine Check Exceptions the non primary siblings have
to be brought up at least partially and are then shut down
again. “nosmt” can be undone via the sysfs interface.</td>
</tr>
<tr style="max-width: 100%;"><td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">nosmt=force</td>
<td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">Has the same effect as “nosmt” but it does not allow to
undo the SMT disable via the sysfs interface.</td>
</tr>
</tbody>
</table></div></div>
</div></blockquote>
<p style="max-width: 100%;">The sysfs interface provides two files:</p>
<ul style="max-width: 100%;">
<li style="max-width: 100%;">/sys/devices/system/cpu/smt/control</li>
<li style="max-width: 100%;">/sys/devices/system/cpu/smt/active</li>
</ul>
<p style="max-width: 100%;">/sys/devices/system/cpu/smt/control:</p>
<blockquote class="simple" style="max-width: 100%; color: rgba(0, 0, 0, 0.65098); margin-left: 2px; margin-right: 6px; padding-left: 16px;">
<div style="max-width: 100%;"><p style="max-width: 100%;">This file allows to read out the SMT control state and provides the
ability to disable or (re)enable SMT. The possible states are:</p>
<blockquote class="simple" style="max-width: 100%; color: rgba(0, 0, 0, 0.65098); margin-left: 2px; margin-right: 6px; padding-left: 16px;">
<div style="max-width: 100%;"><div style="max-width: 100%;"><div class="scrollable" style="max-width: 100%; overflow-x: scroll; word-wrap: normal;"><table border="1" style="max-width: none; font-size: 0.9em; text-align: start; word-wrap: break-word; border-collapse: collapse;">
<colgroup style="max-width: 100%;">
<col width="22%" style="max-width: 100%;">
<col width="78%" style="max-width: 100%;">
</colgroup>
<tbody valign="top" style="max-width: 100%;">
<tr style="max-width: 100%;"><td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">on</td>
<td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">SMT is supported by the CPU and enabled. All
logical CPUs can be onlined and offlined without
restrictions.</td>
</tr>
<tr style="max-width: 100%;"><td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">off</td>
<td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">SMT is supported by the CPU and disabled. Only
the so called primary SMT threads can be onlined
and offlined without restrictions. An attempt to
online a non-primary sibling is rejected</td>
</tr>
<tr style="max-width: 100%;"><td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">forceoff</td>
<td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">Same as ‘off’ but the state cannot be controlled.
Attempts to write to the control file are rejected.</td>
</tr>
<tr style="max-width: 100%;"><td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">notsupported</td>
<td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">The processor does not support SMT. It’s therefore
not affected by the SMT implications of L1TF.
Attempts to write to the control file are rejected.</td>
</tr>
</tbody>
</table></div></div>
</div></blockquote>
<p style="max-width: 100%;">The possible states which can be written into this file to control SMT
state are:</p>
<ul style="max-width: 100%;">
<li style="max-width: 100%;">on</li>
<li style="max-width: 100%;">off</li>
<li style="max-width: 100%;">forceoff</li>
</ul>
</div></blockquote>
<p style="max-width: 100%;">/sys/devices/system/cpu/smt/active:</p>
<blockquote style="max-width: 100%; color: rgba(0, 0, 0, 0.65098); margin-left: 2px; margin-right: 6px; padding-left: 16px; border-left-width: 3px; border-left-style: solid; border-left-color: rgba(0, 0, 0, 0.0980392);">
<p style="max-width: 100%;">This file reports whether SMT is enabled and active, i.e. if on any
physical core two or more sibling threads are online.</p></blockquote>
<p style="max-width: 100%;">SMT control is also possible at boot time via the l1tf kernel command
line parameter in combination with L1D flush control. See
<span class="converted-anchor" style="max-width: 100%;"><span style="max-width: 100%;">Mitigation control on the kernel command line</span></span>.</p>
</div></blockquote>
</div>
<div style="max-width: 100%;">
<h3 style="font-weight: bold; font-size: 1.25em; max-width: 100%;">5. Disabling EPT</h3>
<blockquote style="max-width: 100%; color: rgba(0, 0, 0, 0.65098); margin-left: 2px; margin-right: 6px; padding-left: 16px; border-left-width: 3px; border-left-style: solid; border-left-color: rgba(0, 0, 0, 0.0980392);">
<div style="max-width: 100%;"><p style="max-width: 100%;">Disabling EPT for virtual machines provides full mitigation for L1TF even
with SMT enabled, because the effective page tables for guests are
managed and sanitized by the hypervisor. Though disabling EPT has a
significant performance impact especially when the Meltdown mitigation
KPTI is enabled.</p>
<p style="max-width: 100%;">EPT can be disabled in the hypervisor via the ‘kvm-intel.ept’ parameter.</p>
</div></blockquote>
<p style="max-width: 100%;">There is ongoing research and development for new mitigation mechanisms to
address the performance impact of disabling SMT or EPT.</p>
</div>
</div>
<div style="max-width: 100%;">
<span style="max-width: 100%;"></span><h2 style="font-weight: bold; font-size: 1.43em; max-width: 100%;">Mitigation control on the kernel command line</h2>
<p style="max-width: 100%;">The kernel command line allows to control the L1TF mitigations at boot
time with the option “l1tf=”. The valid arguments for this option are:</p>
<blockquote class="simple" style="max-width: 100%; color: rgba(0, 0, 0, 0.65098); margin-left: 2px; margin-right: 6px; padding-left: 16px;">
<div style="max-width: 100%;"><div style="max-width: 100%;"><div class="scrollable" style="max-width: 100%; overflow-x: scroll; word-wrap: normal;"><table border="1" style="max-width: none; font-size: 0.9em; text-align: start; word-wrap: break-word; border-collapse: collapse;">
<colgroup style="max-width: 100%;">
<col width="16%" style="max-width: 100%;">
<col width="84%" style="max-width: 100%;">
</colgroup>
<tbody valign="top" style="max-width: 100%;">
<tr style="max-width: 100%;"><td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">full</td>
<td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);"><p style="max-width: 100%;">Provides all available mitigations for the L1TF
vulnerability. Disables SMT and enables all mitigations in
the hypervisors, i.e. unconditional L1D flushing</p>
<p style="max-width: 100%;">SMT control and L1D flush control via the sysfs interface
is still possible after boot. Hypervisors will issue a
warning when the first VM is started in a potentially
insecure configuration, i.e. SMT enabled or L1D flush
disabled.</p>
</td>
</tr>
<tr style="max-width: 100%;"><td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">full,force</td>
<td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">Same as ‘full’, but disables SMT and L1D flush runtime
control. Implies the ‘nosmt=force’ command line option.
(i.e. sysfs control of SMT is disabled.)</td>
</tr>
<tr style="max-width: 100%;"><td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">flush</td>
<td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);"><p style="max-width: 100%;">Leaves SMT enabled and enables the default hypervisor
mitigation, i.e. conditional L1D flushing</p>
<p style="max-width: 100%;">SMT control and L1D flush control via the sysfs interface
is still possible after boot. Hypervisors will issue a
warning when the first VM is started in a potentially
insecure configuration, i.e. SMT enabled or L1D flush
disabled.</p>
</td>
</tr>
<tr style="max-width: 100%;"><td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">flush,nosmt</td>
<td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);"><p style="max-width: 100%;">Disables SMT and enables the default hypervisor mitigation,
i.e. conditional L1D flushing.</p>
<p style="max-width: 100%;">SMT control and L1D flush control via the sysfs interface
is still possible after boot. Hypervisors will issue a
warning when the first VM is started in a potentially
insecure configuration, i.e. SMT enabled or L1D flush
disabled.</p>
</td>
</tr>
<tr style="max-width: 100%;"><td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">flush,nowarn</td>
<td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">Same as ‘flush’, but hypervisors will not warn when a VM is
started in a potentially insecure configuration.</td>
</tr>
<tr style="max-width: 100%;"><td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">off</td>
<td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">Disables hypervisor mitigations and doesn’t emit any
warnings.</td>
</tr>
</tbody>
</table></div></div>
</div></blockquote>
<p style="max-width: 100%;">The default is ‘flush’. For details about L1D flushing see <span class="converted-anchor" style="max-width: 100%;"><span style="max-width: 100%;">1. L1D flush on VMENTER</span></span>.</p>
</div>
<div style="max-width: 100%;">
<span style="max-width: 100%;"></span><h2 style="font-weight: bold; font-size: 1.43em; max-width: 100%;">Mitigation control for KVM - module parameter</h2>
<p style="max-width: 100%;">The KVM hypervisor mitigation mechanism, flushing the L1D cache when
entering a guest, can be controlled with a module parameter.</p>
<p style="max-width: 100%;">The option/parameter is “kvm-intel.vmentry_l1d_flush=”. It takes the
following arguments:</p>
<blockquote class="simple" style="max-width: 100%; color: rgba(0, 0, 0, 0.65098); margin-left: 2px; margin-right: 6px; padding-left: 16px;">
<div style="max-width: 100%;"><div style="max-width: 100%;"><div class="scrollable" style="max-width: 100%; overflow-x: scroll; word-wrap: normal;"><table border="1" style="max-width: none; font-size: 0.9em; text-align: start; word-wrap: break-word; border-collapse: collapse;">
<colgroup style="max-width: 100%;">
<col width="16%" style="max-width: 100%;">
<col width="84%" style="max-width: 100%;">
</colgroup>
<tbody valign="top" style="max-width: 100%;">
<tr style="max-width: 100%;"><td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">always</td>
<td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">L1D cache flush on every VMENTER.</td>
</tr>
<tr style="max-width: 100%;"><td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">cond</td>
<td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">Flush L1D on VMENTER only when the code between VMEXIT and
VMENTER can leak host memory which is considered
interesting for an attacker. This still can leak host memory
which allows e.g. to determine the hosts address space layout.</td>
</tr>
<tr style="max-width: 100%;"><td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">never</td>
<td style="max-width: 100%; padding: 0.25em 0.5em; border: 1px solid rgb(216, 216, 216);">Disables the mitigation</td>
</tr>
</tbody>
</table></div></div>
</div></blockquote>
<p style="max-width: 100%;">The parameter can be provided on the kernel command line, as a module
parameter when loading the modules and at runtime modified via the sysfs
file:</p>
<p style="max-width: 100%;">/sys/module/kvm_intel/parameters/vmentry_l1d_flush</p>
<p style="max-width: 100%;">The default is ‘cond’. If ‘l1tf=full,force’ is given on the kernel command
line, then ‘always’ is enforced and the kvm-intel.vmentry_l1d_flush
module parameter is ignored and writes to the sysfs file are rejected.</p>
</div>
<div style="max-width: 100%;">
<h2 style="font-weight: bold; font-size: 1.43em; max-width: 100%;">Mitigation selection guide</h2>
<div style="max-width: 100%;">
<h3 style="font-weight: bold; font-size: 1.25em; max-width: 100%;">1. No virtualization in use</h3>
<blockquote style="max-width: 100%; color: rgba(0, 0, 0, 0.65098); margin-left: 2px; margin-right: 6px; padding-left: 16px; border-left-width: 3px; border-left-style: solid; border-left-color: rgba(0, 0, 0, 0.0980392);">
<p style="max-width: 100%;">The system is protected by the kernel unconditionally and no further
action is required.</p></blockquote>
</div>
<div style="max-width: 100%;">
<h3 style="font-weight: bold; font-size: 1.25em; max-width: 100%;">2. Virtualization with trusted guests</h3>
<blockquote style="max-width: 100%; color: rgba(0, 0, 0, 0.65098); margin-left: 2px; margin-right: 6px; padding-left: 16px; border-left-width: 3px; border-left-style: solid; border-left-color: rgba(0, 0, 0, 0.0980392);">
<div style="max-width: 100%;"><p style="max-width: 100%;">If the guest comes from a trusted source and the guest OS kernel is
guaranteed to have the L1TF mitigations in place the system is fully
protected against L1TF and no further action is required.</p>
<p style="max-width: 100%;">To avoid the overhead of the default L1D flushing on VMENTER the
administrator can disable the flushing via the kernel command line and
sysfs control files. See <span class="converted-anchor" style="max-width: 100%;"><span style="max-width: 100%;">Mitigation control on the kernel command line</span></span> and
<span class="converted-anchor" style="max-width: 100%;"><span style="max-width: 100%;">Mitigation control for KVM - module parameter</span></span>.</p>
</div></blockquote>
</div>
<div style="max-width: 100%;">
<h3 style="font-weight: bold; font-size: 1.25em; max-width: 100%;">3. Virtualization with untrusted guests</h3>
<div style="max-width: 100%;">
<h4 style="font-weight: bold; font-size: 1em; margin: 1em 0px; max-width: 100%;">3.1. SMT not supported or disabled</h4>
<blockquote style="max-width: 100%; color: rgba(0, 0, 0, 0.65098); margin-left: 2px; margin-right: 6px; padding-left: 16px; border-left-width: 3px; border-left-style: solid; border-left-color: rgba(0, 0, 0, 0.0980392);">
<div style="max-width: 100%;"><p style="max-width: 100%;">If SMT is not supported by the processor or disabled in the BIOS or by
the kernel, it’s only required to enforce L1D flushing on VMENTER.</p>
<p style="max-width: 100%;">Conditional L1D flushing is the default behaviour and can be tuned. See
<span class="converted-anchor" style="max-width: 100%;"><span style="max-width: 100%;">Mitigation control on the kernel command line</span></span> and <span class="converted-anchor" style="max-width: 100%;"><span style="max-width: 100%;">Mitigation control for KVM - module parameter</span></span>.</p>
</div></blockquote>
</div>
<div style="max-width: 100%;">
<h4 style="font-weight: bold; font-size: 1em; margin: 1em 0px; max-width: 100%;">3.2. EPT not supported or disabled</h4>
<blockquote style="max-width: 100%; color: rgba(0, 0, 0, 0.65098); margin-left: 2px; margin-right: 6px; padding-left: 16px; border-left-width: 3px; border-left-style: solid; border-left-color: rgba(0, 0, 0, 0.0980392);">
<div style="max-width: 100%;"><p style="max-width: 100%;">If EPT is not supported by the processor or disabled in the hypervisor,
the system is fully protected. SMT can stay enabled and L1D flushing on
VMENTER is not required.</p>
<p style="max-width: 100%;">EPT can be disabled in the hypervisor via the ‘kvm-intel.ept’ parameter.</p>
</div></blockquote>
</div>
<div style="max-width: 100%;">
<h4 style="font-weight: bold; font-size: 1em; margin: 1em 0px; max-width: 100%;">3.3. SMT and EPT supported and active</h4>
<blockquote style="max-width: 100%; color: rgba(0, 0, 0, 0.65098); margin-left: 2px; margin-right: 6px; padding-left: 16px; border-left-width: 3px; border-left-style: solid; border-left-color: rgba(0, 0, 0, 0.0980392);">
<div style="max-width: 100%;"><p style="max-width: 100%;">If SMT and EPT are supported and active then various degrees of
mitigations can be employed:</p>
<ul style="max-width: 100%;">
<li style="max-width: 100%;"><p style="max-width: 100%;">L1D flushing on VMENTER:</p>
<p style="max-width: 100%;">L1D flushing on VMENTER is the minimal protection requirement, but it
is only potent in combination with other mitigation methods.</p>
<p style="max-width: 100%;">Conditional L1D flushing is the default behaviour and can be tuned. See
<span class="converted-anchor" style="max-width: 100%;"><span style="max-width: 100%;">Mitigation control on the kernel command line</span></span> and <span class="converted-anchor" style="max-width: 100%;"><span style="max-width: 100%;">Mitigation control for KVM - module parameter</span></span>.</p>
</li>
<li style="max-width: 100%;"><p style="max-width: 100%;">Guest confinement:</p>
<p style="max-width: 100%;">Confinement of guests to a single or a group of physical cores which
are not running any other processes, can reduce the attack surface
significantly, but interrupts, soft interrupts and kernel threads can
still expose valuable data to a potential attacker. See
<span class="converted-anchor" style="max-width: 100%;"><span style="max-width: 100%;">2. Guest VCPU confinement to dedicated physical cores</span></span>.</p>
</li>
<li style="max-width: 100%;"><p style="max-width: 100%;">Interrupt isolation:</p>
<p style="max-width: 100%;">Isolating the guest CPUs from interrupts can reduce the attack surface
further, but still allows a malicious guest to explore a limited amount
of host physical memory. This can at least be used to gain knowledge
about the host address space layout. The interrupts which have a fixed
affinity to the CPUs which run the untrusted guests can depending on
the scenario still trigger soft interrupts and schedule kernel threads
which might expose valuable information. See
<span class="converted-anchor" style="max-width: 100%;"><span style="max-width: 100%;">3. Interrupt affinity</span></span>.</p>
</li>
</ul>
</div></blockquote>
<p style="max-width: 100%;">The above three mitigation methods combined can provide protection to a
certain degree, but the risk of the remaining attack surface has to be
carefully analyzed. For full protection the following methods are
available:</p>
<blockquote style="max-width: 100%; color: rgba(0, 0, 0, 0.65098); margin-left: 2px; margin-right: 6px; padding-left: 16px; border-left-width: 3px; border-left-style: solid; border-left-color: rgba(0, 0, 0, 0.0980392);">
<div style="max-width: 100%;"><ul style="max-width: 100%;">
<li style="max-width: 100%;"><p style="max-width: 100%;">Disabling SMT:</p>
<p style="max-width: 100%;">Disabling SMT and enforcing the L1D flushing provides the maximum
amount of protection. This mitigation is not depending on any of the
above mitigation methods.</p>
<p style="max-width: 100%;">SMT control and L1D flushing can be tuned by the command line
parameters ‘nosmt’, ‘l1tf’, ‘kvm-intel.vmentry_l1d_flush’ and at run
time with the matching sysfs control files. See <span class="converted-anchor" style="max-width: 100%;"><span style="max-width: 100%;">4. SMT control</span></span>,
<span class="converted-anchor" style="max-width: 100%;"><span style="max-width: 100%;">Mitigation control on the kernel command line</span></span> and
<span class="converted-anchor" style="max-width: 100%;"><span style="max-width: 100%;">Mitigation control for KVM - module parameter</span></span>.</p>
</li>
<li style="max-width: 100%;"><p style="max-width: 100%;">Disabling EPT:</p>
<p style="max-width: 100%;">Disabling EPT provides the maximum amount of protection as well. It is
not depending on any of the above mitigation methods. SMT can stay
enabled and L1D flushing is not required, but the performance impact is
significant.</p>
<p style="max-width: 100%;">EPT can be disabled in the hypervisor via the ‘kvm-intel.ept’
parameter.</p>
</li>
</ul>
</div></blockquote>
</div>
<div style="max-width: 100%;">
<h4 style="font-weight: bold; font-size: 1em; margin: 1em 0px; max-width: 100%;">3.4. Nested virtual machines</h4>
<p style="max-width: 100%;">When nested virtualization is in use, three operating systems are involved:
the bare metal hypervisor, the nested hypervisor and the nested virtual
machine. VMENTER operations from the nested hypervisor into the nested
guest will always be processed by the bare metal hypervisor. If KVM is the
bare metal hypervisor it wiil:</p>
<blockquote style="max-width: 100%; color: rgba(0, 0, 0, 0.65098); margin-left: 2px; margin-right: 6px; padding-left: 16px; border-left-width: 3px; border-left-style: solid; border-left-color: rgba(0, 0, 0, 0.0980392);">
<div style="max-width: 100%;"><ul style="max-width: 100%;">
<li style="max-width: 100%;">Flush the L1D cache on every switch from the nested hypervisor to the
nested virtual machine, so that the nested hypervisor’s secrets are not
exposed to the nested virtual machine;</li>
<li style="max-width: 100%;">Flush the L1D cache on every switch from the nested virtual machine to
the nested hypervisor; this is a complex operation, and flushing the L1D
cache avoids that the bare metal hypervisor’s secrets are exposed to the
nested virtual machine;</li>
<li style="max-width: 100%;">Instruct the nested hypervisor to not perform any L1D cache flush. This
is an optimization to avoid double L1D flushing.</li>
</ul>
</div></blockquote>
</div>
</div>
</div>
<div style="max-width: 100%;">
<span style="max-width: 100%;"></span><h2 style="font-weight: bold; font-size: 1.43em; max-width: 100%;">Default mitigations</h2>
<blockquote style="max-width: 100%; color: rgba(0, 0, 0, 0.65098); margin-left: 2px; margin-right: 6px; padding-left: 16px; border-left-width: 3px; border-left-style: solid; border-left-color: rgba(0, 0, 0, 0.0980392);">
<div style="max-width: 100%;"><p style="max-width: 100%;">The kernel default mitigations for vulnerable processors are:</p>
<ul style="max-width: 100%;">
<li style="max-width: 100%;">PTE inversion to protect against malicious user space. This is done
unconditionally and cannot be controlled.</li>
<li style="max-width: 100%;">L1D conditional flushing on VMENTER when EPT is enabled for
a guest.</li>
</ul>
<p style="max-width: 100%;">The kernel does not by default enforce the disabling of SMT, which leaves
SMT systems vulnerable when running untrusted guests with EPT enabled.</p>
<p style="max-width: 100%;">The rationale for this choice is:</p>
<ul style="max-width: 100%;">
<li style="max-width: 100%;"><p style="max-width: 100%;">Force disabling SMT can break existing setups, especially with
unattended updates.</p>
</li>
<li style="max-width: 100%;"><p style="max-width: 100%;">If regular users run untrusted guests on their machine, then L1TF is
just an add on to other malware which might be embedded in an untrusted
guest, e.g. spam-bots or attacks on the local network.</p>
<p style="max-width: 100%;">There is no technical way to prevent a user from running untrusted code
on their machines blindly.</p>
</li>
<li style="max-width: 100%;"><p style="max-width: 100%;">It’s technically extremely unlikely and from today’s knowledge even
impossible that L1TF can be exploited via the most popular attack
mechanisms like JavaScript because these mechanisms have no way to
control PTEs. If this would be possible and not other mitigation would
be possible, then the default might be different.</p>
</li>
<li style="max-width: 100%;"><p style="max-width: 100%;">The administrators of cloud and hosting setups have to carefully
analyze the risk for their scenarios and make the appropriate
mitigation choices, which might even vary across their deployed
machines and also result in other changes of their overall setup.
There is no way for the kernel to provide a sensible default for this
kind of scenarios.</p>
</li>
</ul>
</div></blockquote>
</div>
<p style="max-width: 100%;">
© Copyright The kernel development community.
</p></div></div></div><br><br><div id="AppleMailSignature">Sent from my phone</div></body></html>