Skip to content

Commit d8e460d

Browse files
PlaidCatPlaidCat
committed
arm64/sve: Discard stale CPU state when handling SVE traps
jira LE-2349 cve CVE-2024-50275 Rebuild_History Non-Buildable kernel-4.18.0-553.37.1.el8_10 commit-author Mark Brown <broonie@kernel.org> commit 751ecf6 Empty-Commit: Cherry-Pick Conflicts during history rebuild. Will be included in final tarball splat. Ref for failed cherry-pick at: ciq/ciq_backports/kernel-4.18.0-553.37.1.el8_10/751ecf6a.failed The logic for handling SVE traps manipulates saved FPSIMD/SVE state incorrectly, and a race with preemption can result in a task having TIF_SVE set and TIF_FOREIGN_FPSTATE clear even though the live CPU state is stale (e.g. with SVE traps enabled). This has been observed to result in warnings from do_sve_acc() where SVE traps are not expected while TIF_SVE is set: | if (test_and_set_thread_flag(TIF_SVE)) | WARN_ON(1); /* SVE access shouldn't have trapped */ Warnings of this form have been reported intermittently, e.g. https://lore.kernel.org/linux-arm-kernel/CA+G9fYtEGe_DhY2Ms7+L7NKsLYUomGsgqpdBj+QwDLeSg=JhGg@mail.gmail.com/ https://lore.kernel.org/linux-arm-kernel/000000000000511e9a060ce5a45c@google.com/ The race can occur when the SVE trap handler is preempted before and after manipulating the saved FPSIMD/SVE state, starting and ending on the same CPU, e.g. | void do_sve_acc(unsigned long esr, struct pt_regs *regs) | { | // Trap on CPU 0 with TIF_SVE clear, SVE traps enabled | // task->fpsimd_cpu is 0. | // per_cpu_ptr(&fpsimd_last_state, 0) is task. | | ... | | // Preempted; migrated from CPU 0 to CPU 1. | // TIF_FOREIGN_FPSTATE is set. | | get_cpu_fpsimd_context(); | | if (test_and_set_thread_flag(TIF_SVE)) | WARN_ON(1); /* SVE access shouldn't have trapped */ | | sve_init_regs() { | if (!test_thread_flag(TIF_FOREIGN_FPSTATE)) { | ... | } else { | fpsimd_to_sve(current); | current->thread.fp_type = FP_STATE_SVE; | } | } | | put_cpu_fpsimd_context(); | | // Preempted; migrated from CPU 1 to CPU 0. | // task->fpsimd_cpu is still 0 | // If per_cpu_ptr(&fpsimd_last_state, 0) is still task then: | // - Stale HW state is reused (with SVE traps enabled) | // - TIF_FOREIGN_FPSTATE is cleared | // - A return to userspace skips HW state restore | } Fix the case where the state is not live and TIF_FOREIGN_FPSTATE is set by calling fpsimd_flush_task_state() to detach from the saved CPU state. This ensures that a subsequent context switch will not reuse the stale CPU state, and will instead set TIF_FOREIGN_FPSTATE, forcing the new state to be reloaded from memory prior to a return to userspace. Fixes: cccb78c ("arm64/sve: Rework SVE access trap to convert state in registers") Reported-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Mark Brown <broonie@kernel.org> Cc: stable@vger.kernel.org Reviewed-by: Mark Rutland <mark.rutland@arm.com> Link: https://lore.kernel.org/r/20241030-arm64-fpsimd-foreign-flush-v1-1-bd7bd66905a2@kernel.org Signed-off-by: Will Deacon <will@kernel.org> (cherry picked from commit 751ecf6) Signed-off-by: Jonathan Maple <jmaple@ciq.com> # Conflicts: # arch/arm64/kernel/fpsimd.c
1 parent 7553d0f commit d8e460d

File tree

1 file changed

+251
-0
lines changed

1 file changed

+251
-0
lines changed

‎ciq/ciq_backports/kernel-4.18.0-553.37.1.el8_10/751ecf6a.failed‎

Lines changed: 251 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,251 @@
1+
arm64/sve: Discard stale CPU state when handling SVE traps
2+
3+
jira LE-2349
4+
cve CVE-2024-50275
5+
Rebuild_History Non-Buildable kernel-4.18.0-553.37.1.el8_10
6+
commit-author Mark Brown <broonie@kernel.org>
7+
commit 751ecf6afd6568adc98f2a6052315552c0483d18
8+
Empty-Commit: Cherry-Pick Conflicts during history rebuild.
9+
Will be included in final tarball splat. Ref for failed cherry-pick at:
10+
ciq/ciq_backports/kernel-4.18.0-553.37.1.el8_10/751ecf6a.failed
11+
12+
The logic for handling SVE traps manipulates saved FPSIMD/SVE state
13+
incorrectly, and a race with preemption can result in a task having
14+
TIF_SVE set and TIF_FOREIGN_FPSTATE clear even though the live CPU state
15+
is stale (e.g. with SVE traps enabled). This has been observed to result
16+
in warnings from do_sve_acc() where SVE traps are not expected while
17+
TIF_SVE is set:
18+
19+
| if (test_and_set_thread_flag(TIF_SVE))
20+
| WARN_ON(1); /* SVE access shouldn't have trapped */
21+
22+
Warnings of this form have been reported intermittently, e.g.
23+
24+
https://lore.kernel.org/linux-arm-kernel/CA+G9fYtEGe_DhY2Ms7+L7NKsLYUomGsgqpdBj+QwDLeSg=JhGg@mail.gmail.com/
25+
https://lore.kernel.org/linux-arm-kernel/000000000000511e9a060ce5a45c@google.com/
26+
27+
The race can occur when the SVE trap handler is preempted before and
28+
after manipulating the saved FPSIMD/SVE state, starting and ending on
29+
the same CPU, e.g.
30+
31+
| void do_sve_acc(unsigned long esr, struct pt_regs *regs)
32+
| {
33+
| // Trap on CPU 0 with TIF_SVE clear, SVE traps enabled
34+
| // task->fpsimd_cpu is 0.
35+
| // per_cpu_ptr(&fpsimd_last_state, 0) is task.
36+
|
37+
| ...
38+
|
39+
| // Preempted; migrated from CPU 0 to CPU 1.
40+
| // TIF_FOREIGN_FPSTATE is set.
41+
|
42+
| get_cpu_fpsimd_context();
43+
|
44+
| if (test_and_set_thread_flag(TIF_SVE))
45+
| WARN_ON(1); /* SVE access shouldn't have trapped */
46+
|
47+
| sve_init_regs() {
48+
| if (!test_thread_flag(TIF_FOREIGN_FPSTATE)) {
49+
| ...
50+
| } else {
51+
| fpsimd_to_sve(current);
52+
| current->thread.fp_type = FP_STATE_SVE;
53+
| }
54+
| }
55+
|
56+
| put_cpu_fpsimd_context();
57+
|
58+
| // Preempted; migrated from CPU 1 to CPU 0.
59+
| // task->fpsimd_cpu is still 0
60+
| // If per_cpu_ptr(&fpsimd_last_state, 0) is still task then:
61+
| // - Stale HW state is reused (with SVE traps enabled)
62+
| // - TIF_FOREIGN_FPSTATE is cleared
63+
| // - A return to userspace skips HW state restore
64+
| }
65+
66+
Fix the case where the state is not live and TIF_FOREIGN_FPSTATE is set
67+
by calling fpsimd_flush_task_state() to detach from the saved CPU
68+
state. This ensures that a subsequent context switch will not reuse the
69+
stale CPU state, and will instead set TIF_FOREIGN_FPSTATE, forcing the
70+
new state to be reloaded from memory prior to a return to userspace.
71+
72+
Fixes: cccb78ce89c4 ("arm64/sve: Rework SVE access trap to convert state in registers")
73+
Reported-by: Mark Rutland <mark.rutland@arm.com>
74+
Signed-off-by: Mark Brown <broonie@kernel.org>
75+
Cc: stable@vger.kernel.org
76+
Reviewed-by: Mark Rutland <mark.rutland@arm.com>
77+
Link: https://lore.kernel.org/r/20241030-arm64-fpsimd-foreign-flush-v1-1-bd7bd66905a2@kernel.org
78+
Signed-off-by: Will Deacon <will@kernel.org>
79+
(cherry picked from commit 751ecf6afd6568adc98f2a6052315552c0483d18)
80+
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
81+
82+
# Conflicts:
83+
# arch/arm64/kernel/fpsimd.c
84+
diff --cc arch/arm64/kernel/fpsimd.c
85+
index 48f6b77fdece,6d21971ae559..000000000000
86+
--- a/arch/arm64/kernel/fpsimd.c
87+
+++ b/arch/arm64/kernel/fpsimd.c
88+
@@@ -934,6 -1218,159 +934,162 @@@ void fpsimd_release_task(struct task_st
89+
90+
#endif /* CONFIG_ARM64_SVE */
91+
92+
++<<<<<<< HEAD
93+
++=======
94+
+ #ifdef CONFIG_ARM64_SME
95+
+
96+
+ /*
97+
+ * Ensure that task->thread.sme_state is allocated and sufficiently large.
98+
+ *
99+
+ * This function should be used only in preparation for replacing
100+
+ * task->thread.sme_state with new data. The memory is always zeroed
101+
+ * here to prevent stale data from showing through: this is done in
102+
+ * the interest of testability and predictability, the architecture
103+
+ * guarantees that when ZA is enabled it will be zeroed.
104+
+ */
105+
+ void sme_alloc(struct task_struct *task, bool flush)
106+
+ {
107+
+ if (task->thread.sme_state) {
108+
+ if (flush)
109+
+ memset(task->thread.sme_state, 0,
110+
+ sme_state_size(task));
111+
+ return;
112+
+ }
113+
+
114+
+ /* This could potentially be up to 64K. */
115+
+ task->thread.sme_state =
116+
+ kzalloc(sme_state_size(task), GFP_KERNEL);
117+
+ }
118+
+
119+
+ static void sme_free(struct task_struct *task)
120+
+ {
121+
+ kfree(task->thread.sme_state);
122+
+ task->thread.sme_state = NULL;
123+
+ }
124+
+
125+
+ void cpu_enable_sme(const struct arm64_cpu_capabilities *__always_unused p)
126+
+ {
127+
+ /* Set priority for all PEs to architecturally defined minimum */
128+
+ write_sysreg_s(read_sysreg_s(SYS_SMPRI_EL1) & ~SMPRI_EL1_PRIORITY_MASK,
129+
+ SYS_SMPRI_EL1);
130+
+
131+
+ /* Allow SME in kernel */
132+
+ write_sysreg(read_sysreg(CPACR_EL1) | CPACR_EL1_SMEN_EL1EN, CPACR_EL1);
133+
+ isb();
134+
+
135+
+ /* Ensure all bits in SMCR are set to known values */
136+
+ write_sysreg_s(0, SYS_SMCR_EL1);
137+
+
138+
+ /* Allow EL0 to access TPIDR2 */
139+
+ write_sysreg(read_sysreg(SCTLR_EL1) | SCTLR_ELx_ENTP2, SCTLR_EL1);
140+
+ isb();
141+
+ }
142+
+
143+
+ void cpu_enable_sme2(const struct arm64_cpu_capabilities *__always_unused p)
144+
+ {
145+
+ /* This must be enabled after SME */
146+
+ BUILD_BUG_ON(ARM64_SME2 <= ARM64_SME);
147+
+
148+
+ /* Allow use of ZT0 */
149+
+ write_sysreg_s(read_sysreg_s(SYS_SMCR_EL1) | SMCR_ELx_EZT0_MASK,
150+
+ SYS_SMCR_EL1);
151+
+ }
152+
+
153+
+ void cpu_enable_fa64(const struct arm64_cpu_capabilities *__always_unused p)
154+
+ {
155+
+ /* This must be enabled after SME */
156+
+ BUILD_BUG_ON(ARM64_SME_FA64 <= ARM64_SME);
157+
+
158+
+ /* Allow use of FA64 */
159+
+ write_sysreg_s(read_sysreg_s(SYS_SMCR_EL1) | SMCR_ELx_FA64_MASK,
160+
+ SYS_SMCR_EL1);
161+
+ }
162+
+
163+
+ void __init sme_setup(void)
164+
+ {
165+
+ struct vl_info *info = &vl_info[ARM64_VEC_SME];
166+
+ int min_bit, max_bit;
167+
+
168+
+ if (!system_supports_sme())
169+
+ return;
170+
+
171+
+ /*
172+
+ * SME doesn't require any particular vector length be
173+
+ * supported but it does require at least one. We should have
174+
+ * disabled the feature entirely while bringing up CPUs but
175+
+ * let's double check here. The bitmap is SVE_VQ_MAP sized for
176+
+ * sharing with SVE.
177+
+ */
178+
+ WARN_ON(bitmap_empty(info->vq_map, SVE_VQ_MAX));
179+
+
180+
+ min_bit = find_last_bit(info->vq_map, SVE_VQ_MAX);
181+
+ info->min_vl = sve_vl_from_vq(__bit_to_vq(min_bit));
182+
+
183+
+ max_bit = find_first_bit(info->vq_map, SVE_VQ_MAX);
184+
+ info->max_vl = sve_vl_from_vq(__bit_to_vq(max_bit));
185+
+
186+
+ WARN_ON(info->min_vl > info->max_vl);
187+
+
188+
+ /*
189+
+ * For the default VL, pick the maximum supported value <= 32
190+
+ * (256 bits) if there is one since this is guaranteed not to
191+
+ * grow the signal frame when in streaming mode, otherwise the
192+
+ * minimum available VL will be used.
193+
+ */
194+
+ set_sme_default_vl(find_supported_vector_length(ARM64_VEC_SME, 32));
195+
+
196+
+ pr_info("SME: minimum available vector length %u bytes per vector\n",
197+
+ info->min_vl);
198+
+ pr_info("SME: maximum available vector length %u bytes per vector\n",
199+
+ info->max_vl);
200+
+ pr_info("SME: default vector length %u bytes per vector\n",
201+
+ get_sme_default_vl());
202+
+ }
203+
+
204+
+ void sme_suspend_exit(void)
205+
+ {
206+
+ u64 smcr = 0;
207+
+
208+
+ if (!system_supports_sme())
209+
+ return;
210+
+
211+
+ if (system_supports_fa64())
212+
+ smcr |= SMCR_ELx_FA64;
213+
+ if (system_supports_sme2())
214+
+ smcr |= SMCR_ELx_EZT0;
215+
+
216+
+ write_sysreg_s(smcr, SYS_SMCR_EL1);
217+
+ write_sysreg_s(0, SYS_SMPRI_EL1);
218+
+ }
219+
+
220+
+ #endif /* CONFIG_ARM64_SME */
221+
+
222+
+ static void sve_init_regs(void)
223+
+ {
224+
+ /*
225+
+ * Convert the FPSIMD state to SVE, zeroing all the state that
226+
+ * is not shared with FPSIMD. If (as is likely) the current
227+
+ * state is live in the registers then do this there and
228+
+ * update our metadata for the current task including
229+
+ * disabling the trap, otherwise update our in-memory copy.
230+
+ * We are guaranteed to not be in streaming mode, we can only
231+
+ * take a SVE trap when not in streaming mode and we can't be
232+
+ * in streaming mode when taking a SME trap.
233+
+ */
234+
+ if (!test_thread_flag(TIF_FOREIGN_FPSTATE)) {
235+
+ unsigned long vq_minus_one =
236+
+ sve_vq_from_vl(task_get_sve_vl(current)) - 1;
237+
+ sve_set_vq(vq_minus_one);
238+
+ sve_flush_live(true, vq_minus_one);
239+
+ fpsimd_bind_task_to_cpu();
240+
+ } else {
241+
+ fpsimd_to_sve(current);
242+
+ current->thread.fp_type = FP_STATE_SVE;
243+
+ fpsimd_flush_task_state(current);
244+
+ }
245+
+ }
246+
+
247+
++>>>>>>> 751ecf6afd65 (arm64/sve: Discard stale CPU state when handling SVE traps)
248+
/*
249+
* Trapped SVE access
250+
*
251+
* Unmerged path arch/arm64/kernel/fpsimd.c

0 commit comments

Comments
 (0)