Mitigate straight-line-speculation
Bugzilla: https://bugzilla.redhat.com/2062177
CVE: CVE-2021-26341
omitted-fix: aa8e73eed7d3 ("crypto: x86/sm3 - Fixup SLS")
our tree doesn't have arch/x86/crypto/sm3-avx-asm_64.S.
Signed-off-by: Wander Lairson Costa wander@redhat.com
This fixes the unconditional branch predictor vulnerability in some AMD processors.
This depends on a version of gcc that supports -mharden-sls=all.
7a0be3ec (Wander Lairson Costa) x86,config: Enable straight-line-speculation fix
7a86ed52 (Wander Lairson Costa) objtool: Fix SLS validation for kcov tail-call replacement
6dddf183 (Wander Lairson Costa) objtool,x86: Replace alternatives with .retpoline_sites
5434a157 (Wander Lairson Costa) objtool: Remove reloc symbol type checks in get_alt_entry()
4d74eb91 (Wander Lairson Costa) objtool: print out the symbol type when complaining about it
2fe9a5a4 (Wander Lairson Costa) objtool: Teach get_alt_entry() about more relocation types
16729909 (Wander Lairson Costa) objtool: Introduce CFI hash
e4c13c03 (Wander Lairson Costa) objtool: Explicitly avoid self modifying code in .altinstr_replacement
54ea82ac (Wander Lairson Costa) objtool: Classify symbols
cde00c0d (Wander Lairson Costa) crypto: x86/poly1305 - Fixup SLS
893b10fc (Wander Lairson Costa) x86/mce: Reduce number of machine checks taken during recovery
acf99804 (Wander Lairson Costa) x86: Add straight-line-speculation mitigation
9a6507ef (Wander Lairson Costa) kbuild: reuse $(cmd_objtool) for cmd_cc_lto_link_modules
15405689 (Wander Lairson Costa) kbuild: detect objtool update without using .SECONDEXPANSION
3aed19cf (Wander Lairson Costa) kbuild: factor out OBJECT_FILES_NON_STANDARD check into a macro
62ce565a (Wander Lairson Costa) kbuild: store the objtool command in *.cmd files
a9eaeef2 (Wander Lairson Costa) kbuild: rename __objtool_obj and reuse it for cmd_cc_lto_link_modules
8c13c52d (Wander Lairson Costa) kbuild: move objtool_args back to scripts/Makefile.build
c66b88c1 (Wander Lairson Costa) kbuild: clean up objtool_args slightly
6aac5d40 (Wander Lairson Costa) kbuild: remove stale *.symversions
ddf32ebc (Wander Lairson Costa) kbuild: remove unused quiet_cmd_update_lto_symversions
2f544ff1 (Wander Lairson Costa) kbuild: Fix TRIM_UNUSED_KSYMS with LTO_CLANG
d81a263c (Wander Lairson Costa) x86/retpoline: Create a retpoline thunk array
2742eeaa (Wander Lairson Costa) x86/retpoline: Move the retpoline thunk declarations to nospec-branch.h
12cb50ad (Wander Lairson Costa) x86/asm: Fixup odd GEN-for-each-reg.h usage
191aa7b9 (Wander Lairson Costa) x86/retpoline: Remove unused replacement symbols
1a9b6fe0 (Wander Lairson Costa) x86/alternative: Relax text_poke_bp() constraint
20f3f5b2 (Wander Lairson Costa) objtool: Add straight-line-speculation validation
d7175c56 (Wander Lairson Costa) objtool: Handle __sanitize_cov*() tail calls
299dd44c (Wander Lairson Costa) x86: Prepare inline-asm for straight-line-speculation
45fecfe7 (Wander Lairson Costa) x86: Prepare asm files for straight-line-speculation
46b7d810 (Wander Lairson Costa) x86/lib/atomic64_386_32: Rename things
639b89cc (Wander Lairson Costa) x86/entry: Use the correct fence macro after swapgs in kernel CR3
ced5c004 (Wander Lairson Costa) x86/entry: Add a fence for kernel entry SWAPGS in paranoid_entry()
b03dc5cf (Wander Lairson Costa) x86/mce: Drop copyin special case for #MC
arch/x86/Kconfig | 12 + arch/x86/Makefile | 4 + arch/x86/boot/compressed/efi_thunk_64.S | 2 +- arch/x86/boot/compressed/head_64.S | 8 +- arch/x86/boot/compressed/mem_encrypt.S | 6 +- arch/x86/crypto/aegis128-aesni-asm.S | 48 +-- arch/x86/crypto/aes_ctrby8_avx-x86_64.S | 2 +- arch/x86/crypto/aesni-intel_asm.S | 56 +-- arch/x86/crypto/aesni-intel_avx-x86_64.S | 40 +- arch/x86/crypto/blake2s-core.S | 4 +- arch/x86/crypto/blowfish-x86_64-asm_64.S | 12 +- arch/x86/crypto/camellia-aesni-avx-asm_64.S | 14 +- arch/x86/crypto/camellia-aesni-avx2-asm_64.S | 14 +- arch/x86/crypto/camellia-x86_64-asm_64.S | 12 +- arch/x86/crypto/cast5-avx-x86_64-asm_64.S | 12 +- arch/x86/crypto/cast6-avx-x86_64-asm_64.S | 10 +- arch/x86/crypto/chacha-avx2-x86_64.S | 6 +- arch/x86/crypto/chacha-avx512vl-x86_64.S | 6 +- arch/x86/crypto/chacha-ssse3-x86_64.S | 8 +- arch/x86/crypto/crc32-pclmul_asm.S | 2 +- arch/x86/crypto/crc32c-pcl-intel-asm_64.S | 2 +- arch/x86/crypto/crct10dif-pcl-asm_64.S | 2 +- arch/x86/crypto/des3_ede-asm_64.S | 4 +- arch/x86/crypto/ghash-clmulni-intel_asm.S | 6 +- arch/x86/crypto/nh-avx2-x86_64.S | 2 +- arch/x86/crypto/nh-sse2-x86_64.S | 2 +- arch/x86/crypto/poly1305-x86_64-cryptogams.pl | 38 +- arch/x86/crypto/serpent-avx-x86_64-asm_64.S | 10 +- arch/x86/crypto/serpent-avx2-asm_64.S | 10 +- arch/x86/crypto/serpent-sse2-i586-asm_32.S | 6 +- arch/x86/crypto/serpent-sse2-x86_64-asm_64.S | 6 +- arch/x86/crypto/sha1_avx2_x86_64_asm.S | 2 +- arch/x86/crypto/sha1_ni_asm.S | 2 +- arch/x86/crypto/sha1_ssse3_asm.S | 2 +- arch/x86/crypto/sha256-avx-asm.S | 2 +- arch/x86/crypto/sha256-avx2-asm.S | 2 +- arch/x86/crypto/sha256-ssse3-asm.S | 2 +- arch/x86/crypto/sha256_ni_asm.S | 2 +- arch/x86/crypto/sha512-avx-asm.S | 2 +- arch/x86/crypto/sha512-avx2-asm.S | 2 +- arch/x86/crypto/sha512-ssse3-asm.S | 2 +- arch/x86/crypto/twofish-avx-x86_64-asm_64.S | 10 +- arch/x86/crypto/twofish-i586-asm_32.S | 4 +- arch/x86/crypto/twofish-x86_64-asm_64-3way.S | 6 +- arch/x86/crypto/twofish-x86_64-asm_64.S | 4 +- arch/x86/entry/entry_32.S | 2 +- arch/x86/entry/entry_64.S | 39 +- arch/x86/entry/thunk_32.S | 2 +- arch/x86/entry/thunk_64.S | 2 +- arch/x86/entry/vdso/vdso32/system_call.S | 2 +- arch/x86/entry/vdso/vsgx.S | 2 +- arch/x86/entry/vsyscall/vsyscall_emu_64.S | 6 +- arch/x86/include/asm/asm-prototypes.h | 18 - arch/x86/include/asm/linkage.h | 14 + arch/x86/include/asm/nospec-branch.h | 13 + arch/x86/include/asm/paravirt.h | 2 +- arch/x86/include/asm/qspinlock_paravirt.h | 4 +- arch/x86/include/asm/static_call.h | 2 +- arch/x86/kernel/acpi/wakeup_32.S | 6 +- arch/x86/kernel/alternative.c | 51 ++- arch/x86/kernel/ftrace.c | 2 +- arch/x86/kernel/ftrace_32.S | 6 +- arch/x86/kernel/ftrace_64.S | 8 +- arch/x86/kernel/head_32.S | 2 +- arch/x86/kernel/irqflags.S | 2 +- arch/x86/kernel/kprobes/core.c | 2 +- arch/x86/kernel/paravirt.c | 2 +- arch/x86/kernel/relocate_kernel_32.S | 10 +- arch/x86/kernel/relocate_kernel_64.S | 10 +- arch/x86/kernel/sev_verify_cbit.S | 2 +- arch/x86/kernel/static_call.c | 5 +- arch/x86/kernel/verify_cpu.S | 4 +- arch/x86/kernel/vmlinux.lds.S | 14 + arch/x86/kvm/emulate.c | 4 +- arch/x86/kvm/svm/vmenter.S | 4 +- arch/x86/kvm/vmx/vmenter.S | 14 +- arch/x86/lib/atomic64_386_32.S | 86 +++-- arch/x86/lib/atomic64_cx8_32.S | 16 +- arch/x86/lib/checksum_32.S | 8 +- arch/x86/lib/clear_page_64.S | 6 +- arch/x86/lib/cmpxchg16b_emu.S | 4 +- arch/x86/lib/cmpxchg8b_emu.S | 4 +- arch/x86/lib/copy_mc_64.S | 6 +- arch/x86/lib/copy_page_64.S | 4 +- arch/x86/lib/copy_user_64.S | 30 +- arch/x86/lib/csum-copy_64.S | 2 +- arch/x86/lib/error-inject.c | 3 +- arch/x86/lib/getuser.S | 22 +- arch/x86/lib/hweight.S | 6 +- arch/x86/lib/iomap_copy_64.S | 2 +- arch/x86/lib/memcpy_64.S | 12 +- arch/x86/lib/memmove_64.S | 4 +- arch/x86/lib/memset_64.S | 6 +- arch/x86/lib/msr-reg.S | 4 +- arch/x86/lib/putuser.S | 6 +- arch/x86/lib/retpoline.S | 60 +-- arch/x86/math-emu/div_Xsig.S | 2 +- arch/x86/math-emu/div_small.S | 2 +- arch/x86/math-emu/mul_Xsig.S | 6 +- arch/x86/math-emu/polynom_Xsig.S | 2 +- arch/x86/math-emu/reg_norm.S | 6 +- arch/x86/math-emu/reg_round.S | 2 +- arch/x86/math-emu/reg_u_add.S | 2 +- arch/x86/math-emu/reg_u_div.S | 2 +- arch/x86/math-emu/reg_u_mul.S | 2 +- arch/x86/math-emu/reg_u_sub.S | 2 +- arch/x86/math-emu/round_Xsig.S | 4 +- arch/x86/math-emu/shr_Xsig.S | 8 +- arch/x86/math-emu/wm_shrx.S | 16 +- arch/x86/mm/mem_encrypt_boot.S | 4 +- arch/x86/net/bpf_jit_comp.c | 1 - arch/x86/platform/efi/efi_stub_32.S | 2 +- arch/x86/platform/efi/efi_stub_64.S | 2 +- arch/x86/platform/efi/efi_thunk_64.S | 2 +- arch/x86/platform/olpc/xo1-wakeup.S | 6 +- arch/x86/power/hibernate_asm_32.S | 4 +- arch/x86/power/hibernate_asm_64.S | 4 +- arch/x86/um/checksum_32.S | 4 +- arch/x86/um/setjmp_32.S | 2 +- arch/x86/um/setjmp_64.S | 2 +- arch/x86/xen/xen-asm.S | 8 +- redhat/configs/common/generic/CONFIG_SLS | 1 + samples/ftrace/ftrace-direct-modify.c | 4 +- samples/ftrace/ftrace-direct-too.c | 2 +- samples/ftrace/ftrace-direct.c | 2 +- scripts/Makefile.build | 84 +++-- scripts/Makefile.lib | 17 +- scripts/Makefile.modfinal | 21 +- scripts/Makefile.modpost | 22 +- scripts/gen_autoksyms.sh | 12 - scripts/link-vmlinux.sh | 3 + tools/objtool/arch/x86/decode.c | 151 ++------ tools/objtool/builtin-check.c | 3 +- tools/objtool/check.c | 515 +++++++++++++++++++------- tools/objtool/elf.c | 84 ----- tools/objtool/include/objtool/arch.h | 4 +- tools/objtool/include/objtool/builtin.h | 2 +- tools/objtool/include/objtool/cfi.h | 2 + tools/objtool/include/objtool/check.h | 2 +- tools/objtool/include/objtool/elf.h | 8 +- tools/objtool/orc_gen.c | 15 +- tools/objtool/special.c | 26 +- 142 files changed, 1053 insertions(+), 981 deletions(-)