From patchwork Tue Dec 14 00:52:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 524355 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC548C433EF for ; Tue, 14 Dec 2021 00:59:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244569AbhLNA7k (ORCPT ); Mon, 13 Dec 2021 19:59:40 -0500 Received: from mga05.intel.com ([192.55.52.43]:21411 "EHLO mga05.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229672AbhLNA7i (ORCPT ); Mon, 13 Dec 2021 19:59:38 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1639443578; x=1670979578; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=Z3pTsdcyEg0Cysyg0rQo7WWdsgj62MRS390COBMyxXg=; b=i9CjC1bySUSxj+PGfYsWFoxHiXSfk4mqMC0UCyN7o33hF5CuU4QXNgMS AwQii8uAXgPo2+MEu/xApD+FsHa75udlSFhE1WXNU7J97fw9F/WaLybs2 rHK3W24Hgo180niRur6HPkG+DGJy2Zcbx/4WOaB35QF3HdqT3VU5jBoKY OSnOw4DmdCXC9ZGqHKFrEHPlJ9WJOkOokPSDa6LxQomIRBFvHHZiIV2dZ IkbuovBwh2vpERnVjlBLbzMxUuGHaNq3b4NP08qs8Ne/+FUf8EjF9CSCU 0PbM9e9Rmsr4YKtYNVWoBw6TQWxGuARuZ12GnkmRXidKK/cHFhZz2GkT/ w==; X-IronPort-AV: E=McAfee;i="6200,9189,10197"; a="325139615" X-IronPort-AV: E=Sophos;i="5.88,203,1635231600"; d="scan'208";a="325139615" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Dec 2021 16:59:38 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,203,1635231600"; d="scan'208";a="505148570" Received: from chang-linux-3.sc.intel.com ([172.25.66.175]) by orsmga007.jf.intel.com with ESMTP; 13 Dec 2021 16:59:37 -0800 From: "Chang S. Bae" To: tglx@linutronix.de, bp@suse.de, dave.hansen@linux.intel.com, mingo@kernel.org, luto@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, ebiggers@kernel.org, dan.j.williams@intel.com, charishma1.gairuboyina@intel.com, kumar.n.dwarakanath@intel.com, lalithambika.krishnakumar@intel.com, ravi.v.shankar@intel.com, chang.seok.bae@intel.com Subject: [PATCH v4 02/13] x86/cpufeature: Enumerate Key Locker feature Date: Mon, 13 Dec 2021 16:52:01 -0800 Message-Id: <20211214005212.20588-3-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20211214005212.20588-1-chang.seok.bae@intel.com> References: <20211214005212.20588-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Key Locker is a CPU feature to minimize exposure of clear-text key material. An encoded form, called 'key handle', is referenced for data encryption or decryption instead of accessing the clear text key. A wrapping key loaded in the CPU's software-inaccessible state is used to transform a user key into a key handle. It supports Advanced Encryption Standard (AES) cipher algorithm with new SIMD instruction set like its predecessor (AES-NI). So a new AES implementation will follow in the kernel's crypto library. Here add it to enumerate the hardware capability, but it will not be shown in /proc/cpuinfo as userspace usage is not supported. Make the feature depend on XMM2 as it comes with AES SIMD instructions. Add X86_FEATURE_KEYLOCKER to the disabled features mask. It will be enabled under a new config option. Signed-off-by: Chang S. Bae Reviewed-by: Dan Williams Cc: x86@kernel.org Cc: linux-kernel@vger.kernel.org --- Changes from RFC v2: * Do not publish the feature flag to userspace. * Update the changelog. Changes from RFC v1: * Updated the changelog. --- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/disabled-features.h | 8 +++++++- arch/x86/include/uapi/asm/processor-flags.h | 2 ++ arch/x86/kernel/cpu/cpuid-deps.c | 1 + 4 files changed, 11 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index d5b5f2ab87a0..e1964446bbe5 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -361,6 +361,7 @@ #define X86_FEATURE_AVX512_VPOPCNTDQ (16*32+14) /* POPCNT for vectors of DW/QW */ #define X86_FEATURE_LA57 (16*32+16) /* 5-level page tables */ #define X86_FEATURE_RDPID (16*32+22) /* RDPID instruction */ +#define X86_FEATURE_KEYLOCKER (16*32+23) /* "" Key Locker */ #define X86_FEATURE_BUS_LOCK_DETECT (16*32+24) /* Bus Lock detect */ #define X86_FEATURE_CLDEMOTE (16*32+25) /* CLDEMOTE instruction */ #define X86_FEATURE_MOVDIRI (16*32+27) /* MOVDIRI instruction */ diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h index 8f28fafa98b3..75e1e87640d4 100644 --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -44,6 +44,12 @@ # define DISABLE_OSPKE (1<<(X86_FEATURE_OSPKE & 31)) #endif /* CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS */ +#ifdef CONFIG_X86_KEYLOCKER +# define DISABLE_KEYLOCKER 0 +#else +# define DISABLE_KEYLOCKER (1<<(X86_FEATURE_KEYLOCKER & 31)) +#endif /* CONFIG_X86_KEYLOCKER */ + #ifdef CONFIG_X86_5LEVEL # define DISABLE_LA57 0 #else @@ -85,7 +91,7 @@ #define DISABLED_MASK14 0 #define DISABLED_MASK15 0 #define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP| \ - DISABLE_ENQCMD) + DISABLE_ENQCMD|DISABLE_KEYLOCKER) #define DISABLED_MASK17 0 #define DISABLED_MASK18 0 #define DISABLED_MASK19 0 diff --git a/arch/x86/include/uapi/asm/processor-flags.h b/arch/x86/include/uapi/asm/processor-flags.h index bcba3c643e63..b958a95a0908 100644 --- a/arch/x86/include/uapi/asm/processor-flags.h +++ b/arch/x86/include/uapi/asm/processor-flags.h @@ -124,6 +124,8 @@ #define X86_CR4_PCIDE _BITUL(X86_CR4_PCIDE_BIT) #define X86_CR4_OSXSAVE_BIT 18 /* enable xsave and xrestore */ #define X86_CR4_OSXSAVE _BITUL(X86_CR4_OSXSAVE_BIT) +#define X86_CR4_KEYLOCKER_BIT 19 /* enable Key Locker */ +#define X86_CR4_KEYLOCKER _BITUL(X86_CR4_KEYLOCKER_BIT) #define X86_CR4_SMEP_BIT 20 /* enable SMEP support */ #define X86_CR4_SMEP _BITUL(X86_CR4_SMEP_BIT) #define X86_CR4_SMAP_BIT 21 /* enable SMAP support */ diff --git a/arch/x86/kernel/cpu/cpuid-deps.c b/arch/x86/kernel/cpu/cpuid-deps.c index c881bcafba7d..abe7e04b27d9 100644 --- a/arch/x86/kernel/cpu/cpuid-deps.c +++ b/arch/x86/kernel/cpu/cpuid-deps.c @@ -78,6 +78,7 @@ static const struct cpuid_dep cpuid_deps[] = { { X86_FEATURE_XFD, X86_FEATURE_XSAVES }, { X86_FEATURE_XFD, X86_FEATURE_XGETBV1 }, { X86_FEATURE_AMX_TILE, X86_FEATURE_XFD }, + { X86_FEATURE_KEYLOCKER, X86_FEATURE_XMM2 }, {} }; From patchwork Tue Dec 14 00:52:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 524354 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3CC98C433F5 for ; Tue, 14 Dec 2021 00:59:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229672AbhLNA7l (ORCPT ); Mon, 13 Dec 2021 19:59:41 -0500 Received: from mga05.intel.com ([192.55.52.43]:21415 "EHLO mga05.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244546AbhLNA7j (ORCPT ); Mon, 13 Dec 2021 19:59:39 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1639443579; x=1670979579; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=z/ErRUKlY7D4FY0txm0LkQ3s6d6OybLiabjK2vRaL2s=; b=WSM2EVpoJwrq+2tTNHawm8W58Q70/wKv1a6tMfsQCmMngorqJcwwy605 nIkgZH7fbiUQxQ6v+p9eeW6lCPWBL4BmLpPrdWLzA75h01hcV6mLxoWi8 ixcMfF/IeY5miZ91cv5fowJozV8rdFjKAe66xVYflLl3gOZIjV1w2Md0D o1tE1ly8TPWNXbc+6MTAyO1t3QajCwEzz3TUyLIu/Kpf6NcGgYecyiwvs RijsXJAwKGFZuFJZaXwfmk0A4t9zhzVger5WvL60rppqn76qur7WcefSm jxDEi3H4OM6POh/F793sgttuWs/8sha6G5LlEYa0jjf1ZPfMEgaNeqceg A==; X-IronPort-AV: E=McAfee;i="6200,9189,10197"; a="325139619" X-IronPort-AV: E=Sophos;i="5.88,203,1635231600"; d="scan'208";a="325139619" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Dec 2021 16:59:39 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,203,1635231600"; d="scan'208";a="505148577" Received: from chang-linux-3.sc.intel.com ([172.25.66.175]) by orsmga007.jf.intel.com with ESMTP; 13 Dec 2021 16:59:38 -0800 From: "Chang S. Bae" To: tglx@linutronix.de, bp@suse.de, dave.hansen@linux.intel.com, mingo@kernel.org, luto@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, ebiggers@kernel.org, dan.j.williams@intel.com, charishma1.gairuboyina@intel.com, kumar.n.dwarakanath@intel.com, lalithambika.krishnakumar@intel.com, ravi.v.shankar@intel.com, chang.seok.bae@intel.com Subject: [PATCH v4 04/13] x86/asm: Add a wrapper function for the LOADIWKEY instruction Date: Mon, 13 Dec 2021 16:52:03 -0800 Message-Id: <20211214005212.20588-5-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20211214005212.20588-1-chang.seok.bae@intel.com> References: <20211214005212.20588-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Key Locker introduces a CPU-internal wrapping key to encode a user key to a key handle. Then a key handle is referenced instead of the plain text key. The new instruction loads an internal wrapping key in the software-inaccessible CPU state. It operates only in kernel mode. Define struct iwkey to pass the key value. The kernel will use this function to load a new key at boot time when the feature is enabled. Signed-off-by: Chang S. Bae Reviewed-by: Dan Williams Cc: x86@kernel.org Cc: linux-kernel@vger.kernel.org --- Changes from RFC v2: * Separate out the code as a new patch. * Improve the usability with the new struct as an argument. (Dan Williams) Note, Dan wondered if: WARN_ON(!irq_fpu_usable()); would be appropriate in the load_xmm_iwkey() function. --- arch/x86/include/asm/keylocker.h | 25 ++++++++++++++++++++++ arch/x86/include/asm/special_insns.h | 32 ++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+) create mode 100644 arch/x86/include/asm/keylocker.h diff --git a/arch/x86/include/asm/keylocker.h b/arch/x86/include/asm/keylocker.h new file mode 100644 index 000000000000..df84c83228a1 --- /dev/null +++ b/arch/x86/include/asm/keylocker.h @@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +#ifndef _ASM_KEYLOCKER_H +#define _ASM_KEYLOCKER_H + +#ifndef __ASSEMBLY__ + +#include + +/** + * struct iwkey - A temporary internal wrapping key storage. + * @integrity_key: A 128-bit key to check that key handles have not + * been tampered with. + * @encryption_key: A 256-bit encryption key used in + * wrapping/unwrapping a clear text key. + * + * This storage should be flushed immediately after loaded. + */ +struct iwkey { + struct reg_128_bit integrity_key; + struct reg_128_bit encryption_key[2]; +}; + +#endif /*__ASSEMBLY__ */ +#endif /* _ASM_KEYLOCKER_H */ diff --git a/arch/x86/include/asm/special_insns.h b/arch/x86/include/asm/special_insns.h index 68c257a3de0d..3e50d8396ee1 100644 --- a/arch/x86/include/asm/special_insns.h +++ b/arch/x86/include/asm/special_insns.h @@ -9,6 +9,7 @@ #include #include #include +#include /* * The compiler should not reorder volatile asm statements with respect to each @@ -294,6 +295,37 @@ static inline int enqcmds(void __iomem *dst, const void *src) return 0; } +/** + * load_xmm_iwkey - Load a CPU-internal wrapping key + * @key: A struct iwkey pointer. + * + * Load @key to XMMs then do LOADIWKEY. After this, flush XMM + * registers. Caller is responsible for kernel_cpu_begin(). + */ +static inline void load_xmm_iwkey(struct iwkey *key) +{ + struct reg_128_bit zeros = { 0 }; + + asm volatile ("movdqu %0, %%xmm0; movdqu %1, %%xmm1; movdqu %2, %%xmm2;" + :: "m"(key->integrity_key), "m"(key->encryption_key[0]), + "m"(key->encryption_key[1])); + + /* + * LOADIWKEY %xmm1,%xmm2 + * + * EAX and XMM0 are implicit operands. Load a key value + * from XMM0-2 to a software-invisible CPU state. With zero + * in EAX, CPU does not do hardware randomization and the key + * backup is allowed. + * + * This instruction is supported by binutils >= 2.36. + */ + asm volatile (".byte 0xf3,0x0f,0x38,0xdc,0xd1" :: "a"(0)); + + asm volatile ("movdqu %0, %%xmm0; movdqu %0, %%xmm1; movdqu %0, %%xmm2;" + :: "m"(zeros)); +} + #endif /* __KERNEL__ */ #endif /* _ASM_X86_SPECIAL_INSNS_H */ From patchwork Tue Dec 14 00:52:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 524353 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E4B2C4332F for ; Tue, 14 Dec 2021 00:59:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244528AbhLNA7n (ORCPT ); Mon, 13 Dec 2021 19:59:43 -0500 Received: from mga05.intel.com ([192.55.52.43]:21411 "EHLO mga05.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244571AbhLNA7k (ORCPT ); Mon, 13 Dec 2021 19:59:40 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1639443580; x=1670979580; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=IzmmcNwPyrly7VAToU85hx2yPsdSXI9+7O903lBKZz8=; b=kkF158Gw7e7fSw6gjQEZEWibEY3B7FZY3rKHOt/COyHaExFE8Cxppi0Q 9EzEH1TSSnGztJXe14K9gX1W0u8dsGHdJ7DST3X1V6bRztX8rpIVkjy5c yeDqtnkkLdWdszimFJo5RCAiPK9oFVFSV4t/rGPnU3PJufv6161sLmBIb M+9coupZ2vtp8eahZNX2KdvIsasBv5CFiUUkTklI65f4xDsOzqEF+bjbc +brUTFwUaERZIhL0mhS6o2N3jHoIRsSOJmT+onUGJsPLc2ZWKZsFtIx3D rMFHVvVuP2CAB0A5knTCUEcq15gRQeRb8kqZlox4/fJVXKAb1QuzGod1P Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10197"; a="325139622" X-IronPort-AV: E=Sophos;i="5.88,203,1635231600"; d="scan'208";a="325139622" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Dec 2021 16:59:39 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,203,1635231600"; d="scan'208";a="505148581" Received: from chang-linux-3.sc.intel.com ([172.25.66.175]) by orsmga007.jf.intel.com with ESMTP; 13 Dec 2021 16:59:39 -0800 From: "Chang S. Bae" To: tglx@linutronix.de, bp@suse.de, dave.hansen@linux.intel.com, mingo@kernel.org, luto@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, ebiggers@kernel.org, dan.j.williams@intel.com, charishma1.gairuboyina@intel.com, kumar.n.dwarakanath@intel.com, lalithambika.krishnakumar@intel.com, ravi.v.shankar@intel.com, chang.seok.bae@intel.com Subject: [PATCH v4 05/13] x86/msr-index: Add MSRs for Key Locker internal wrapping key Date: Mon, 13 Dec 2021 16:52:04 -0800 Message-Id: <20211214005212.20588-6-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20211214005212.20588-1-chang.seok.bae@intel.com> References: <20211214005212.20588-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org The CPU state that contains the internal wrapping key is in the same power domain as the cache. So any sleep state that would invalidate the cache (like S3) also invalidates the state of the wrapping key. But, since the state is inaccessible to software, it needs a special mechanism to save and restore the key during deep sleep. A set of new MSRs are provided as an abstract interface to save and restore the wrapping key, and to check the key status. The wrapping key is saved in a platform-scoped state of non-volatile media. The backup itself and its path from the CPU are encrypted and integrity protected. But this storage's non-volatility is not architecturally guaranteed across off states, such as S5 and G3. The MSRs will be used to back up the key for S3/4 sleep states. Then the kernel code writes one of them to restore the key in each CPU state. Signed-off-by: Chang S. Bae Reviewed-by: Dan Williams Cc: x86@kernel.org Cc: linux-kernel@vger.kernel.org --- Changes from RFC v2: * Update the changelog. (Dan Williams) * Rename the MSRs. (Dan Williams) --- arch/x86/include/asm/msr-index.h | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h index 01e2650b9585..7f11a3b3a75b 100644 --- a/arch/x86/include/asm/msr-index.h +++ b/arch/x86/include/asm/msr-index.h @@ -941,4 +941,10 @@ #define MSR_VM_IGNNE 0xc0010115 #define MSR_VM_HSAVE_PA 0xc0010117 +/* MSRs for managing an internal wrapping key for Key Locker. */ +#define MSR_IA32_IWKEY_COPY_STATUS 0x00000990 +#define MSR_IA32_IWKEY_BACKUP_STATUS 0x00000991 +#define MSR_IA32_BACKUP_IWKEY_TO_PLATFORM 0x00000d91 +#define MSR_IA32_COPY_IWKEY_TO_LOCAL 0x00000d92 + #endif /* _ASM_X86_MSR_INDEX_H */ From patchwork Tue Dec 14 00:52:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 524352 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01522C433FE for ; Tue, 14 Dec 2021 01:00:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244624AbhLNBAB (ORCPT ); Mon, 13 Dec 2021 20:00:01 -0500 Received: from mga05.intel.com ([192.55.52.43]:21431 "EHLO mga05.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244571AbhLNBAA (ORCPT ); Mon, 13 Dec 2021 20:00:00 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1639443600; x=1670979600; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=fW8vhlFreabo0r9odugP6EelsYg2vkpR+vxEaDYnIss=; b=DBfWnaxeOIhBYVS/URvNeXFW30EFlHkEAUskXYP1WH1tQ+tAVOLJP5ja zoc+zDsICEocxFToqjSwhpSppg0KAFbjGd+fUxf9UROlP8afi2OttcvZM 4FOjDQAkkkkzcTG/63wv2cZefbxULPbVyVwk6EZWAwjBr6mNvp0utxN5d C3d5bLQUV8lmT0kcWSlJHqqCrLK0cjjPsIEMenamSgM3eZjMEct+RF6kd NNcm6ipS7/P1zZOww81m7K/RodDWgGQLmHXOymjfVXICgvfP5Jr1VuZkX aBTdoXhkEwqzC09Ti43qwgCha2xCzbogVA2C0+PCF0AMlBQhPgq7fEIX2 Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10197"; a="325139632" X-IronPort-AV: E=Sophos;i="5.88,203,1635231600"; d="scan'208";a="325139632" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Dec 2021 16:59:40 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,203,1635231600"; d="scan'208";a="505148596" Received: from chang-linux-3.sc.intel.com ([172.25.66.175]) by orsmga007.jf.intel.com with ESMTP; 13 Dec 2021 16:59:40 -0800 From: "Chang S. Bae" To: tglx@linutronix.de, bp@suse.de, dave.hansen@linux.intel.com, mingo@kernel.org, luto@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, ebiggers@kernel.org, dan.j.williams@intel.com, charishma1.gairuboyina@intel.com, kumar.n.dwarakanath@intel.com, lalithambika.krishnakumar@intel.com, ravi.v.shankar@intel.com, chang.seok.bae@intel.com, linux-pm@vger.kernel.org Subject: [PATCH v4 08/13] x86/power/keylocker: Restore internal wrapping key from the ACPI S3/4 sleep states Date: Mon, 13 Dec 2021 16:52:07 -0800 Message-Id: <20211214005212.20588-9-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20211214005212.20588-1-chang.seok.bae@intel.com> References: <20211214005212.20588-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org When the system state switches to these sleep states, the internal wrapping key gets reset in the CPU state. The primary use case for the feature is bare metal dm-crypt. The key needs to be restored properly on wakeup, as dm-crypt does not prompt for the key on resume from suspend. Even the prompt it does perform for unlocking the volume where the hibernation image is stored, it still expects to reuse the key handles within the hibernation image once it is loaded. So it is motivated to meet dm-crypt's expectation that the key handles in the suspend-image remain valid after resume from an S-state. Key Locker provides a mechanism to back up the internal wrapping key in non-volatile storage. The kernel requests a backup right after the key is loaded at boot time. It is copied back to each CPU upon wakeup. While the backup may be maintained in NVM across S5 and G3 "off" states it is not architecturally guaranteed, nor is it expected by dm-crypt which expects to prompt for the key each time the volume is started. The entirety of Key Locker needs to be disabled if the backup mechanism is not available unless CONFIG_SUSPEND=n, otherwise dm-crypt requires the backup to be available. In the event of a key restore failure the kernel proceeds with an initialized IWKey state. This has the effect of invalidating any key handles that might be present in a suspend-image. When this happens dm-crypt will see I/O errors resulting from error returns from crypto_skcipher_{en,de}crypt(). While this will disrupt operations in the current boot, data is not at risk and access is restored at the next reboot to create new handles relative to the current IWKey. Manage a feature-specific flag to communicate with the crypto implementation. This ensures to stop using the AES instructions upon the key restore failure while not turning off the feature. Signed-off-by: Chang S. Bae Reviewed-by: Dan Williams Cc: x86@kernel.org Cc: linux-kernel@vger.kernel.org Cc: linux-pm@vger.kernel.org --- Changes from v3: * Fix the build issue with !X86_KEYLOCKER. (Eric Biggers) Changes from RFC v2: * Change the backup key failure handling. (Dan Williams) Changes from RFC v1: * Folded the warning message into the if condition check. (Rafael Wysocki) * Rebased on the changes of the previous patches. * Added error code for key restoration failures. * Moved the restore helper. * Added function descriptions. --- arch/x86/include/asm/keylocker.h | 4 + arch/x86/kernel/keylocker.c | 124 ++++++++++++++++++++++++++++++- arch/x86/power/cpu.c | 2 + 3 files changed, 128 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/keylocker.h b/arch/x86/include/asm/keylocker.h index 820ac29c06d9..c1d27fb5a1c3 100644 --- a/arch/x86/include/asm/keylocker.h +++ b/arch/x86/include/asm/keylocker.h @@ -32,9 +32,13 @@ struct iwkey { #ifdef CONFIG_X86_KEYLOCKER void setup_keylocker(struct cpuinfo_x86 *c); void destroy_keylocker_data(void); +void restore_keylocker(void); +extern bool valid_keylocker(void); #else #define setup_keylocker(c) do { } while (0) #define destroy_keylocker_data() do { } while (0) +#define restore_keylocker() do { } while (0) +static inline bool valid_keylocker(void) { return false; } #endif #endif /*__ASSEMBLY__ */ diff --git a/arch/x86/kernel/keylocker.c b/arch/x86/kernel/keylocker.c index 87d775a65716..ff0e012e3dd5 100644 --- a/arch/x86/kernel/keylocker.c +++ b/arch/x86/kernel/keylocker.c @@ -11,11 +11,26 @@ #include #include #include +#include static __initdata struct keylocker_setup_data { + bool initialized; struct iwkey key; } kl_setup; +/* + * This flag is set with IWKey load. When the key restore fails, it is + * reset. This restore state is exported to the crypto library, then AES-KL + * will not be used there. So, the feature is soft-disabled with this flag. + */ +static bool valid_kl; + +bool valid_keylocker(void) +{ + return valid_kl; +} +EXPORT_SYMBOL_GPL(valid_keylocker); + static void __init generate_keylocker_data(void) { get_random_bytes(&kl_setup.key.integrity_key, sizeof(kl_setup.key.integrity_key)); @@ -25,6 +40,8 @@ static void __init generate_keylocker_data(void) void __init destroy_keylocker_data(void) { memset(&kl_setup.key, KEY_DESTROY, sizeof(kl_setup.key)); + kl_setup.initialized = true; + valid_kl = true; } static void __init load_keylocker(void) @@ -34,6 +51,27 @@ static void __init load_keylocker(void) kernel_fpu_end(); } +/** + * copy_keylocker - Copy the internal wrapping key from the backup. + * + * Request hardware to copy the key in non-volatile storage to the CPU + * state. + * + * Returns: -EBUSY if the copy fails, 0 if successful. + */ +static int copy_keylocker(void) +{ + u64 status; + + wrmsrl(MSR_IA32_COPY_IWKEY_TO_LOCAL, 1); + + rdmsrl(MSR_IA32_IWKEY_COPY_STATUS, status); + if (status & BIT(0)) + return 0; + else + return -EBUSY; +} + /** * setup_keylocker - Enable the feature. * @c: A pointer to struct cpuinfo_x86 @@ -49,6 +87,7 @@ void __ref setup_keylocker(struct cpuinfo_x86 *c) if (c == &boot_cpu_data) { u32 eax, ebx, ecx, edx; + bool backup_available; cpuid_count(KEYLOCKER_CPUID, 0, &eax, &ebx, &ecx, &edx); /* @@ -62,10 +101,49 @@ void __ref setup_keylocker(struct cpuinfo_x86 *c) goto disable; } + backup_available = (ebx & KEYLOCKER_CPUID_EBX_BACKUP) ? true : false; + /* + * The internal wrapping key in CPU state is volatile in + * S3/4 states. So ensure the backup capability along with + * S-states. + */ + if (!backup_available && IS_ENABLED(CONFIG_SUSPEND)) { + pr_debug("x86/keylocker: No key backup support with possible S3/4.\n"); + goto disable; + } + generate_keylocker_data(); - } + load_keylocker(); - load_keylocker(); + /* Backup an internal wrapping key in non-volatile media. */ + if (backup_available) + wrmsrl(MSR_IA32_BACKUP_IWKEY_TO_PLATFORM, 1); + } else { + int rc; + + /* + * Load the internal wrapping key directly when available + * in memory, which is only possible at boot-time. + * + * NB: When system wakes up, this path also recovers the + * internal wrapping key. + */ + if (!kl_setup.initialized) { + load_keylocker(); + } else if (valid_kl) { + rc = copy_keylocker(); + /* + * The boot CPU was successful but the key copy + * fails here. Then, the subsequent feature use + * will have inconsistent keys and failures. So, + * invalidate the feature via the flag. + */ + if (rc) { + valid_kl = false; + pr_err_once("x86/keylocker: Invalid copy status (rc: %d).\n", rc); + } + } + } pr_info_once("x86/keylocker: Enabled.\n"); return; @@ -77,3 +155,45 @@ void __ref setup_keylocker(struct cpuinfo_x86 *c) /* Make sure the feature disabled for kexec-reboot. */ cr4_clear_bits(X86_CR4_KEYLOCKER); } + +/** + * restore_keylocker - Restore the internal wrapping key. + * + * The boot CPU executes this while other CPUs restore it through the setup + * function. + */ +void restore_keylocker(void) +{ + u64 backup_status; + int rc; + + if (!cpu_feature_enabled(X86_FEATURE_KEYLOCKER) || !valid_kl) + return; + + /* + * The IA32_IWKEYBACKUP_STATUS MSR contains a bitmap that indicates + * an invalid backup if bit 0 is set and a read (or write) error if + * bit 2 is set. + */ + rdmsrl(MSR_IA32_IWKEY_BACKUP_STATUS, backup_status); + if (backup_status & BIT(0)) { + rc = copy_keylocker(); + if (rc) + pr_err("x86/keylocker: Invalid copy state (rc: %d).\n", rc); + else + return; + } else { + pr_err("x86/keylocker: The key backup access failed with %s.\n", + (backup_status & BIT(2)) ? "read error" : "invalid status"); + } + + /* + * Now the backup key is not available. Invalidate the feature via + * the flag to avoid any subsequent use. But keep the feature with + * zero IWKeys instead of disabling it. The current users will see + * key handle integrity failure but that's because of the internal + * key change. + */ + pr_err("x86/keylocker: Failed to restore internal wrapping key.\n"); + valid_kl = false; +} diff --git a/arch/x86/power/cpu.c b/arch/x86/power/cpu.c index 9f2b251e83c5..1a290f529c73 100644 --- a/arch/x86/power/cpu.c +++ b/arch/x86/power/cpu.c @@ -25,6 +25,7 @@ #include #include #include +#include #ifdef CONFIG_X86_32 __visible unsigned long saved_context_ebx; @@ -262,6 +263,7 @@ static void notrace __restore_processor_state(struct saved_context *ctxt) mtrr_bp_restore(); perf_restore_debug_store(); msr_restore_context(ctxt); + restore_keylocker(); c = &cpu_data(smp_processor_id()); if (cpu_has(c, X86_FEATURE_MSR_IA32_FEAT_CTL)) From patchwork Tue Dec 14 00:52:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 524351 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A52E9C43217 for ; Tue, 14 Dec 2021 01:00:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244644AbhLNBAF (ORCPT ); Mon, 13 Dec 2021 20:00:05 -0500 Received: from mga05.intel.com ([192.55.52.43]:21433 "EHLO mga05.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244627AbhLNBAB (ORCPT ); Mon, 13 Dec 2021 20:00:01 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1639443601; x=1670979601; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=RwjvKKHQ02rvrTTx+OGrcLBSlByDLmeRBT+6TGEQEAo=; b=WzkazXKGrjKZah93Gt4FpnNHGuoQaAy8JhF5T2yiRLCu/TZ9WZLLoBW2 dG34DkeOCUXLF5osodPIBNSbLWdSwqm2SyDCzIOF3ooWaj7YxTttdLGyc F/M2OYgUB0SaWwp0jskZJ6Gx4OBoRNZ31L0fku8O8BoBw0En4V6RjLADR ZguJ3TVSSox0GJxwqMj97Nb91+0sNsJlraSEgL73JiPAF00ptltGZEO3H sc60GEzPYB8VCAwYEdI0Rs0UPVcW+6qi86y1XBusjNyyHT2sLOAETAk5C wOD+/VY299FyodTbn+CjstKqFol6yZLiXRYg3FSrGfdmD5eTk4tJnuuds Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10197"; a="325139638" X-IronPort-AV: E=Sophos;i="5.88,203,1635231600"; d="scan'208";a="325139638" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Dec 2021 16:59:41 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,203,1635231600"; d="scan'208";a="505148613" Received: from chang-linux-3.sc.intel.com ([172.25.66.175]) by orsmga007.jf.intel.com with ESMTP; 13 Dec 2021 16:59:41 -0800 From: "Chang S. Bae" To: tglx@linutronix.de, bp@suse.de, dave.hansen@linux.intel.com, mingo@kernel.org, luto@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, ebiggers@kernel.org, dan.j.williams@intel.com, charishma1.gairuboyina@intel.com, kumar.n.dwarakanath@intel.com, lalithambika.krishnakumar@intel.com, ravi.v.shankar@intel.com, chang.seok.bae@intel.com, Ard Biesheuvel Subject: [PATCH v4 10/13] crypto: x86/aes - Prepare for a new AES implementation Date: Mon, 13 Dec 2021 16:52:09 -0800 Message-Id: <20211214005212.20588-11-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20211214005212.20588-1-chang.seok.bae@intel.com> References: <20211214005212.20588-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Key Locker's AES instruction set ('AES-KL') has a similar programming interface to AES-NI. The internal ABI in the assembly code will have the same prototype as AES-NI. Then, the glue code will be the same as AES-NI's. Refactor the common C code to avoid code duplication. The AES-NI code uses it with a function pointer argument to call back the AES-NI assembly code. So will the AES-KL code. Introduce wrappers for data transformation functions to return an error code. AES-KL may populate an error during data transformation. Export those refactored functions and new wrappers that the AES-KL code will reference. Also move some constant values to a common assembly code. No functional change intended. Signed-off-by: Chang S. Bae Acked-by: Dan Williams Cc: Ard Biesheuvel Cc: x86@kernel.org Cc: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org --- Changes from v3: * Drop ECB and CTR mode changes. (Eric Biggers) * Export symbols. (Eric Biggers) Changes from RFC v2: * Massage the changelog. (Dan Williams) Changes from RFC v1: * Added as a new patch. (Ard Biesheuvel) Link: https://lore.kernel.org/lkml/CAMj1kXGa4f21eH0mdxd1pQsZMUjUr1Btq+Dgw-gC=O-yYft7xw@mail.gmail.com/ --- arch/x86/crypto/Makefile | 2 +- arch/x86/crypto/aes-intel_asm.S | 26 +++ arch/x86/crypto/aes-intel_glue.c | 153 +++++++++++++++ arch/x86/crypto/aes-intel_glue.h | 48 +++++ arch/x86/crypto/aesni-intel_asm.S | 74 +++---- arch/x86/crypto/aesni-intel_glue.c | 301 +++++++++-------------------- arch/x86/crypto/aesni-intel_glue.h | 22 +++ 7 files changed, 372 insertions(+), 254 deletions(-) create mode 100644 arch/x86/crypto/aes-intel_asm.S create mode 100644 arch/x86/crypto/aes-intel_glue.c create mode 100644 arch/x86/crypto/aes-intel_glue.h create mode 100644 arch/x86/crypto/aesni-intel_glue.h diff --git a/arch/x86/crypto/Makefile b/arch/x86/crypto/Makefile index f307c93fc90a..ef6c0b9f69c6 100644 --- a/arch/x86/crypto/Makefile +++ b/arch/x86/crypto/Makefile @@ -47,7 +47,7 @@ chacha-x86_64-y := chacha-avx2-x86_64.o chacha-ssse3-x86_64.o chacha_glue.o chacha-x86_64-$(CONFIG_AS_AVX512) += chacha-avx512vl-x86_64.o obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o -aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o +aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o aes-intel_glue.o aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o aes_ctrby8_avx-x86_64.o obj-$(CONFIG_CRYPTO_SHA1_SSSE3) += sha1-ssse3.o diff --git a/arch/x86/crypto/aes-intel_asm.S b/arch/x86/crypto/aes-intel_asm.S new file mode 100644 index 000000000000..98abf875af79 --- /dev/null +++ b/arch/x86/crypto/aes-intel_asm.S @@ -0,0 +1,26 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ + +/* + * Constant values shared between AES implementations: + */ + +.pushsection .rodata +.align 16 +.Lcts_permute_table: + .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 + .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 + .byte 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07 + .byte 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f + .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 + .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 +#ifdef __x86_64__ +.Lbswap_mask: + .byte 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 +#endif +.popsection + +.section .rodata.cst16.gf128mul_x_ble_mask, "aM", @progbits, 16 +.align 16 +.Lgf128mul_x_ble_mask: + .octa 0x00000000000000010000000000000087 +.previous diff --git a/arch/x86/crypto/aes-intel_glue.c b/arch/x86/crypto/aes-intel_glue.c new file mode 100644 index 000000000000..e7f90897653d --- /dev/null +++ b/arch/x86/crypto/aes-intel_glue.c @@ -0,0 +1,153 @@ +// SPDX-License-Identifier: GPL-2.0-or-later + +#include +#include +#include +#include +#include +#include +#include +#include "aes-intel_glue.h" + +int cbc_crypt_common(struct skcipher_request *req, + int (*fn)(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv)) +{ + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); + struct crypto_aes_ctx *ctx = aes_ctx(crypto_skcipher_ctx(tfm)); + struct skcipher_walk walk; + unsigned int nbytes; + int err; + + err = skcipher_walk_virt(&walk, req, false); + + while ((nbytes = walk.nbytes)) { + kernel_fpu_begin(); + err = fn(ctx, walk.dst.virt.addr, walk.src.virt.addr, nbytes & AES_BLOCK_MASK, + walk.iv); + kernel_fpu_end(); + if (err) + return err; + + nbytes &= AES_BLOCK_SIZE - 1; + err = skcipher_walk_done(&walk, nbytes); + } + + return err; +} +EXPORT_SYMBOL_GPL(cbc_crypt_common); + +int xts_setkey_common(struct crypto_skcipher *tfm, const u8 *key, unsigned int keylen, + int (*fn)(struct crypto_tfm *tfm, void *raw_ctx, + const u8 *in_key, unsigned int key_len)) +{ + struct aes_xts_ctx *ctx = crypto_skcipher_ctx(tfm); + int err; + + err = xts_verify_key(tfm, key, keylen); + if (err) + return err; + + keylen /= 2; + + /* first half of xts-key is for crypt */ + err = fn(crypto_skcipher_tfm(tfm), ctx->raw_crypt_ctx, key, keylen); + if (err) + return err; + + /* second half of xts-key is for tweak */ + return fn(crypto_skcipher_tfm(tfm), ctx->raw_tweak_ctx, key + keylen, keylen); +} +EXPORT_SYMBOL_GPL(xts_setkey_common); + +int xts_crypt_common(struct skcipher_request *req, + int (*crypt_fn)(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv), + int (*crypt1_fn)(const void *ctx, u8 *out, const u8 *in)) +{ + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); + struct aes_xts_ctx *ctx = crypto_skcipher_ctx(tfm); + int tail = req->cryptlen % AES_BLOCK_SIZE; + struct skcipher_request subreq; + struct skcipher_walk walk; + int err; + + if (req->cryptlen < AES_BLOCK_SIZE) + return -EINVAL; + + err = skcipher_walk_virt(&walk, req, false); + if (!walk.nbytes) + return err; + + if (unlikely(tail > 0 && walk.nbytes < walk.total)) { + int blocks = DIV_ROUND_UP(req->cryptlen, AES_BLOCK_SIZE) - 2; + + skcipher_walk_abort(&walk); + + skcipher_request_set_tfm(&subreq, tfm); + skcipher_request_set_callback(&subreq, + skcipher_request_flags(req), + NULL, NULL); + skcipher_request_set_crypt(&subreq, req->src, req->dst, + blocks * AES_BLOCK_SIZE, req->iv); + req = &subreq; + + err = skcipher_walk_virt(&walk, req, false); + if (!walk.nbytes) + return err; + } else { + tail = 0; + } + + kernel_fpu_begin(); + + /* calculate first value of T */ + err = crypt1_fn(aes_ctx(ctx->raw_tweak_ctx), walk.iv, walk.iv); + if (err) + return err; + + while (walk.nbytes > 0) { + int nbytes = walk.nbytes; + + if (nbytes < walk.total) + nbytes &= ~(AES_BLOCK_SIZE - 1); + + err = crypt_fn(aes_ctx(ctx->raw_crypt_ctx), walk.dst.virt.addr, walk.src.virt.addr, + nbytes, walk.iv); + kernel_fpu_end(); + if (err) + return err; + + err = skcipher_walk_done(&walk, walk.nbytes - nbytes); + + if (walk.nbytes > 0) + kernel_fpu_begin(); + } + + if (unlikely(tail > 0 && !err)) { + struct scatterlist sg_src[2], sg_dst[2]; + struct scatterlist *src, *dst; + + dst = src = scatterwalk_ffwd(sg_src, req->src, req->cryptlen); + if (req->dst != req->src) + dst = scatterwalk_ffwd(sg_dst, req->dst, req->cryptlen); + + skcipher_request_set_crypt(req, src, dst, AES_BLOCK_SIZE + tail, + req->iv); + + err = skcipher_walk_virt(&walk, &subreq, false); + if (err) + return err; + + kernel_fpu_begin(); + err = crypt_fn(aes_ctx(ctx->raw_crypt_ctx), walk.dst.virt.addr, walk.src.virt.addr, + walk.nbytes, walk.iv); + kernel_fpu_end(); + if (err) + return err; + + err = skcipher_walk_done(&walk, 0); + } + return err; +} +EXPORT_SYMBOL_GPL(xts_crypt_common); diff --git a/arch/x86/crypto/aes-intel_glue.h b/arch/x86/crypto/aes-intel_glue.h new file mode 100644 index 000000000000..c8bd783fa787 --- /dev/null +++ b/arch/x86/crypto/aes-intel_glue.h @@ -0,0 +1,48 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* + * Shared glue code between AES implementations, refactored from the AES-NI's. + */ + +#ifndef _AES_INTEL_GLUE_H +#define _AES_INTEL_GLUE_H + +#include + +#define AES_ALIGN 16 +#define AES_ALIGN_ATTR __attribute__((__aligned__(AES_ALIGN))) +#define AES_BLOCK_MASK (~(AES_BLOCK_SIZE - 1)) +#define AES_ALIGN_EXTRA ((AES_ALIGN - 1) & ~(CRYPTO_MINALIGN - 1)) +#define CRYPTO_AES_CTX_SIZE (sizeof(struct crypto_aes_ctx) + AES_ALIGN_EXTRA) +#define XTS_AES_CTX_SIZE (sizeof(struct aes_xts_ctx) + AES_ALIGN_EXTRA) + +struct aes_xts_ctx { + u8 raw_tweak_ctx[sizeof(struct crypto_aes_ctx)] AES_ALIGN_ATTR; + u8 raw_crypt_ctx[sizeof(struct crypto_aes_ctx)] AES_ALIGN_ATTR; +}; + +static inline struct crypto_aes_ctx *aes_ctx(void *raw_ctx) +{ + unsigned long addr = (unsigned long)raw_ctx; + unsigned long align = AES_ALIGN; + + if (align <= crypto_tfm_ctx_alignment()) + align = 1; + + return (struct crypto_aes_ctx *)ALIGN(addr, align); +} + +int cbc_crypt_common(struct skcipher_request *req, + int (*fn)(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv)); + +int xts_setkey_common(struct crypto_skcipher *tfm, const u8 *key, unsigned int keylen, + int (*fn)(struct crypto_tfm *tfm, void *raw_ctx, + const u8 *in_key, unsigned int key_len)); + +int xts_crypt_common(struct skcipher_request *req, + int (*crypt_fn)(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv), + int (*crypt1_fn)(const void *ctx, u8 *out, const u8 *in)); + +#endif /* _AES_INTEL_GLUE_H */ diff --git a/arch/x86/crypto/aesni-intel_asm.S b/arch/x86/crypto/aesni-intel_asm.S index 4e3972570916..695093a3c4aa 100644 --- a/arch/x86/crypto/aesni-intel_asm.S +++ b/arch/x86/crypto/aesni-intel_asm.S @@ -28,6 +28,7 @@ #include #include #include +#include "aes-intel_asm.S" /* * The following macros are used to move an (un)aligned 16 byte value to/from @@ -1822,10 +1823,10 @@ SYM_FUNC_START_LOCAL(_key_expansion_256b) SYM_FUNC_END(_key_expansion_256b) /* - * int aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key, - * unsigned int key_len) + * int _aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key, + * unsigned int key_len) */ -SYM_FUNC_START(aesni_set_key) +SYM_FUNC_START(_aesni_set_key) FRAME_BEGIN #ifndef __x86_64__ pushl KEYP @@ -1934,12 +1935,12 @@ SYM_FUNC_START(aesni_set_key) #endif FRAME_END ret -SYM_FUNC_END(aesni_set_key) +SYM_FUNC_END(_aesni_set_key) /* - * void aesni_enc(const void *ctx, u8 *dst, const u8 *src) + * void _aesni_enc(const void *ctx, u8 *dst, const u8 *src) */ -SYM_FUNC_START(aesni_enc) +SYM_FUNC_START(_aesni_enc) FRAME_BEGIN #ifndef __x86_64__ pushl KEYP @@ -1958,7 +1959,7 @@ SYM_FUNC_START(aesni_enc) #endif FRAME_END ret -SYM_FUNC_END(aesni_enc) +SYM_FUNC_END(_aesni_enc) /* * _aesni_enc1: internal ABI @@ -2126,9 +2127,9 @@ SYM_FUNC_START_LOCAL(_aesni_enc4) SYM_FUNC_END(_aesni_enc4) /* - * void aesni_dec (const void *ctx, u8 *dst, const u8 *src) + * void _aesni_dec (const void *ctx, u8 *dst, const u8 *src) */ -SYM_FUNC_START(aesni_dec) +SYM_FUNC_START(_aesni_dec) FRAME_BEGIN #ifndef __x86_64__ pushl KEYP @@ -2148,7 +2149,7 @@ SYM_FUNC_START(aesni_dec) #endif FRAME_END ret -SYM_FUNC_END(aesni_dec) +SYM_FUNC_END(_aesni_dec) /* * _aesni_dec1: internal ABI @@ -2437,10 +2438,10 @@ SYM_FUNC_START(aesni_ecb_dec) SYM_FUNC_END(aesni_ecb_dec) /* - * void aesni_cbc_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, - * size_t len, u8 *iv) + * void _aesni_cbc_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, + * size_t len, u8 *iv) */ -SYM_FUNC_START(aesni_cbc_enc) +SYM_FUNC_START(_aesni_cbc_enc) FRAME_BEGIN #ifndef __x86_64__ pushl IVP @@ -2478,13 +2479,13 @@ SYM_FUNC_START(aesni_cbc_enc) #endif FRAME_END ret -SYM_FUNC_END(aesni_cbc_enc) +SYM_FUNC_END(_aesni_cbc_enc) /* - * void aesni_cbc_dec(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, - * size_t len, u8 *iv) + * void _aesni_cbc_dec(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, + * size_t len, u8 *iv) */ -SYM_FUNC_START(aesni_cbc_dec) +SYM_FUNC_START(_aesni_cbc_dec) FRAME_BEGIN #ifndef __x86_64__ pushl IVP @@ -2571,7 +2572,7 @@ SYM_FUNC_START(aesni_cbc_dec) #endif FRAME_END ret -SYM_FUNC_END(aesni_cbc_dec) +SYM_FUNC_END(_aesni_cbc_dec) /* * void aesni_cts_cbc_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, @@ -2691,21 +2692,6 @@ SYM_FUNC_START(aesni_cts_cbc_dec) ret SYM_FUNC_END(aesni_cts_cbc_dec) -.pushsection .rodata -.align 16 -.Lcts_permute_table: - .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 - .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 - .byte 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07 - .byte 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f - .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 - .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 -#ifdef __x86_64__ -.Lbswap_mask: - .byte 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 -#endif -.popsection - #ifdef __x86_64__ /* * _aesni_inc_init: internal ABI @@ -2821,12 +2807,6 @@ SYM_FUNC_END(aesni_ctr_enc) #endif -.section .rodata.cst16.gf128mul_x_ble_mask, "aM", @progbits, 16 -.align 16 -.Lgf128mul_x_ble_mask: - .octa 0x00000000000000010000000000000087 -.previous - /* * _aesni_gf128mul_x_ble: internal ABI * Multiply in GF(2^128) for XTS IVs @@ -2846,10 +2826,10 @@ SYM_FUNC_END(aesni_ctr_enc) pxor KEY, IV; /* - * void aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *dst, - * const u8 *src, unsigned int len, le128 *iv) + * void _aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *dst, + * const u8 *src, unsigned int len, le128 *iv) */ -SYM_FUNC_START(aesni_xts_encrypt) +SYM_FUNC_START(_aesni_xts_encrypt) FRAME_BEGIN #ifndef __x86_64__ pushl IVP @@ -2998,13 +2978,13 @@ SYM_FUNC_START(aesni_xts_encrypt) movups STATE, (OUTP) jmp .Lxts_enc_ret -SYM_FUNC_END(aesni_xts_encrypt) +SYM_FUNC_END(_aesni_xts_encrypt) /* - * void aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *dst, - * const u8 *src, unsigned int len, le128 *iv) + * void _aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *dst, + * const u8 *src, unsigned int len, le128 *iv) */ -SYM_FUNC_START(aesni_xts_decrypt) +SYM_FUNC_START(_aesni_xts_decrypt) FRAME_BEGIN #ifndef __x86_64__ pushl IVP @@ -3160,4 +3140,4 @@ SYM_FUNC_START(aesni_xts_decrypt) movups STATE, (OUTP) jmp .Lxts_dec_ret -SYM_FUNC_END(aesni_xts_decrypt) +SYM_FUNC_END(_aesni_xts_decrypt) diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-intel_glue.c index e09f4672dd38..24d97de6b9e2 100644 --- a/arch/x86/crypto/aesni-intel_glue.c +++ b/arch/x86/crypto/aesni-intel_glue.c @@ -36,33 +36,24 @@ #include #include +#include "aes-intel_glue.h" +#include "aesni-intel_glue.h" -#define AESNI_ALIGN 16 -#define AESNI_ALIGN_ATTR __attribute__ ((__aligned__(AESNI_ALIGN))) -#define AES_BLOCK_MASK (~(AES_BLOCK_SIZE - 1)) #define RFC4106_HASH_SUBKEY_SIZE 16 -#define AESNI_ALIGN_EXTRA ((AESNI_ALIGN - 1) & ~(CRYPTO_MINALIGN - 1)) -#define CRYPTO_AES_CTX_SIZE (sizeof(struct crypto_aes_ctx) + AESNI_ALIGN_EXTRA) -#define XTS_AES_CTX_SIZE (sizeof(struct aesni_xts_ctx) + AESNI_ALIGN_EXTRA) /* This data is stored at the end of the crypto_tfm struct. * It's a type of per "session" data storage location. * This needs to be 16 byte aligned. */ struct aesni_rfc4106_gcm_ctx { - u8 hash_subkey[16] AESNI_ALIGN_ATTR; - struct crypto_aes_ctx aes_key_expanded AESNI_ALIGN_ATTR; + u8 hash_subkey[16] AES_ALIGN_ATTR; + struct crypto_aes_ctx aes_key_expanded AES_ALIGN_ATTR; u8 nonce[4]; }; struct generic_gcmaes_ctx { - u8 hash_subkey[16] AESNI_ALIGN_ATTR; - struct crypto_aes_ctx aes_key_expanded AESNI_ALIGN_ATTR; -}; - -struct aesni_xts_ctx { - u8 raw_tweak_ctx[sizeof(struct crypto_aes_ctx)] AESNI_ALIGN_ATTR; - u8 raw_crypt_ctx[sizeof(struct crypto_aes_ctx)] AESNI_ALIGN_ATTR; + u8 hash_subkey[16] AES_ALIGN_ATTR; + struct crypto_aes_ctx aes_key_expanded AES_ALIGN_ATTR; }; #define GCM_BLOCK_LEN 16 @@ -80,31 +71,84 @@ struct gcm_context_data { u8 hash_keys[GCM_BLOCK_LEN * 16]; }; -asmlinkage int aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key, - unsigned int key_len); -asmlinkage void aesni_enc(const void *ctx, u8 *out, const u8 *in); -asmlinkage void aesni_dec(const void *ctx, u8 *out, const u8 *in); +asmlinkage int _aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key, + unsigned int key_len); +asmlinkage void _aesni_enc(const void *ctx, u8 *out, const u8 *in); +asmlinkage void _aesni_dec(const void *ctx, u8 *out, const u8 *in); asmlinkage void aesni_ecb_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len); asmlinkage void aesni_ecb_dec(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len); -asmlinkage void aesni_cbc_enc(struct crypto_aes_ctx *ctx, u8 *out, - const u8 *in, unsigned int len, u8 *iv); -asmlinkage void aesni_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out, - const u8 *in, unsigned int len, u8 *iv); +asmlinkage void _aesni_cbc_enc(struct crypto_aes_ctx *ctx, u8 *out, + const u8 *in, unsigned int len, u8 *iv); +asmlinkage void _aesni_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out, + const u8 *in, unsigned int len, u8 *iv); asmlinkage void aesni_cts_cbc_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len, u8 *iv); asmlinkage void aesni_cts_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len, u8 *iv); +int aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key, + unsigned int key_len) +{ + return _aesni_set_key(ctx, in_key, key_len); +} +EXPORT_SYMBOL_GPL(aesni_set_key); + +int aesni_enc(const void *ctx, u8 *out, const u8 *in) +{ + _aesni_enc(ctx, out, in); + return 0; +} +EXPORT_SYMBOL_GPL(aesni_enc); + +int aesni_dec(const void *ctx, u8 *out, const u8 *in) +{ + _aesni_dec(ctx, out, in); + return 0; +} +EXPORT_SYMBOL_GPL(aesni_dec); + +int aesni_cbc_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv) +{ + _aesni_cbc_enc(ctx, out, in, len, iv); + return 0; +} +EXPORT_SYMBOL_GPL(aesni_cbc_enc); + +int aesni_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv) +{ + _aesni_cbc_dec(ctx, out, in, len, iv); + return 0; +} +EXPORT_SYMBOL_GPL(aesni_cbc_dec); + #define AVX_GEN2_OPTSIZE 640 #define AVX_GEN4_OPTSIZE 4096 -asmlinkage void aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out, - const u8 *in, unsigned int len, u8 *iv); +asmlinkage void _aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out, + const u8 *in, unsigned int len, u8 *iv); -asmlinkage void aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out, - const u8 *in, unsigned int len, u8 *iv); +asmlinkage void _aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out, + const u8 *in, unsigned int len, u8 *iv); + +int aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv) +{ + _aesni_xts_encrypt(ctx, out, in, len, iv); + return 0; +} +EXPORT_SYMBOL_GPL(aesni_xts_encrypt); + +int aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv) +{ + _aesni_xts_decrypt(ctx, out, in, len, iv); + return 0; +} +EXPORT_SYMBOL_GPL(aesni_xts_decrypt); #ifdef CONFIG_X86_64 @@ -187,7 +231,7 @@ static __ro_after_init DEFINE_STATIC_KEY_FALSE(gcm_use_avx2); static inline struct aesni_rfc4106_gcm_ctx *aesni_rfc4106_gcm_ctx_get(struct crypto_aead *tfm) { - unsigned long align = AESNI_ALIGN; + unsigned long align = AES_ALIGN; if (align <= crypto_tfm_ctx_alignment()) align = 1; @@ -197,7 +241,7 @@ aesni_rfc4106_gcm_ctx *aesni_rfc4106_gcm_ctx_get(struct crypto_aead *tfm) static inline struct generic_gcmaes_ctx *generic_gcmaes_ctx_get(struct crypto_aead *tfm) { - unsigned long align = AESNI_ALIGN; + unsigned long align = AES_ALIGN; if (align <= crypto_tfm_ctx_alignment()) align = 1; @@ -205,16 +249,6 @@ generic_gcmaes_ctx *generic_gcmaes_ctx_get(struct crypto_aead *tfm) } #endif -static inline struct crypto_aes_ctx *aes_ctx(void *raw_ctx) -{ - unsigned long addr = (unsigned long)raw_ctx; - unsigned long align = AESNI_ALIGN; - - if (align <= crypto_tfm_ctx_alignment()) - align = 1; - return (struct crypto_aes_ctx *)ALIGN(addr, align); -} - static int aes_set_key_common(struct crypto_tfm *tfm, void *raw_ctx, const u8 *in_key, unsigned int key_len) { @@ -229,7 +263,7 @@ static int aes_set_key_common(struct crypto_tfm *tfm, void *raw_ctx, err = aes_expandkey(ctx, in_key, key_len); else { kernel_fpu_begin(); - err = aesni_set_key(ctx, in_key, key_len); + err = _aesni_set_key(ctx, in_key, key_len); kernel_fpu_end(); } @@ -250,7 +284,7 @@ static void aesni_encrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src) aes_encrypt(ctx, dst, src); } else { kernel_fpu_begin(); - aesni_enc(ctx, dst, src); + _aesni_enc(ctx, dst, src); kernel_fpu_end(); } } @@ -263,7 +297,7 @@ static void aesni_decrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src) aes_decrypt(ctx, dst, src); } else { kernel_fpu_begin(); - aesni_dec(ctx, dst, src); + _aesni_dec(ctx, dst, src); kernel_fpu_end(); } } @@ -321,46 +355,12 @@ static int ecb_decrypt(struct skcipher_request *req) static int cbc_encrypt(struct skcipher_request *req) { - struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); - struct crypto_aes_ctx *ctx = aes_ctx(crypto_skcipher_ctx(tfm)); - struct skcipher_walk walk; - unsigned int nbytes; - int err; - - err = skcipher_walk_virt(&walk, req, false); - - while ((nbytes = walk.nbytes)) { - kernel_fpu_begin(); - aesni_cbc_enc(ctx, walk.dst.virt.addr, walk.src.virt.addr, - nbytes & AES_BLOCK_MASK, walk.iv); - kernel_fpu_end(); - nbytes &= AES_BLOCK_SIZE - 1; - err = skcipher_walk_done(&walk, nbytes); - } - - return err; + return cbc_crypt_common(req, aesni_cbc_enc); } static int cbc_decrypt(struct skcipher_request *req) { - struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); - struct crypto_aes_ctx *ctx = aes_ctx(crypto_skcipher_ctx(tfm)); - struct skcipher_walk walk; - unsigned int nbytes; - int err; - - err = skcipher_walk_virt(&walk, req, false); - - while ((nbytes = walk.nbytes)) { - kernel_fpu_begin(); - aesni_cbc_dec(ctx, walk.dst.virt.addr, walk.src.virt.addr, - nbytes & AES_BLOCK_MASK, walk.iv); - kernel_fpu_end(); - nbytes &= AES_BLOCK_SIZE - 1; - err = skcipher_walk_done(&walk, nbytes); - } - - return err; + return cbc_crypt_common(req, aesni_cbc_dec); } static int cts_cbc_encrypt(struct skcipher_request *req) @@ -477,7 +477,7 @@ static int cts_cbc_decrypt(struct skcipher_request *req) #ifdef CONFIG_X86_64 static void aesni_ctr_enc_avx_tfm(struct crypto_aes_ctx *ctx, u8 *out, - const u8 *in, unsigned int len, u8 *iv) + const u8 *in, unsigned int len, u8 *iv) { /* * based on key length, override with the by8 version @@ -514,7 +514,7 @@ static int ctr_crypt(struct skcipher_request *req) nbytes &= ~AES_BLOCK_MASK; if (walk.nbytes == walk.total && nbytes > 0) { - aesni_enc(ctx, keystream, walk.iv); + _aesni_enc(ctx, keystream, walk.iv); crypto_xor_cpy(walk.dst.virt.addr + walk.nbytes - nbytes, walk.src.virt.addr + walk.nbytes - nbytes, keystream, nbytes); @@ -606,8 +606,8 @@ static int gcmaes_crypt_by_sg(bool enc, struct aead_request *req, u8 *iv, void *aes_ctx, u8 *auth_tag, unsigned long auth_tag_len) { - u8 databuf[sizeof(struct gcm_context_data) + (AESNI_ALIGN - 8)] __aligned(8); - struct gcm_context_data *data = PTR_ALIGN((void *)databuf, AESNI_ALIGN); + u8 databuf[sizeof(struct gcm_context_data) + (AES_ALIGN - 8)] __aligned(8); + struct gcm_context_data *data = PTR_ALIGN((void *)databuf, AES_ALIGN); unsigned long left = req->cryptlen; struct scatter_walk assoc_sg_walk; struct skcipher_walk walk; @@ -762,8 +762,8 @@ static int helper_rfc4106_encrypt(struct aead_request *req) struct crypto_aead *tfm = crypto_aead_reqtfm(req); struct aesni_rfc4106_gcm_ctx *ctx = aesni_rfc4106_gcm_ctx_get(tfm); void *aes_ctx = &(ctx->aes_key_expanded); - u8 ivbuf[16 + (AESNI_ALIGN - 8)] __aligned(8); - u8 *iv = PTR_ALIGN(&ivbuf[0], AESNI_ALIGN); + u8 ivbuf[16 + (AES_ALIGN - 8)] __aligned(8); + u8 *iv = PTR_ALIGN(&ivbuf[0], AES_ALIGN); unsigned int i; __be32 counter = cpu_to_be32(1); @@ -790,8 +790,8 @@ static int helper_rfc4106_decrypt(struct aead_request *req) struct crypto_aead *tfm = crypto_aead_reqtfm(req); struct aesni_rfc4106_gcm_ctx *ctx = aesni_rfc4106_gcm_ctx_get(tfm); void *aes_ctx = &(ctx->aes_key_expanded); - u8 ivbuf[16 + (AESNI_ALIGN - 8)] __aligned(8); - u8 *iv = PTR_ALIGN(&ivbuf[0], AESNI_ALIGN); + u8 ivbuf[16 + (AES_ALIGN - 8)] __aligned(8); + u8 *iv = PTR_ALIGN(&ivbuf[0], AES_ALIGN); unsigned int i; if (unlikely(req->assoclen != 16 && req->assoclen != 20)) @@ -816,128 +816,17 @@ static int helper_rfc4106_decrypt(struct aead_request *req) static int xts_aesni_setkey(struct crypto_skcipher *tfm, const u8 *key, unsigned int keylen) { - struct aesni_xts_ctx *ctx = crypto_skcipher_ctx(tfm); - int err; - - err = xts_verify_key(tfm, key, keylen); - if (err) - return err; - - keylen /= 2; - - /* first half of xts-key is for crypt */ - err = aes_set_key_common(crypto_skcipher_tfm(tfm), ctx->raw_crypt_ctx, - key, keylen); - if (err) - return err; - - /* second half of xts-key is for tweak */ - return aes_set_key_common(crypto_skcipher_tfm(tfm), ctx->raw_tweak_ctx, - key + keylen, keylen); -} - -static int xts_crypt(struct skcipher_request *req, bool encrypt) -{ - struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); - struct aesni_xts_ctx *ctx = crypto_skcipher_ctx(tfm); - int tail = req->cryptlen % AES_BLOCK_SIZE; - struct skcipher_request subreq; - struct skcipher_walk walk; - int err; - - if (req->cryptlen < AES_BLOCK_SIZE) - return -EINVAL; - - err = skcipher_walk_virt(&walk, req, false); - if (!walk.nbytes) - return err; - - if (unlikely(tail > 0 && walk.nbytes < walk.total)) { - int blocks = DIV_ROUND_UP(req->cryptlen, AES_BLOCK_SIZE) - 2; - - skcipher_walk_abort(&walk); - - skcipher_request_set_tfm(&subreq, tfm); - skcipher_request_set_callback(&subreq, - skcipher_request_flags(req), - NULL, NULL); - skcipher_request_set_crypt(&subreq, req->src, req->dst, - blocks * AES_BLOCK_SIZE, req->iv); - req = &subreq; - - err = skcipher_walk_virt(&walk, req, false); - if (!walk.nbytes) - return err; - } else { - tail = 0; - } - - kernel_fpu_begin(); - - /* calculate first value of T */ - aesni_enc(aes_ctx(ctx->raw_tweak_ctx), walk.iv, walk.iv); - - while (walk.nbytes > 0) { - int nbytes = walk.nbytes; - - if (nbytes < walk.total) - nbytes &= ~(AES_BLOCK_SIZE - 1); - - if (encrypt) - aesni_xts_encrypt(aes_ctx(ctx->raw_crypt_ctx), - walk.dst.virt.addr, walk.src.virt.addr, - nbytes, walk.iv); - else - aesni_xts_decrypt(aes_ctx(ctx->raw_crypt_ctx), - walk.dst.virt.addr, walk.src.virt.addr, - nbytes, walk.iv); - kernel_fpu_end(); - - err = skcipher_walk_done(&walk, walk.nbytes - nbytes); - - if (walk.nbytes > 0) - kernel_fpu_begin(); - } - - if (unlikely(tail > 0 && !err)) { - struct scatterlist sg_src[2], sg_dst[2]; - struct scatterlist *src, *dst; - - dst = src = scatterwalk_ffwd(sg_src, req->src, req->cryptlen); - if (req->dst != req->src) - dst = scatterwalk_ffwd(sg_dst, req->dst, req->cryptlen); - - skcipher_request_set_crypt(req, src, dst, AES_BLOCK_SIZE + tail, - req->iv); - - err = skcipher_walk_virt(&walk, &subreq, false); - if (err) - return err; - - kernel_fpu_begin(); - if (encrypt) - aesni_xts_encrypt(aes_ctx(ctx->raw_crypt_ctx), - walk.dst.virt.addr, walk.src.virt.addr, - walk.nbytes, walk.iv); - else - aesni_xts_decrypt(aes_ctx(ctx->raw_crypt_ctx), - walk.dst.virt.addr, walk.src.virt.addr, - walk.nbytes, walk.iv); - kernel_fpu_end(); - - err = skcipher_walk_done(&walk, 0); - } - return err; + return xts_setkey_common(tfm, key, keylen, aes_set_key_common); } static int xts_encrypt(struct skcipher_request *req) { - return xts_crypt(req, true); + return xts_crypt_common(req, aesni_xts_encrypt, aesni_enc); } static int xts_decrypt(struct skcipher_request *req) { - return xts_crypt(req, false); + return xts_crypt_common(req, aesni_xts_decrypt, aesni_enc); } static struct crypto_alg aesni_cipher_alg = { @@ -1066,8 +955,8 @@ static int generic_gcmaes_encrypt(struct aead_request *req) struct crypto_aead *tfm = crypto_aead_reqtfm(req); struct generic_gcmaes_ctx *ctx = generic_gcmaes_ctx_get(tfm); void *aes_ctx = &(ctx->aes_key_expanded); - u8 ivbuf[16 + (AESNI_ALIGN - 8)] __aligned(8); - u8 *iv = PTR_ALIGN(&ivbuf[0], AESNI_ALIGN); + u8 ivbuf[16 + (AES_ALIGN - 8)] __aligned(8); + u8 *iv = PTR_ALIGN(&ivbuf[0], AES_ALIGN); __be32 counter = cpu_to_be32(1); memcpy(iv, req->iv, 12); @@ -1083,8 +972,8 @@ static int generic_gcmaes_decrypt(struct aead_request *req) struct crypto_aead *tfm = crypto_aead_reqtfm(req); struct generic_gcmaes_ctx *ctx = generic_gcmaes_ctx_get(tfm); void *aes_ctx = &(ctx->aes_key_expanded); - u8 ivbuf[16 + (AESNI_ALIGN - 8)] __aligned(8); - u8 *iv = PTR_ALIGN(&ivbuf[0], AESNI_ALIGN); + u8 ivbuf[16 + (AES_ALIGN - 8)] __aligned(8); + u8 *iv = PTR_ALIGN(&ivbuf[0], AES_ALIGN); memcpy(iv, req->iv, 12); *((__be32 *)(iv+12)) = counter; @@ -1107,7 +996,7 @@ static struct aead_alg aesni_aeads[] = { { .cra_flags = CRYPTO_ALG_INTERNAL, .cra_blocksize = 1, .cra_ctxsize = sizeof(struct aesni_rfc4106_gcm_ctx), - .cra_alignmask = AESNI_ALIGN - 1, + .cra_alignmask = AES_ALIGN - 1, .cra_module = THIS_MODULE, }, }, { @@ -1124,7 +1013,7 @@ static struct aead_alg aesni_aeads[] = { { .cra_flags = CRYPTO_ALG_INTERNAL, .cra_blocksize = 1, .cra_ctxsize = sizeof(struct generic_gcmaes_ctx), - .cra_alignmask = AESNI_ALIGN - 1, + .cra_alignmask = AES_ALIGN - 1, .cra_module = THIS_MODULE, }, } }; diff --git a/arch/x86/crypto/aesni-intel_glue.h b/arch/x86/crypto/aesni-intel_glue.h new file mode 100644 index 000000000000..ca202063f760 --- /dev/null +++ b/arch/x86/crypto/aesni-intel_glue.h @@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* + * Support for Intel AES-NI instructions. This file contains function + * prototypes to be referenced for other AES implementations + */ + +int aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key, unsigned int key_len); + +int aesni_enc(const void *ctx, u8 *out, const u8 *in); +int aesni_dec(const void *ctx, u8 *out, const u8 *in); + +int aesni_cbc_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv); +int aesni_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv); + +int aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv); +int aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv); + From patchwork Tue Dec 14 00:52:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chang S. Bae" X-Patchwork-Id: 524350 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D401C4332F for ; Tue, 14 Dec 2021 01:00:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244660AbhLNBAH (ORCPT ); Mon, 13 Dec 2021 20:00:07 -0500 Received: from mga05.intel.com ([192.55.52.43]:21439 "EHLO mga05.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244638AbhLNBAE (ORCPT ); Mon, 13 Dec 2021 20:00:04 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1639443604; x=1670979604; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=u2FuZ27PzznQBUCR8gc9cvubZ2Y4L1Ihu6/LbDPAjmA=; b=B+xFSgukIL+EZakiAKz3ftf+vIIwoFtsNg4//WAoBFGMYiVDYGYh4Pwb B12R3qTVkg6K05vn/gTqRiC3219AF0EBw7dld2iQTNiB3+OL/VBSzhveX NIKxVK+XwPhbsiAwN5mevBxL0QySKgp3YoSJ2yqr++q/IjX7vAL56YvaA dlQsPA9OPjyFjBHblgWyfxMgrig4PWYrT77v0EIkwrUTP9Glupn5mFoni f2MvwaqNKhrVT6DmRqrato/6ni1F8SGHnZ8XNiW6cABH7axlS6v0tGguH z4ihxKDSFaM+xYkRY8xzMaMpBlEH8AYv60VPIgWHSSBa0588Ec2CBm1OU A==; X-IronPort-AV: E=McAfee;i="6200,9189,10197"; a="325139646" X-IronPort-AV: E=Sophos;i="5.88,203,1635231600"; d="scan'208";a="325139646" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Dec 2021 16:59:43 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,203,1635231600"; d="scan'208";a="505148631" Received: from chang-linux-3.sc.intel.com ([172.25.66.175]) by orsmga007.jf.intel.com with ESMTP; 13 Dec 2021 16:59:42 -0800 From: "Chang S. Bae" To: tglx@linutronix.de, bp@suse.de, dave.hansen@linux.intel.com, mingo@kernel.org, luto@kernel.org, x86@kernel.org, herbert@gondor.apana.org.au Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, ebiggers@kernel.org, dan.j.williams@intel.com, charishma1.gairuboyina@intel.com, kumar.n.dwarakanath@intel.com, lalithambika.krishnakumar@intel.com, ravi.v.shankar@intel.com, chang.seok.bae@intel.com Subject: [PATCH v4 13/13] crypto: x86/aes-kl - Support XTS mode Date: Mon, 13 Dec 2021 16:52:12 -0800 Message-Id: <20211214005212.20588-14-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20211214005212.20588-1-chang.seok.bae@intel.com> References: <20211214005212.20588-1-chang.seok.bae@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Implement XTS mode using AES-KL. Export the methods with a lower priority than AES-NI to avoid from selected by default. The assembly code clobbers more than eight 128-bit registers to make use of performant wide instructions that process eight 128-bit blocks at once. But that many 128-bit registers are not available in 32-bit mode, so support 64-bit mode only. Signed-off-by: Chang S. Bae Acked-by: Dan Williams Cc: Herbert Xu Cc: x86@kernel.org Cc: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org --- Changes from RFC v2: * Separate out the code as a new patch. --- arch/x86/crypto/aeskl-intel_asm.S | 449 +++++++++++++++++++++++++++++ arch/x86/crypto/aeskl-intel_glue.c | 86 ++++++ 2 files changed, 535 insertions(+) diff --git a/arch/x86/crypto/aeskl-intel_asm.S b/arch/x86/crypto/aeskl-intel_asm.S index 5ee7b24ee3c8..731081daba51 100644 --- a/arch/x86/crypto/aeskl-intel_asm.S +++ b/arch/x86/crypto/aeskl-intel_asm.S @@ -563,3 +563,452 @@ SYM_FUNC_START(_aeskl_cbc_dec) ret SYM_FUNC_END(_aeskl_cbc_dec) +#ifdef __x86_64__ + +/* + * XTS implementation + */ + +/* + * _aeskl_gf128mul_x_ble: internal ABI + * Multiply in GF(2^128) for XTS IVs + * input: + * IV: current IV + * GF128MUL_MASK == mask with 0x87 and 0x01 + * output: + * IV: next IV + * changed: + * CTR: == temporary value + */ +#define _aeskl_gf128mul_x_ble() \ + pshufd $0x13, IV, KEY; \ + paddq IV, IV; \ + psrad $31, KEY; \ + pand GF128MUL_MASK, KEY; \ + pxor KEY, IV; + +/* + * int _aeskl_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *dst, + * const u8 *src, unsigned int len, le128 *iv) + */ +SYM_FUNC_START(_aeskl_xts_encrypt) + FRAME_BEGIN + movdqa .Lgf128mul_x_ble_mask(%rip), GF128MUL_MASK + movups (IVP), IV + + mov 480(HANDLEP), KLEN + +.Lxts_enc8: + sub $128, LEN + jl .Lxts_enc1_pre + + movdqa IV, STATE1 + movdqu (INP), INC + pxor INC, STATE1 + movdqu IV, (OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE2 + movdqu 0x10(INP), INC + pxor INC, STATE2 + movdqu IV, 0x10(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE3 + movdqu 0x20(INP), INC + pxor INC, STATE3 + movdqu IV, 0x20(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE4 + movdqu 0x30(INP), INC + pxor INC, STATE4 + movdqu IV, 0x30(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE5 + movdqu 0x40(INP), INC + pxor INC, STATE5 + movdqu IV, 0x40(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE6 + movdqu 0x50(INP), INC + pxor INC, STATE6 + movdqu IV, 0x50(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE7 + movdqu 0x60(INP), INC + pxor INC, STATE7 + movdqu IV, 0x60(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE8 + movdqu 0x70(INP), INC + pxor INC, STATE8 + movdqu IV, 0x70(OUTP) + + cmp $16, KLEN + je .Lxts_enc8_128 + aesencwide256kl (%rdi) + jz .Lxts_enc_ret_err + jmp .Lxts_enc8_end +.Lxts_enc8_128: + aesencwide128kl (%rdi) + jz .Lxts_enc_ret_err + +.Lxts_enc8_end: + movdqu 0x00(OUTP), INC + pxor INC, STATE1 + movdqu STATE1, 0x00(OUTP) + + movdqu 0x10(OUTP), INC + pxor INC, STATE2 + movdqu STATE2, 0x10(OUTP) + + movdqu 0x20(OUTP), INC + pxor INC, STATE3 + movdqu STATE3, 0x20(OUTP) + + movdqu 0x30(OUTP), INC + pxor INC, STATE4 + movdqu STATE4, 0x30(OUTP) + + movdqu 0x40(OUTP), INC + pxor INC, STATE5 + movdqu STATE5, 0x40(OUTP) + + movdqu 0x50(OUTP), INC + pxor INC, STATE6 + movdqu STATE6, 0x50(OUTP) + + movdqu 0x60(OUTP), INC + pxor INC, STATE7 + movdqu STATE7, 0x60(OUTP) + + movdqu 0x70(OUTP), INC + pxor INC, STATE8 + movdqu STATE8, 0x70(OUTP) + + _aeskl_gf128mul_x_ble() + + add $128, INP + add $128, OUTP + test LEN, LEN + jnz .Lxts_enc8 + +.Lxts_enc_ret_iv: + movups IV, (IVP) +.Lxts_enc_ret_noerr: + xor AREG, AREG + jmp .Lxts_enc_ret +.Lxts_enc_ret_err: + mov $1, AREG +.Lxts_enc_ret: + FRAME_END + ret + +.Lxts_enc1_pre: + add $128, LEN + jz .Lxts_enc_ret_iv + sub $16, LEN + jl .Lxts_enc_cts4 + +.Lxts_enc1: + movdqu (INP), STATE1 + pxor IV, STATE1 + + cmp $16, KLEN + je .Lxts_enc1_128 + aesenc256kl (HANDLEP), STATE1 + jz .Lxts_enc_ret_err + jmp .Lxts_enc1_end +.Lxts_enc1_128: + aesenc128kl (HANDLEP), STATE1 + jz .Lxts_enc_ret_err + +.Lxts_enc1_end: + pxor IV, STATE1 + _aeskl_gf128mul_x_ble() + + test LEN, LEN + jz .Lxts_enc1_out + + add $16, INP + sub $16, LEN + jl .Lxts_enc_cts1 + + movdqu STATE1, (OUTP) + add $16, OUTP + jmp .Lxts_enc1 + +.Lxts_enc1_out: + movdqu STATE1, (OUTP) + jmp .Lxts_enc_ret_iv + +.Lxts_enc_cts4: + movdqu STATE8, STATE1 + sub $16, OUTP + +.Lxts_enc_cts1: + lea .Lcts_permute_table(%rip), T1 + add LEN, INP /* rewind input pointer */ + add $16, LEN /* # bytes in final block */ + movups (INP), IN1 + + mov T1, IVP + add $32, IVP + add LEN, T1 + sub LEN, IVP + add OUTP, LEN + + movups (T1), STATE2 + movaps STATE1, STATE3 + pshufb STATE2, STATE1 + movups STATE1, (LEN) + + movups (IVP), STATE1 + pshufb STATE1, IN1 + pblendvb STATE3, IN1 + movaps IN1, STATE1 + + pxor IV, STATE1 + + cmp $16, KLEN + je .Lxts_enc1_cts_128 + aesenc256kl (HANDLEP), STATE1 + jz .Lxts_enc_ret_err + jmp .Lxts_enc1_cts_end +.Lxts_enc1_cts_128: + aesenc128kl (HANDLEP), STATE1 + jz .Lxts_enc_ret_err + +.Lxts_enc1_cts_end: + pxor IV, STATE1 + movups STATE1, (OUTP) + jmp .Lxts_enc_ret_noerr +SYM_FUNC_END(_aeskl_xts_encrypt) + +/* + * int _aeskl_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *dst, + * const u8 *src, unsigned int len, le128 *iv) + */ +SYM_FUNC_START(_aeskl_xts_decrypt) + FRAME_BEGIN + movdqa .Lgf128mul_x_ble_mask(%rip), GF128MUL_MASK + movups (IVP), IV + + mov 480(HANDLEP), KLEN + + test $15, LEN + jz .Lxts_dec8 + sub $16, LEN + +.Lxts_dec8: + sub $128, LEN + jl .Lxts_dec1_pre + + movdqa IV, STATE1 + movdqu (INP), INC + pxor INC, STATE1 + movdqu IV, (OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE2 + movdqu 0x10(INP), INC + pxor INC, STATE2 + movdqu IV, 0x10(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE3 + movdqu 0x20(INP), INC + pxor INC, STATE3 + movdqu IV, 0x20(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE4 + movdqu 0x30(INP), INC + pxor INC, STATE4 + movdqu IV, 0x30(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE5 + movdqu 0x40(INP), INC + pxor INC, STATE5 + movdqu IV, 0x40(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE6 + movdqu 0x50(INP), INC + pxor INC, STATE6 + movdqu IV, 0x50(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE7 + movdqu 0x60(INP), INC + pxor INC, STATE7 + movdqu IV, 0x60(OUTP) + + _aeskl_gf128mul_x_ble() + movdqa IV, STATE8 + movdqu 0x70(INP), INC + pxor INC, STATE8 + movdqu IV, 0x70(OUTP) + + cmp $16, KLEN + je .Lxts_dec8_128 + aesdecwide256kl (%rdi) + jz .Lxts_dec_ret_err + jmp .Lxts_dec8_end +.Lxts_dec8_128: + aesdecwide128kl (%rdi) + jz .Lxts_dec_ret_err + +.Lxts_dec8_end: + movdqu 0x00(OUTP), INC + pxor INC, STATE1 + movdqu STATE1, 0x00(OUTP) + + movdqu 0x10(OUTP), INC + pxor INC, STATE2 + movdqu STATE2, 0x10(OUTP) + + movdqu 0x20(OUTP), INC + pxor INC, STATE3 + movdqu STATE3, 0x20(OUTP) + + movdqu 0x30(OUTP), INC + pxor INC, STATE4 + movdqu STATE4, 0x30(OUTP) + + movdqu 0x40(OUTP), INC + pxor INC, STATE5 + movdqu STATE5, 0x40(OUTP) + + movdqu 0x50(OUTP), INC + pxor INC, STATE6 + movdqu STATE6, 0x50(OUTP) + + movdqu 0x60(OUTP), INC + pxor INC, STATE7 + movdqu STATE7, 0x60(OUTP) + + movdqu 0x70(OUTP), INC + pxor INC, STATE8 + movdqu STATE8, 0x70(OUTP) + + _aeskl_gf128mul_x_ble() + + add $128, INP + add $128, OUTP + test LEN, LEN + jnz .Lxts_dec8 + +.Lxts_dec_ret_iv: + movups IV, (IVP) +.Lxts_dec_ret_noerr: + xor AREG, AREG + jmp .Lxts_dec_ret +.Lxts_dec_ret_err: + mov $1, AREG +.Lxts_dec_ret: + FRAME_END + ret + +.Lxts_dec1_pre: + add $128, LEN + jz .Lxts_dec_ret_iv + +.Lxts_dec1: + movdqu (INP), STATE1 + + add $16, INP + sub $16, LEN + jl .Lxts_dec_cts1 + + pxor IV, STATE1 + + cmp $16, KLEN + je .Lxts_dec1_128 + aesdec256kl (HANDLEP), STATE1 + jz .Lxts_dec_ret_err + jmp .Lxts_dec1_end +.Lxts_dec1_128: + aesdec128kl (HANDLEP), STATE1 + jz .Lxts_dec_ret_err + +.Lxts_dec1_end: + pxor IV, STATE1 + _aeskl_gf128mul_x_ble() + + test LEN, LEN + jz .Lxts_dec1_out + + movdqu STATE1, (OUTP) + add $16, OUTP + jmp .Lxts_dec1 + +.Lxts_dec1_out: + movdqu STATE1, (OUTP) + jmp .Lxts_dec_ret_iv + +.Lxts_dec_cts1: + movdqa IV, STATE5 + _aeskl_gf128mul_x_ble() + + pxor IV, STATE1 + + cmp $16, KLEN + je .Lxts_dec1_cts_pre_128 + aesdec256kl (HANDLEP), STATE1 + jz .Lxts_dec_ret_err + jmp .Lxts_dec1_cts_pre_end +.Lxts_dec1_cts_pre_128: + aesdec128kl (HANDLEP), STATE1 + jz .Lxts_dec_ret_err + +.Lxts_dec1_cts_pre_end: + pxor IV, STATE1 + + lea .Lcts_permute_table(%rip), T1 + add LEN, INP /* rewind input pointer */ + add $16, LEN /* # bytes in final block */ + movups (INP), IN1 + + mov T1, IVP + add $32, IVP + add LEN, T1 + sub LEN, IVP + add OUTP, LEN + + movups (T1), STATE2 + movaps STATE1, STATE3 + pshufb STATE2, STATE1 + movups STATE1, (LEN) + + movups (IVP), STATE1 + pshufb STATE1, IN1 + pblendvb STATE3, IN1 + movaps IN1, STATE1 + + pxor STATE5, STATE1 + + cmp $16, KLEN + je .Lxts_dec1_cts_128 + aesdec256kl (HANDLEP), STATE1 + jz .Lxts_dec_ret_err + jmp .Lxts_dec1_cts_end +.Lxts_dec1_cts_128: + aesdec128kl (HANDLEP), STATE1 + jz .Lxts_dec_ret_err + +.Lxts_dec1_cts_end: + pxor STATE5, STATE1 + + movups STATE1, (OUTP) + jmp .Lxts_dec_ret_noerr + +SYM_FUNC_END(_aeskl_xts_decrypt) + +#endif diff --git a/arch/x86/crypto/aeskl-intel_glue.c b/arch/x86/crypto/aeskl-intel_glue.c index 22b74d13150c..d8ed937cb0e5 100644 --- a/arch/x86/crypto/aeskl-intel_glue.c +++ b/arch/x86/crypto/aeskl-intel_glue.c @@ -32,6 +32,13 @@ asmlinkage int _aeskl_cbc_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, asmlinkage int _aeskl_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len, u8 *iv); +#ifdef CONFIG_X86_64 +asmlinkage int _aeskl_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv); +asmlinkage int _aeskl_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv); +#endif + static int aeskl_setkey_common(struct crypto_tfm *tfm, void *raw_ctx, const u8 *in_key, unsigned int key_len) { @@ -117,6 +124,36 @@ static int aeskl_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsi return 0; } +#ifdef CONFIG_X86_64 + +static inline int aeskl_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv) +{ + if (unlikely(ctx->key_length == AES_KEYSIZE_192)) + return -EINVAL; + else if (!valid_keylocker()) + return -ENODEV; + else if (_aeskl_xts_encrypt(ctx, out, in, len, iv)) + return -EINVAL; + else + return 0; +} + +static inline int aeskl_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, + unsigned int len, u8 *iv) +{ + if (unlikely(ctx->key_length == AES_KEYSIZE_192)) + return -EINVAL; + else if (!valid_keylocker()) + return -ENODEV; + else if (_aeskl_xts_decrypt(ctx, out, in, len, iv)) + return -EINVAL; + else + return 0; +} + +#endif /* CONFIG_X86_64 */ + static int aeskl_skcipher_setkey(struct crypto_skcipher *tfm, const u8 *key, unsigned int len) { @@ -146,6 +183,36 @@ static int cbc_decrypt(struct skcipher_request *req) return cbc_crypt_common(req, aesni_cbc_dec); } +#ifdef CONFIG_X86_64 + +static int aeskl_xts_setkey(struct crypto_skcipher *tfm, const u8 *key, + unsigned int keylen) +{ + return xts_setkey_common(tfm, key, keylen, aeskl_setkey_common); +} + +static int xts_encrypt(struct skcipher_request *req) +{ + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); + + if (likely(keylength(crypto_skcipher_ctx(tfm)) != AES_KEYSIZE_192)) + return xts_crypt_common(req, aeskl_xts_encrypt, aeskl_enc); + else + return xts_crypt_common(req, aesni_xts_encrypt, aesni_enc); +} + +static int xts_decrypt(struct skcipher_request *req) +{ + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); + + if (likely(keylength(crypto_skcipher_ctx(tfm)) != AES_KEYSIZE_192)) + return xts_crypt_common(req, aeskl_xts_decrypt, aeskl_enc); + else + return xts_crypt_common(req, aesni_xts_decrypt, aesni_enc); +} + +#endif /* CONFIG_X86_64 */ + static struct skcipher_alg aeskl_skciphers[] = { { .base = { @@ -163,6 +230,25 @@ static struct skcipher_alg aeskl_skciphers[] = { .setkey = aeskl_skcipher_setkey, .encrypt = cbc_encrypt, .decrypt = cbc_decrypt, +#ifdef CONFIG_X86_64 + }, { + .base = { + .cra_name = "__xts(aes)", + .cra_driver_name = "__xts-aes-aeskl", + .cra_priority = 200, + .cra_flags = CRYPTO_ALG_INTERNAL, + .cra_blocksize = AES_BLOCK_SIZE, + .cra_ctxsize = XTS_AES_CTX_SIZE, + .cra_module = THIS_MODULE, + }, + .min_keysize = 2 * AES_MIN_KEY_SIZE, + .max_keysize = 2 * AES_MAX_KEY_SIZE, + .ivsize = AES_BLOCK_SIZE, + .walksize = 2 * AES_BLOCK_SIZE, + .setkey = aeskl_xts_setkey, + .encrypt = xts_encrypt, + .decrypt = xts_decrypt, +#endif } };