From patchwork Fri Nov 18 09:02:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 626245 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F018C4332F for ; Fri, 18 Nov 2022 09:04:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240803AbiKRJEL (ORCPT ); Fri, 18 Nov 2022 04:04:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54266 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241455AbiKRJEJ (ORCPT ); Fri, 18 Nov 2022 04:04:09 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AAF101143 for ; Fri, 18 Nov 2022 01:04:08 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 3F0346230C for ; Fri, 18 Nov 2022 09:04:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7DCEAC433D7; Fri, 18 Nov 2022 09:04:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668762247; bh=cS7/6QLD7TgC3YuEBW48zGmDobm+o6walAidsI/8OGs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NjtJJyf8H7nZ96eNiOfXsu5McNg8JK5zV0FckJdxHo8dDY1NKgZewBSzMQLrYjZf3 SQqLbctXfcuL4XCZjEr4uavRjOYolvNXe42Abn/F/cnd3krlFwnTV45bfevAgH6lz2 80E4RvUb7RkdTzZHOgTv9LEbCVnQYW0qTa5lV++HqJfze5XVfPfOnukxyMG4ZvZP2c bvzL0z1nrfFSUJBVT39wzLqEq/rW2tK3pLR02zpLG/l4Ft48+lkJZwRcrj2jM57h2g SYPkpj0gY3Jbwk2oLs2tE28qGg4wJ32KKjk+N+YhFwrqqWe0BdS2uJe0A5YH6UVcba jS5k7ZOOtX3ZA== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: x86@kernel.org, linux-arm-kernel@lists.infradead.org, Sami Tolvanen Subject: [PATCH 01/11] crypto: x86/aegis128 - fix crash with CFI enabled Date: Fri, 18 Nov 2022 01:02:10 -0800 Message-Id: <20221118090220.398819-2-ebiggers@kernel.org> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221118090220.398819-1-ebiggers@kernel.org> References: <20221118090220.398819-1-ebiggers@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org From: Eric Biggers Some functions in aegis128-aesni-asm.S are called via indirect function calls. These functions need to use SYM_TYPED_FUNC_START instead of SYM_FUNC_START to cause type hashes to be emitted when the kernel is built with CONFIG_CFI_CLANG=y. Otherwise, the code crashes with a CFI failure. Fixes: 3c516f89e17e ("x86: Add support for CONFIG_CFI_CLANG") Signed-off-by: Eric Biggers --- arch/x86/crypto/aegis128-aesni-asm.S | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/arch/x86/crypto/aegis128-aesni-asm.S b/arch/x86/crypto/aegis128-aesni-asm.S index b48ddebb47489..cdf3215ec272c 100644 --- a/arch/x86/crypto/aegis128-aesni-asm.S +++ b/arch/x86/crypto/aegis128-aesni-asm.S @@ -7,6 +7,7 @@ */ #include +#include #include #define STATE0 %xmm0 @@ -402,7 +403,7 @@ SYM_FUNC_END(crypto_aegis128_aesni_ad) * void crypto_aegis128_aesni_enc(void *state, unsigned int length, * const void *src, void *dst); */ -SYM_FUNC_START(crypto_aegis128_aesni_enc) +SYM_TYPED_FUNC_START(crypto_aegis128_aesni_enc) FRAME_BEGIN cmp $0x10, LEN @@ -499,7 +500,7 @@ SYM_FUNC_END(crypto_aegis128_aesni_enc) * void crypto_aegis128_aesni_enc_tail(void *state, unsigned int length, * const void *src, void *dst); */ -SYM_FUNC_START(crypto_aegis128_aesni_enc_tail) +SYM_TYPED_FUNC_START(crypto_aegis128_aesni_enc_tail) FRAME_BEGIN /* load the state: */ @@ -556,7 +557,7 @@ SYM_FUNC_END(crypto_aegis128_aesni_enc_tail) * void crypto_aegis128_aesni_dec(void *state, unsigned int length, * const void *src, void *dst); */ -SYM_FUNC_START(crypto_aegis128_aesni_dec) +SYM_TYPED_FUNC_START(crypto_aegis128_aesni_dec) FRAME_BEGIN cmp $0x10, LEN @@ -653,7 +654,7 @@ SYM_FUNC_END(crypto_aegis128_aesni_dec) * void crypto_aegis128_aesni_dec_tail(void *state, unsigned int length, * const void *src, void *dst); */ -SYM_FUNC_START(crypto_aegis128_aesni_dec_tail) +SYM_TYPED_FUNC_START(crypto_aegis128_aesni_dec_tail) FRAME_BEGIN /* load the state: */ From patchwork Fri Nov 18 09:02:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 626242 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1214C43217 for ; Fri, 18 Nov 2022 09:04:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241530AbiKRJEV (ORCPT ); Fri, 18 Nov 2022 04:04:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54352 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241704AbiKRJEM (ORCPT ); Fri, 18 Nov 2022 04:04:12 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EEF7382BC9 for ; Fri, 18 Nov 2022 01:04:09 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 1BB78623BB for ; Fri, 18 Nov 2022 09:04:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 62593C433D7; Fri, 18 Nov 2022 09:04:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668762248; bh=izas7t+6oGMs9+g/37vH55gFtFtlIMabHc0IYfz658Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Zf8+T9vZhJ4QB3orc2MRN4ra0Ac9rGJfEPJ53UqrkXAPvoo44QU3Vu7eUy/ch24I7 AHTqR9zvBWC12+JTY29n6NBcr/crrWNjbKxqztDHjwYgRxaSX0vApyVLgdTFYBvK9K u8ZUFQFlt3bCQZaUO3BbD1owS0hPOpx6hlKuuCg70BSYk4ueI5RtPQGXwn2RVzFox8 GzUw7yvAArNuDMULWjziqgd/Fppz6Z06e9j76DD8YaV215kQb8yzTnR2Kut3QniMxe nQ9MSxd0vciqlDwuWJoAHQnwgwQ5BxxvLWpg/DGL60PQ8I3KfUp8J0Ge6kvK9xzTsg 6iLV/zxHtLD2w== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: x86@kernel.org, linux-arm-kernel@lists.infradead.org, Sami Tolvanen Subject: [PATCH 04/11] crypto: x86/sha1 - fix possible crash with CFI enabled Date: Fri, 18 Nov 2022 01:02:13 -0800 Message-Id: <20221118090220.398819-5-ebiggers@kernel.org> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221118090220.398819-1-ebiggers@kernel.org> References: <20221118090220.398819-1-ebiggers@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org From: Eric Biggers sha1_transform_ssse3(), sha1_transform_avx(), and sha1_ni_transform() (but not sha1_transform_avx2()) are called via indirect function calls. These functions need to use SYM_TYPED_FUNC_START instead of SYM_FUNC_START to cause type hashes to be emitted when the kernel is built with CONFIG_CFI_CLANG=y. Otherwise, the code crashes with a CFI failure (if the compiler didn't happen to optimize out the indirect calls). Fixes: 3c516f89e17e ("x86: Add support for CONFIG_CFI_CLANG") Signed-off-by: Eric Biggers --- arch/x86/crypto/sha1_ni_asm.S | 3 ++- arch/x86/crypto/sha1_ssse3_asm.S | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/arch/x86/crypto/sha1_ni_asm.S b/arch/x86/crypto/sha1_ni_asm.S index 2f94ec0e763bf..3cae5a1bb3d6e 100644 --- a/arch/x86/crypto/sha1_ni_asm.S +++ b/arch/x86/crypto/sha1_ni_asm.S @@ -54,6 +54,7 @@ */ #include +#include #define DIGEST_PTR %rdi /* 1st arg */ #define DATA_PTR %rsi /* 2nd arg */ @@ -93,7 +94,7 @@ */ .text .align 32 -SYM_FUNC_START(sha1_ni_transform) +SYM_TYPED_FUNC_START(sha1_ni_transform) push %rbp mov %rsp, %rbp sub $FRAME_SIZE, %rsp diff --git a/arch/x86/crypto/sha1_ssse3_asm.S b/arch/x86/crypto/sha1_ssse3_asm.S index 263f916362e02..f54988c80eb40 100644 --- a/arch/x86/crypto/sha1_ssse3_asm.S +++ b/arch/x86/crypto/sha1_ssse3_asm.S @@ -25,6 +25,7 @@ */ #include +#include #define CTX %rdi // arg1 #define BUF %rsi // arg2 @@ -67,7 +68,7 @@ * param: function's name */ .macro SHA1_VECTOR_ASM name - SYM_FUNC_START(\name) + SYM_TYPED_FUNC_START(\name) push %rbx push %r12 From patchwork Fri Nov 18 09:02:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 626243 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70192C433FE for ; Fri, 18 Nov 2022 09:04:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241668AbiKRJET (ORCPT ); Fri, 18 Nov 2022 04:04:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54344 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241436AbiKRJEM (ORCPT ); Fri, 18 Nov 2022 04:04:12 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EEECA82BC5 for ; Fri, 18 Nov 2022 01:04:09 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6890F623C0 for ; Fri, 18 Nov 2022 09:04:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A7AB4C43147; Fri, 18 Nov 2022 09:04:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668762248; bh=mJsH5L9LpAZtcysiSyHXhJEv09v85eGVUHWCSa6xvLc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sA1CiL1XzosLH61OUxoCSm3C1QMDTQZshxtbEDoz3+1vk8+AzfNScnLOcD23KoKI9 jwlUpcSmq8gdvTeTi9a6KxA/zMHVR3e4RquZ9PE1RaEpFHgAUvc2m6EFgsPERAmdME UUptYxUzbVxH0ID07b45ZnAGqEDvd1I+M4yKsuYMENsd0epuuXT148thCyt73NrWYn xqzYILvnzM0Nmng7KEa9BQfRu4POghIeyzzA0WBgxIMFnosIuCMgJviO9KeD0nUCeM 2hvBJ2oddwzRv6RciknvcNnRF0+VttvJ9EzVg9iqfeBJY/+30QevbwQ8x47FO9JG2G o+jSLeehssCSA== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: x86@kernel.org, linux-arm-kernel@lists.infradead.org, Sami Tolvanen Subject: [PATCH 05/11] crypto: x86/sha256 - fix possible crash with CFI enabled Date: Fri, 18 Nov 2022 01:02:14 -0800 Message-Id: <20221118090220.398819-6-ebiggers@kernel.org> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221118090220.398819-1-ebiggers@kernel.org> References: <20221118090220.398819-1-ebiggers@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org From: Eric Biggers sha256_transform_ssse3(), sha256_transform_avx(), sha256_transform_rorx(), and sha256_transform_ni() are called via indirect function calls. These functions need to use SYM_TYPED_FUNC_START instead of SYM_FUNC_START to cause type hashes to be emitted when the kernel is built with CONFIG_CFI_CLANG=y. Otherwise, the code crashes with a CFI failure (if the compiler didn't happen to optimize out the indirect calls). Fixes: 3c516f89e17e ("x86: Add support for CONFIG_CFI_CLANG") Signed-off-by: Eric Biggers --- arch/x86/crypto/sha256-avx-asm.S | 3 ++- arch/x86/crypto/sha256-avx2-asm.S | 3 ++- arch/x86/crypto/sha256-ssse3-asm.S | 3 ++- arch/x86/crypto/sha256_ni_asm.S | 3 ++- 4 files changed, 8 insertions(+), 4 deletions(-) diff --git a/arch/x86/crypto/sha256-avx-asm.S b/arch/x86/crypto/sha256-avx-asm.S index 3baa1ec390974..06ea30c20828d 100644 --- a/arch/x86/crypto/sha256-avx-asm.S +++ b/arch/x86/crypto/sha256-avx-asm.S @@ -48,6 +48,7 @@ ######################################################################## #include +#include ## assume buffers not aligned #define VMOVDQ vmovdqu @@ -346,7 +347,7 @@ a = TMP_ ## arg 3 : Num blocks ######################################################################## .text -SYM_FUNC_START(sha256_transform_avx) +SYM_TYPED_FUNC_START(sha256_transform_avx) .align 32 pushq %rbx pushq %r12 diff --git a/arch/x86/crypto/sha256-avx2-asm.S b/arch/x86/crypto/sha256-avx2-asm.S index 9bcdbc47b8b4b..2d2be531a11ed 100644 --- a/arch/x86/crypto/sha256-avx2-asm.S +++ b/arch/x86/crypto/sha256-avx2-asm.S @@ -49,6 +49,7 @@ ######################################################################## #include +#include ## assume buffers not aligned #define VMOVDQ vmovdqu @@ -523,7 +524,7 @@ STACK_SIZE = _CTX + _CTX_SIZE ## arg 3 : Num blocks ######################################################################## .text -SYM_FUNC_START(sha256_transform_rorx) +SYM_TYPED_FUNC_START(sha256_transform_rorx) .align 32 pushq %rbx pushq %r12 diff --git a/arch/x86/crypto/sha256-ssse3-asm.S b/arch/x86/crypto/sha256-ssse3-asm.S index c4a5db612c327..7db28839108dd 100644 --- a/arch/x86/crypto/sha256-ssse3-asm.S +++ b/arch/x86/crypto/sha256-ssse3-asm.S @@ -47,6 +47,7 @@ ######################################################################## #include +#include ## assume buffers not aligned #define MOVDQ movdqu @@ -355,7 +356,7 @@ a = TMP_ ## arg 3 : Num blocks ######################################################################## .text -SYM_FUNC_START(sha256_transform_ssse3) +SYM_TYPED_FUNC_START(sha256_transform_ssse3) .align 32 pushq %rbx pushq %r12 diff --git a/arch/x86/crypto/sha256_ni_asm.S b/arch/x86/crypto/sha256_ni_asm.S index 94d50dd27cb53..47f93937f798a 100644 --- a/arch/x86/crypto/sha256_ni_asm.S +++ b/arch/x86/crypto/sha256_ni_asm.S @@ -54,6 +54,7 @@ */ #include +#include #define DIGEST_PTR %rdi /* 1st arg */ #define DATA_PTR %rsi /* 2nd arg */ @@ -97,7 +98,7 @@ .text .align 32 -SYM_FUNC_START(sha256_ni_transform) +SYM_TYPED_FUNC_START(sha256_ni_transform) shl $6, NUM_BLKS /* convert to bytes */ jz .Ldone_hash From patchwork Fri Nov 18 09:02:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 626244 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3BE9BC433FE for ; Fri, 18 Nov 2022 09:04:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241719AbiKRJER (ORCPT ); Fri, 18 Nov 2022 04:04:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54342 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241455AbiKRJEM (ORCPT ); Fri, 18 Nov 2022 04:04:12 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EEDDA7EC8F for ; Fri, 18 Nov 2022 01:04:09 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 431F1623BD for ; Fri, 18 Nov 2022 09:04:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EC565C43146; Fri, 18 Nov 2022 09:04:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668762249; bh=cvZrVNyaorZiFckp5I/DQO2Na69JdiQKH9GTwngz5kw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Txbmq1+8Os6ha8bRpkOEpMYl4yfovwRGORc/Q36bVbY0f/kt0jgLZs757o5lelPUR M4/GAy/RPij5/oInIsOl55+yuMlU2sQFJ05AcXKY4PudJAEycWR9iXNmcx1h5h3YNR 209MaRi+OoMvgNO4JI6rRGhf58knvpGrLNL6l5Kryv6wNaau6jo0dH5ItNYO++N6F1 PgC/p10LMij1YENAKkUG7CjntqP+312yJl+rZHiDardnnksxhPWLC1/rZoymXxqI5v dJORoyoJ2st87QAEJ/29Yi/9ygXONH/V34yN4BAKpsyZDJUpXzhspBzN7sSC19jbu0 KFZ777WpuVDDw== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: x86@kernel.org, linux-arm-kernel@lists.infradead.org, Sami Tolvanen Subject: [PATCH 06/11] crypto: x86/sha512 - fix possible crash with CFI enabled Date: Fri, 18 Nov 2022 01:02:15 -0800 Message-Id: <20221118090220.398819-7-ebiggers@kernel.org> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221118090220.398819-1-ebiggers@kernel.org> References: <20221118090220.398819-1-ebiggers@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org From: Eric Biggers sha512_transform_rorx(), sha512_transform_ssse3(), and sha512_transform_avx() are called via indirect function calls. These functions need to use SYM_TYPED_FUNC_START instead of SYM_FUNC_START to cause type hashes to be emitted when the kernel is built with CONFIG_CFI_CLANG=y. Otherwise, the code crashes with a CFI failure (if the compiler didn't happen to optimize out the indirect calls). Fixes: 3c516f89e17e ("x86: Add support for CONFIG_CFI_CLANG") Signed-off-by: Eric Biggers --- arch/x86/crypto/sha512-avx-asm.S | 3 ++- arch/x86/crypto/sha512-avx2-asm.S | 3 ++- arch/x86/crypto/sha512-ssse3-asm.S | 3 ++- 3 files changed, 6 insertions(+), 3 deletions(-) diff --git a/arch/x86/crypto/sha512-avx-asm.S b/arch/x86/crypto/sha512-avx-asm.S index 1fefe6dd3a9e2..b0984f19fdb40 100644 --- a/arch/x86/crypto/sha512-avx-asm.S +++ b/arch/x86/crypto/sha512-avx-asm.S @@ -48,6 +48,7 @@ ######################################################################## #include +#include .text @@ -273,7 +274,7 @@ frame_size = frame_WK + WK_SIZE # of SHA512 message blocks. # "blocks" is the message length in SHA512 blocks ######################################################################## -SYM_FUNC_START(sha512_transform_avx) +SYM_TYPED_FUNC_START(sha512_transform_avx) test msglen, msglen je nowork diff --git a/arch/x86/crypto/sha512-avx2-asm.S b/arch/x86/crypto/sha512-avx2-asm.S index 5cdaab7d69015..b1ca99055ef99 100644 --- a/arch/x86/crypto/sha512-avx2-asm.S +++ b/arch/x86/crypto/sha512-avx2-asm.S @@ -50,6 +50,7 @@ ######################################################################## #include +#include .text @@ -565,7 +566,7 @@ frame_size = frame_CTX + CTX_SIZE # of SHA512 message blocks. # "blocks" is the message length in SHA512 blocks ######################################################################## -SYM_FUNC_START(sha512_transform_rorx) +SYM_TYPED_FUNC_START(sha512_transform_rorx) # Save GPRs push %rbx push %r12 diff --git a/arch/x86/crypto/sha512-ssse3-asm.S b/arch/x86/crypto/sha512-ssse3-asm.S index b84c22e06c5f7..c06afb5270e5f 100644 --- a/arch/x86/crypto/sha512-ssse3-asm.S +++ b/arch/x86/crypto/sha512-ssse3-asm.S @@ -48,6 +48,7 @@ ######################################################################## #include +#include .text @@ -274,7 +275,7 @@ frame_size = frame_WK + WK_SIZE # of SHA512 message blocks. # "blocks" is the message length in SHA512 blocks. ######################################################################## -SYM_FUNC_START(sha512_transform_ssse3) +SYM_TYPED_FUNC_START(sha512_transform_ssse3) test msglen, msglen je nowork From patchwork Fri Nov 18 09:02:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 626240 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35B30C433FE for ; Fri, 18 Nov 2022 09:04:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241702AbiKRJEs (ORCPT ); Fri, 18 Nov 2022 04:04:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54400 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241712AbiKRJEQ (ORCPT ); Fri, 18 Nov 2022 04:04:16 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4815E13D4F for ; Fri, 18 Nov 2022 01:04:12 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id DE6D1B82274 for ; Fri, 18 Nov 2022 09:04:10 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8187AC43146; Fri, 18 Nov 2022 09:04:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668762249; bh=fc2JKYQatpTxXu11VT4a3V3fN6ivlyjI0KHBLOLHcEA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tWBHjlxK1blh/DEay6JaIGEL7wJIjBmtDxcPiGMfrHpdnkXk+LoFlctITaF3RXAYV 2yWhgt/dqlHzVPsnZthunPoK6jWayfFXHAD6LUeActAsOQ8ePYimylbS5Kz7BRh8oj hzDBNypHtok8kY0GHME3FQbOZZgA420/7KWbEY0uCwenBN4Kdp+SbSWcI2wcaITHuP 3vVdytMQFZYh14BrSbYisTfcmf8u34OleBaax7luuOy+9SX3OT7ImWs7FGdn2yoMBi KuLCzI2RD8DYd6sIbjyWH13Sa0UJqcfyybEMoAvCiY6CVmQ2BK2dHRazBnf4TQ4NEg pAKiN/iYuB9uA== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: x86@kernel.org, linux-arm-kernel@lists.infradead.org, Sami Tolvanen Subject: [PATCH 08/11] crypto: arm64/nhpoly1305 - eliminate unnecessary CFI wrapper Date: Fri, 18 Nov 2022 01:02:17 -0800 Message-Id: <20221118090220.398819-9-ebiggers@kernel.org> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221118090220.398819-1-ebiggers@kernel.org> References: <20221118090220.398819-1-ebiggers@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org From: Eric Biggers Since the CFI implementation now supports indirect calls to assembly functions, take advantage of that rather than use a wrapper function. Signed-off-by: Eric Biggers --- arch/arm64/crypto/nh-neon-core.S | 5 +++-- arch/arm64/crypto/nhpoly1305-neon-glue.c | 11 ++--------- 2 files changed, 5 insertions(+), 11 deletions(-) diff --git a/arch/arm64/crypto/nh-neon-core.S b/arch/arm64/crypto/nh-neon-core.S index 51c0a534ef87c..13eda08fda1e5 100644 --- a/arch/arm64/crypto/nh-neon-core.S +++ b/arch/arm64/crypto/nh-neon-core.S @@ -8,6 +8,7 @@ */ #include +#include KEY .req x0 MESSAGE .req x1 @@ -58,11 +59,11 @@ /* * void nh_neon(const u32 *key, const u8 *message, size_t message_len, - * u8 hash[NH_HASH_BYTES]) + * __le64 hash[NH_NUM_PASSES]) * * It's guaranteed that message_len % 16 == 0. */ -SYM_FUNC_START(nh_neon) +SYM_TYPED_FUNC_START(nh_neon) ld1 {K0.4s,K1.4s}, [KEY], #32 movi PASS0_SUMS.2d, #0 diff --git a/arch/arm64/crypto/nhpoly1305-neon-glue.c b/arch/arm64/crypto/nhpoly1305-neon-glue.c index c5405e6a6db76..cd882c35d9252 100644 --- a/arch/arm64/crypto/nhpoly1305-neon-glue.c +++ b/arch/arm64/crypto/nhpoly1305-neon-glue.c @@ -14,14 +14,7 @@ #include asmlinkage void nh_neon(const u32 *key, const u8 *message, size_t message_len, - u8 hash[NH_HASH_BYTES]); - -/* wrapper to avoid indirect call to assembly, which doesn't work with CFI */ -static void _nh_neon(const u32 *key, const u8 *message, size_t message_len, - __le64 hash[NH_NUM_PASSES]) -{ - nh_neon(key, message, message_len, (u8 *)hash); -} + __le64 hash[NH_NUM_PASSES]); static int nhpoly1305_neon_update(struct shash_desc *desc, const u8 *src, unsigned int srclen) @@ -33,7 +26,7 @@ static int nhpoly1305_neon_update(struct shash_desc *desc, unsigned int n = min_t(unsigned int, srclen, SZ_4K); kernel_neon_begin(); - crypto_nhpoly1305_update_helper(desc, src, n, _nh_neon); + crypto_nhpoly1305_update_helper(desc, src, n, nh_neon); kernel_neon_end(); src += n; srclen -= n; From patchwork Fri Nov 18 09:02:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 626241 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56D50C43217 for ; Fri, 18 Nov 2022 09:04:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241426AbiKRJEX (ORCPT ); Fri, 18 Nov 2022 04:04:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54394 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241710AbiKRJEQ (ORCPT ); Fri, 18 Nov 2022 04:04:16 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 61C3682BF5 for ; Fri, 18 Nov 2022 01:04:11 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id C2822623B2 for ; Fri, 18 Nov 2022 09:04:10 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1685FC433D6; Fri, 18 Nov 2022 09:04:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668762250; bh=ZkpnHZpH6zqpcaYDXylCMZLyYKMTqb6ZC8hlYT4Izo8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=R2q5ZahIINyliBIF3hGitjFBaxgXe56Xs/ajq8hQ9BzHkYYdLsTgyQc1vuiSh9+II A4s2sslrbAetPUZYuXi5H+VrV0fLTlcyAv5NX1+nP3zQvorkEFN2+IR+VvgvrTGcb6 lqNzs3zcmptAv2UmuXiYpir0qL+WcvzbMaSQCoONYTaYYZrp2pTMnwLmfzURkEXpSW eCFl4arg6ka2UyoIGdcva1Pko+SP+kvRAwlpDC8W8fqixvLqQMiwyvmygshZHgys2C r0mc2YrT5iakwAQg1E8sbyR3VsmN8nSZ1EXoIzio9gKli3pshLwXWcINSXMRfSaMEx jE58Qto0vDAjw== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: x86@kernel.org, linux-arm-kernel@lists.infradead.org, Sami Tolvanen Subject: [PATCH 10/11] crypto: arm/nhpoly1305 - eliminate unnecessary CFI wrapper Date: Fri, 18 Nov 2022 01:02:19 -0800 Message-Id: <20221118090220.398819-11-ebiggers@kernel.org> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221118090220.398819-1-ebiggers@kernel.org> References: <20221118090220.398819-1-ebiggers@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org From: Eric Biggers The arm architecture doesn't support CFI yet, and even if it did, the new CFI implementation supports indirect calls to assembly functions. Therefore, there's no need to use a wrapper function for nh_neon(). Signed-off-by: Eric Biggers --- arch/arm/crypto/nh-neon-core.S | 2 +- arch/arm/crypto/nhpoly1305-neon-glue.c | 11 ++--------- 2 files changed, 3 insertions(+), 10 deletions(-) diff --git a/arch/arm/crypto/nh-neon-core.S b/arch/arm/crypto/nh-neon-core.S index 434d80ab531c2..01620a0782ca9 100644 --- a/arch/arm/crypto/nh-neon-core.S +++ b/arch/arm/crypto/nh-neon-core.S @@ -69,7 +69,7 @@ /* * void nh_neon(const u32 *key, const u8 *message, size_t message_len, - * u8 hash[NH_HASH_BYTES]) + * __le64 hash[NH_NUM_PASSES]) * * It's guaranteed that message_len % 16 == 0. */ diff --git a/arch/arm/crypto/nhpoly1305-neon-glue.c b/arch/arm/crypto/nhpoly1305-neon-glue.c index ffa8d73fe722c..e93e41ff26566 100644 --- a/arch/arm/crypto/nhpoly1305-neon-glue.c +++ b/arch/arm/crypto/nhpoly1305-neon-glue.c @@ -14,14 +14,7 @@ #include asmlinkage void nh_neon(const u32 *key, const u8 *message, size_t message_len, - u8 hash[NH_HASH_BYTES]); - -/* wrapper to avoid indirect call to assembly, which doesn't work with CFI */ -static void _nh_neon(const u32 *key, const u8 *message, size_t message_len, - __le64 hash[NH_NUM_PASSES]) -{ - nh_neon(key, message, message_len, (u8 *)hash); -} + __le64 hash[NH_NUM_PASSES]); static int nhpoly1305_neon_update(struct shash_desc *desc, const u8 *src, unsigned int srclen) @@ -33,7 +26,7 @@ static int nhpoly1305_neon_update(struct shash_desc *desc, unsigned int n = min_t(unsigned int, srclen, SZ_4K); kernel_neon_begin(); - crypto_nhpoly1305_update_helper(desc, src, n, _nh_neon); + crypto_nhpoly1305_update_helper(desc, src, n, nh_neon); kernel_neon_end(); src += n; srclen -= n;