From patchwork Mon Oct 8 21:15:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 148445 Delivered-To: patch@linaro.org Received: by 2002:a2e:8595:0:0:0:0:0 with SMTP id b21-v6csp4125256lji; Mon, 8 Oct 2018 14:16:09 -0700 (PDT) X-Google-Smtp-Source: ACcGV61iqVvZj3TnGK3n2wr/ugFJ4OZnmMO5CNtoOwmOKrWrY0VWKoKkHUPls4BTmsz0KdU34ME1 X-Received: by 2002:a62:70c7:: with SMTP id l190-v6mr26639166pfc.186.1539033369543; Mon, 08 Oct 2018 14:16:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539033369; cv=none; d=google.com; s=arc-20160816; b=gdevJ7+I39AuxKtzp9TuMUP8rYPOuTZWCtIo4mZdw+moQ+4DqZtZHOKrX0hkMo7rcc leHADcrLcuiyRIVwkvrXNDU3clG5pTc8xMWOUnufZ/nlCSSJ1qDhdqummJoxk+QTSAVp rrCoQAAdyhnfge7I9lbpjyPJt4RVSlPYuRPHYim3J4GBQZwTimfc8sDIw+eldIjfqPhS cV8oO7+zoWHAp/P+F1bwLwiPnWDIptnsGWBw06HqhFxAYAcemDQdyRHS5+Is+pRuv3Qd eqCAFbSWfrrBY/NKipLflCTpIzqs+DCtz+M6K4CbudvayJNaLNCYzHefMfhQd0d7XfvT JeYw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=yVyxxH1ii59qYZF1tTV85yOwfr9+lABlveFtUiVcJ1o=; b=TeZjE9KrGoU5h/FYJ2MPtlUB37srAZqqF0Mi25xFtc5PxxpVIPjWhHwSRRosJ8jAys DPNdGvOFLWtQT1SM9PypQbePWl0Kd591jBlzvTdWmty2ROFg0XsZUL4zinnJtGtS4Vk0 0NNnkULO7YAgQWgeSpxZsbVitjKqxbKM9oHQSPK/7DCB0Ggqi4CDpyvEgYdqf/iOJ7r6 rEiV1vTkBaSCz0tGpo1Cpvl4BGEy9gUSc811J9pcA6EeBuwB3OMDdCYA5PvEfOOtPVxo rgqWA/qNBaKPdxY3Z6hHMGalNfBQ3DStx3vhuf48wf8+DethNBd37lUmo4dJ7MTCQwcJ oPFA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=PBqbW6uT; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id w4-v6si18552811pll.214.2018.10.08.14.16.09; Mon, 08 Oct 2018 14:16:09 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=PBqbW6uT; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726830AbeJIE3t (ORCPT + 2 others); Tue, 9 Oct 2018 00:29:49 -0400 Received: from mail-wm1-f67.google.com ([209.85.128.67]:40723 "EHLO mail-wm1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725828AbeJIE3t (ORCPT ); Tue, 9 Oct 2018 00:29:49 -0400 Received: by mail-wm1-f67.google.com with SMTP id z204-v6so9612029wmc.5 for ; Mon, 08 Oct 2018 14:16:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=yVyxxH1ii59qYZF1tTV85yOwfr9+lABlveFtUiVcJ1o=; b=PBqbW6uTfN5iffbPRE1SIC5r3SoXgOP4NpaaUNKHUlamM5mbc7Y7g5XrgIAlvLYQrN lL+nGpFu9zOhdoBJsSE4TIyUcq5aX+0vRITgY4CXvXiWROA5n6RutE5HQ7av3uDRu4PJ CrcyaB/LWlQR5HTRec71Ei5nAmBeBTtuqjJEA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=yVyxxH1ii59qYZF1tTV85yOwfr9+lABlveFtUiVcJ1o=; b=LYntiDLxevbeAYBv+ujuIsjN25arHhBUISpb5xfqOZ1/EA3aPn8YOFlVJjbUHpj+G1 RDDJttY+NJZ0neTxXOc3b7jouoHxlB+6WMX8r0XFxGosPgUGfTK4yIwvdCTiZho6drpS bFTS/SA9ct5HoIJrT6GIUi36iEsNrEdidh8QxtEvSEmGSP4BiIEbEUVpAls3j0db2kbH 2CqOx0iqbvW8XYqvFmQNr65NpEXAIAF6euT3mVPwIc8MdOEwiZZc97Fk/ZwhNscJJPI/ DveAvZMbgWnHIvaon8PNfaI0gBljhMESx51CR17st1/n3kiUY+DnihUc35lX8Qujoe2F trmg== X-Gm-Message-State: ABuFfohjob3VVirn4AbHlY3H6HHemWeIgnfp8RPWhT/Fwcf/+/FK/1do vqcSwneQrZ2HElmV3PepsoU2zHiKIgw= X-Received: by 2002:a1c:b58e:: with SMTP id e136-v6mr15619838wmf.114.1539033366609; Mon, 08 Oct 2018 14:16:06 -0700 (PDT) Received: from localhost.localdomain ([2a01:cb1d:112:6f00:8084:9715:d038:c67d]) by smtp.gmail.com with ESMTPSA id s24-v6sm7563308wmc.7.2018.10.08.14.16.05 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 08 Oct 2018 14:16:05 -0700 (PDT) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, arnd@arndb.de, jason@zx2c4.com, ebiggers@google.com, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel Subject: [PATCH 1/3] crypto: memneq - use unaligned accessors for aligned fast path Date: Mon, 8 Oct 2018 23:15:52 +0200 Message-Id: <20181008211554.5355-2-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20181008211554.5355-1-ard.biesheuvel@linaro.org> References: <20181008211554.5355-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org On ARM v6 and later, we define CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS because the ordinary load/store instructions (ldr, ldrh, ldrb) can tolerate any misalignment of the memory address. However, load/store double and load/store multiple instructions (ldrd, ldm) may still only be used on memory addresses that are 32-bit aligned, and so we have to use the CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS macro with care, or we may end up with a severe performance hit due to alignment traps that require fixups by the kernel. Fortunately, the get_unaligned() accessors do the right thing: when building for ARMv6 or later, the compiler will emit unaligned accesses using the ordinary load/store instructions (but avoid the ones that require 32-bit alignment). When building for older ARM, those accessors will emit the appropriate sequence of ldrb/mov/orr instructions. And on architectures that can truly tolerate any kind of misalignment, the get_unaligned() accessors resolve to the leXX_to_cpup accessors that operate on aligned addresses. So switch to the unaligned accessors for the aligned fast path. This will create the exact same code on architectures that can really tolerate any kind of misalignment, and generate code for ARMv6+ that avoids load/store instructions that trigger alignment faults. Signed-off-by: Ard Biesheuvel --- crypto/memneq.c | 24 ++++++++++++++------ 1 file changed, 17 insertions(+), 7 deletions(-) -- 2.11.0 diff --git a/crypto/memneq.c b/crypto/memneq.c index afed1bd16aee..0f46a6150f22 100644 --- a/crypto/memneq.c +++ b/crypto/memneq.c @@ -60,6 +60,7 @@ */ #include +#include #ifndef __HAVE_ARCH_CRYPTO_MEMNEQ @@ -71,7 +72,10 @@ __crypto_memneq_generic(const void *a, const void *b, size_t size) #if defined(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) while (size >= sizeof(unsigned long)) { - neq |= *(unsigned long *)a ^ *(unsigned long *)b; + unsigned long const *p = a; + unsigned long const *q = b; + + neq |= get_unaligned(p) ^ get_unaligned(q); OPTIMIZER_HIDE_VAR(neq); a += sizeof(unsigned long); b += sizeof(unsigned long); @@ -95,18 +99,24 @@ static inline unsigned long __crypto_memneq_16(const void *a, const void *b) #ifdef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS if (sizeof(unsigned long) == 8) { - neq |= *(unsigned long *)(a) ^ *(unsigned long *)(b); + unsigned long const *p = a; + unsigned long const *q = b; + + neq |= get_unaligned(p++) ^ get_unaligned(q++); OPTIMIZER_HIDE_VAR(neq); - neq |= *(unsigned long *)(a+8) ^ *(unsigned long *)(b+8); + neq |= get_unaligned(p) ^ get_unaligned(q); OPTIMIZER_HIDE_VAR(neq); } else if (sizeof(unsigned int) == 4) { - neq |= *(unsigned int *)(a) ^ *(unsigned int *)(b); + unsigned int const *p = a; + unsigned int const *q = b; + + neq |= get_unaligned(p++) ^ get_unaligned(q++); OPTIMIZER_HIDE_VAR(neq); - neq |= *(unsigned int *)(a+4) ^ *(unsigned int *)(b+4); + neq |= get_unaligned(p++) ^ get_unaligned(q++); OPTIMIZER_HIDE_VAR(neq); - neq |= *(unsigned int *)(a+8) ^ *(unsigned int *)(b+8); + neq |= get_unaligned(p++) ^ get_unaligned(q++); OPTIMIZER_HIDE_VAR(neq); - neq |= *(unsigned int *)(a+12) ^ *(unsigned int *)(b+12); + neq |= get_unaligned(p) ^ get_unaligned(q); OPTIMIZER_HIDE_VAR(neq); } else #endif /* CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS */