From patchwork Mon Dec 4 12:26:37 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 120521 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp4365830qgn; Mon, 4 Dec 2017 04:27:34 -0800 (PST) X-Google-Smtp-Source: AGs4zMZ78e1QwV8xEXhx7c9F9zAteI6PAGCoIvOjQM2SN8hF1wyhF5I+iTisbjSLdmiC6mnAI3vT X-Received: by 10.101.102.66 with SMTP id z2mr14013510pgv.352.1512390454403; Mon, 04 Dec 2017 04:27:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512390454; cv=none; d=google.com; s=arc-20160816; b=L+aNCmZzcblnhphvBLF2pZuuE/PV/mXBzVzWBKYnDXgDj0fEBO47Mv/IaxLJE/lb/g ktF/AX0LbAsyvybxDzulEZ1HMn383YrbLRYR2BIYtSh3zlswAk0YGpI3gOT715MePtjE OXNAVAg3wB0dns9PE2IXrCUehlq5+e/tIzdxOwvFuVmtdmOlPwaXiqfYAV27ZGus1foW s4MCRFjggYyK2cj71ELNLtSkMRAV4/5My1xABAPYQ47J58MBcfec/ajZnMUC3GnOC8y1 vtrpq2wQBWB2kcSRUIerbEorjbsI675yREOCgr6BodNXuIUiA4MFPHgAOa99QeJNDp24 N2Eg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=iLnvUryj6wgRmRZyo3vDg6ZHKz55Pt1BEioZv1Vwiy8=; b=nxB2Z8ej4hhZFLZNdJpONw/jmMOy5ZKxsiYhfuHcysj3qwFrl53vOeiTD/NVGKEVg2 wHZq+y/f0PCPC6v1x3YCHKkIv5ylHCY5SlQa+I9xoF6wQHlgtQwrK6ae/zttVo8bmYyx po5cKfB946CiHtsJurIgISDGnjNUAP+KWI9gZ68oRyiEcpnr11jdVRv9Twx6bqB4SyDJ 7ZEOaI/FH4JgIdQ8bvYmrELNPvikgTop1E1g9TUVl4GPRqR7cFO1GFNEGS/0gk0Ftikv jmGz7tOGF0+aiilN6cCcZ1kj5OsD2mbehElVU6U8C9gTv43kDTb4SzueIbBhpI4tk5ne o35A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=RNFQ6A9W; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id d23si10177768pfe.339.2017.12.04.04.27.34; Mon, 04 Dec 2017 04:27:34 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=RNFQ6A9W; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753275AbdLDM1c (ORCPT + 1 other); Mon, 4 Dec 2017 07:27:32 -0500 Received: from mail-wm0-f66.google.com ([74.125.82.66]:45616 "EHLO mail-wm0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753271AbdLDM13 (ORCPT ); Mon, 4 Dec 2017 07:27:29 -0500 Received: by mail-wm0-f66.google.com with SMTP id 9so5448131wme.4 for ; Mon, 04 Dec 2017 04:27:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=iLnvUryj6wgRmRZyo3vDg6ZHKz55Pt1BEioZv1Vwiy8=; b=RNFQ6A9WBnbvP5M4fVOCAJpZH4ICe6gFetdbaxRv+3k/VjNUOFjrtlOpAkXrk79Xyc DxOmUjE4C0YiYnKhqS0ZYK6ft1DGvGI3Rfr4c3IwiqQeAuiB6zqpg5GkfgZoMZTGOJkg Sj6u4HwE2OG6/e0VYYzYHTW5gkomlrQJPDtfw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=iLnvUryj6wgRmRZyo3vDg6ZHKz55Pt1BEioZv1Vwiy8=; b=jyKxAreH9MggLfeiHqDWxWTITIyRfKXCS91u8/SoLZ3qlD6zoAzycQgpwKHsfFwG6h ZNd51U1GDDLQDau8p7S3sgPtZzws2Yg0qh4K21x1FUHU8KAtY//qzgLri+4g7fWiuaVr WJB89748j6F+2rnVwg5bxeLXpUHFOa6HiKxzhTRyf0CyY7+0gNYpYy1vNEi0Cw6rRijG 5EkUJabj0QKUySWnEYF0+jUEw7VtyxOd9wx8P4BuIoQYAitw0JYZ9qJRm7x4ENakAUdW hJxkdWDP9PLFL/bBlwqW2IT+GTEcXMneqamybanNfEEinU21CVkyQ810r78ZyYQWOMhS ks4A== X-Gm-Message-State: AJaThX5Gfm93jrJ31XpeFTPXt/629rE2vkr3CTBmv40aKGfuPUHWPj7J M4mSRWh23ngFivI3X+ppNpkWINSbxaw= X-Received: by 10.28.198.139 with SMTP id w133mr8048759wmf.13.1512390447568; Mon, 04 Dec 2017 04:27:27 -0800 (PST) Received: from localhost.localdomain ([105.150.171.234]) by smtp.gmail.com with ESMTPSA id a8sm7665839wmh.41.2017.12.04.04.27.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 04 Dec 2017 04:27:26 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v2 11/19] arm64: assembler: add macro to conditionally yield the NEON under PREEMPT Date: Mon, 4 Dec 2017 12:26:37 +0000 Message-Id: <20171204122645.31535-12-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171204122645.31535-1-ard.biesheuvel@linaro.org> References: <20171204122645.31535-1-ard.biesheuvel@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Add a support macro to conditionally yield the NEON (and thus the CPU) that may be called from the assembler code. Given that especially the instruction based accelerated crypto code may use very tight loops, add some parametrization so that the TIF_NEED_RESCHED flag test is only executed every so many loop iterations. In some cases, yielding the NEON involves saving and restoring a non trivial amount of context (especially in the CRC folding algorithms), and so the macro is split into two, and the code in between is only executed when the yield path is taken, allowing the contex to be preserved. The second macro takes a label argument that marks the resume-from-yield path, which should restore the preserved context again. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 50 ++++++++++++++++++++ 1 file changed, 50 insertions(+) -- 2.11.0 diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index aef72d886677..917b026d3e00 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -512,4 +512,54 @@ alternative_else_nop_endif #endif .endm +/* + * yield_neon - check whether to yield to another runnable task from + * kernel mode NEON code (running with preemption disabled) + * + * - Check whether the preempt count is exactly 1, in which case disabling + * preemption once will make the task preemptible. If this is not the case, + * yielding is pointless. + * - Check whether TIF_NEED_RESCHED is set, and if so, disable and re-enable + * kernel mode NEON (which will trigger a reschedule), and branch to the + * yield fixup code at @lbl. + */ + .macro yield_neon, lbl:req, ctr, order, stride, loop + yield_neon_pre \ctr, \order, \stride, \loop + yield_neon_post \lbl + .endm + + .macro yield_neon_pre, ctr, order=0, stride, loop=4444f +#ifdef CONFIG_PREEMPT + /* + * With some algorithms, it makes little sense to poll the + * TIF_NEED_RESCHED flag after every iteration, so only perform + * the check every 2^order strides. + */ + .if \order > 1 + .if (\stride & (\stride - 1)) != 0 + .error "stride should be a power of 2" + .endif + tst \ctr, #((1 << \order) * \stride - 1) & ~(\stride - 1) + b.ne \loop + .endif + + get_thread_info x0 + ldr w1, [x0, #TSK_TI_PREEMPT] + ldr x0, [x0, #TSK_TI_FLAGS] + cmp w1, #1 // == PREEMPT_OFFSET + csel x0, x0, xzr, eq + tbnz x0, #TIF_NEED_RESCHED, 5555f // needs rescheduling? +4444: +#endif + .subsection 1 +5555: + .endm + + .macro yield_neon_post, lbl:req + bl kernel_neon_end + bl kernel_neon_begin + b \lbl + .previous + .endm + #endif /* __ASM_ASSEMBLER_H */