From patchwork Sat Jan 6 01:09:58 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 123593 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp40495qgn; Fri, 5 Jan 2018 17:18:18 -0800 (PST) X-Google-Smtp-Source: ACJfBotUKAuxlPnNGVaSe76t1wK6tJ7TQKO7Vqcf37rFDqNz7nS8iTY4GptuhK9n9x+UGvWIBaJL X-Received: by 10.99.150.18 with SMTP id c18mr3906091pge.136.1515201498786; Fri, 05 Jan 2018 17:18:18 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515201498; cv=none; d=google.com; s=arc-20160816; b=r1uaQb2xeVp0vKrfhv8EmOhwZsDIiDU1IHkXD++TtjEX4O761iifp4QrXtHd+HJkd0 ixtetGlf7xu/4yF5WK9r6pNyIta382BrhAnFX7UdaM3yLaRtNIb3TfnYBp7c20DmiEut qJoIngLY3n3Rkeir7Vnp/pVLoFZrpPrdub8I177t8h7QFSsCnWFCV6b4aGMYmjFy3PkP wKVir0JTnnt0TrBGWcFpw/GNbrUMcl2ew8cCOGZkMmpSJ9ygTp9/kylzkNZLpZbLnmDs WmWGCPL7YzMBCi3q41XiNs5jr7CpiE7cNzl6vIxW3ANIVvWVrd/a43iqyh+ae7lS3AxE 2v3w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:cc:to:from :subject:arc-authentication-results; bh=hYwPUuYE5peQbZpkuqBNMM1rLdtu/XXphsiVeAI6SM0=; b=Ij2n+P9kpPeYcbqkUD155fA6ws+JlOEeKac7uq9XqpR9+RUBhOcSAkUmARmXqF2gns y/rpugkUfB4kAKudiKYEvcpvfvtOUz7BpiUKhY0ev+H8QcodP6Cb0iz1cJz0XSyqrSeN fiWZ8HRfA907Mtcss8TXYQgn12UnhX9n+fJcwzz1zJlzDH442YAHa6vOmMaxzYKlyqww C+UPH6Wa+902qIAwrw3CaVyKKh1iFfyKfpfmYYNjKAXwQlmwQ3hLoh2hvn4BetcPiZIu pxq4xXowbJ/3G6SUmgZGThqJFbg0JGr3BPTNWTOEKsCwFFnLUFTamQN9jjt5QoTiC5Dr XiOA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id k1si1803835pgo.823.2018.01.05.17.18.17; Fri, 05 Jan 2018 17:18:18 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753604AbeAFBSQ (ORCPT + 28 others); Fri, 5 Jan 2018 20:18:16 -0500 Received: from mga06.intel.com ([134.134.136.31]:15544 "EHLO mga06.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753389AbeAFBSM (ORCPT ); Fri, 5 Jan 2018 20:18:12 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Jan 2018 17:18:11 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.46,320,1511856000"; d="scan'208";a="190352416" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.16]) by orsmga005.jf.intel.com with ESMTP; 05 Jan 2018 17:18:11 -0800 Subject: [PATCH 01/18] asm-generic/barrier: add generic nospec helpers From: Dan Williams To: linux-kernel@vger.kernel.org Cc: Mark Rutland , linux-arch@vger.kernel.org, peterz@infradead.org, netdev@vger.kernel.org, Will Deacon , gregkh@linuxfoundation.org, tglx@linutronix.de, torvalds@linux-foundation.org, alan@linux.intel.com Date: Fri, 05 Jan 2018 17:09:58 -0800 Message-ID: <151520099810.32271.11023910901471332353.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <151520099201.32271.4677179499894422956.stgit@dwillia2-desk3.amr.corp.intel.com> References: <151520099201.32271.4677179499894422956.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.17.1-9-g687f MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Mark Rutland Under speculation, CPUs may mis-predict branches in bounds checks. Thus, memory accesses under a bounds check may be speculated even if the bounds check fails, providing a primitive for building a side channel. This patch adds helpers which can be used to inhibit the use of out-of-bounds pointers under speculation. A generic implementation is provided for compatibility, but does not guarantee safety under speculation. Architectures are expected to override these helpers as necessary. Signed-off-by: Mark Rutland Signed-off-by: Will Deacon Cc: Daniel Willams Cc: Peter Zijlstra Signed-off-by: Dan Williams --- include/asm-generic/barrier.h | 68 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+) diff --git a/include/asm-generic/barrier.h b/include/asm-generic/barrier.h index fe297b599b0a..91c3071f49e5 100644 --- a/include/asm-generic/barrier.h +++ b/include/asm-generic/barrier.h @@ -54,6 +54,74 @@ #define read_barrier_depends() do { } while (0) #endif +/* + * Inhibit subsequent speculative memory accesses. + * + * Architectures with a suitable memory barrier should provide an + * implementation. This is non-portable, and generic code should use + * nospec_ptr(). + */ +#ifndef __nospec_barrier +#define __nospec_barrier() do { } while (0) +#endif + +/** + * nospec_ptr() - Ensure a pointer is bounded, even under speculation. + * + * @ptr: the pointer to test + * @lo: the lower valid bound for @ptr, inclusive + * @hi: the upper valid bound for @ptr, exclusive + * + * If @ptr falls in the interval [@lo, @i), returns @ptr, otherwise returns + * NULL. + * + * Architectures which do not provide __nospec_barrier() should override this + * to ensure that ptr falls in the [lo, hi) interval both under architectural + * execution and under speculation, preventing propagation of an out-of-bounds + * pointer to code which is speculatively executed. + */ +#ifndef nospec_ptr +#define nospec_ptr(ptr, lo, hi) \ +({ \ + typeof (ptr) __ret; \ + typeof (ptr) __ptr = (ptr); \ + typeof (ptr) __lo = (lo); \ + typeof (ptr) __hi = (hi); \ + \ + __ret = (__lo <= __ptr && __ptr < __hi) ? __ptr : NULL; \ + \ + __nospec_barrier(); \ + \ + __ret; \ +}) +#endif + +/** + * nospec_array_ptr - Generate a pointer to an array element, ensuring the + * pointer is bounded under speculation. + * + * @arr: the base of the array + * @idx: the index of the element + * @sz: the number of elements in the array + * + * If @idx falls in the interval [0, @sz), returns the pointer to @arr[@idx], + * otherwise returns NULL. + * + * This is a wrapper around nospec_ptr(), provided for convenience. + * Architectures should implement nospec_ptr() to ensure this is the case + * under speculation. + */ +#define nospec_array_ptr(arr, idx, sz) \ +({ \ + typeof(*(arr)) *__arr = (arr); \ + typeof(idx) __idx = (idx); \ + typeof(sz) __sz = (sz); \ + \ + nospec_ptr(__arr + __idx, __arr, __arr + __sz); \ +}) + +#undef __nospec_barrier + #ifndef __smp_mb #define __smp_mb() mb() #endif From patchwork Sat Jan 6 01:10:03 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 123594 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp40617qgn; Fri, 5 Jan 2018 17:18:27 -0800 (PST) X-Google-Smtp-Source: ACJfBotV1Sn4SCkESG4UFGKSUxWLyRV7vqB9/amIsOxuct3ptFymLjcfKwLGC7XxyRdg7sfFc5Fy X-Received: by 10.84.240.72 with SMTP id h8mr4859631plt.412.1515201507101; Fri, 05 Jan 2018 17:18:27 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515201507; cv=none; d=google.com; s=arc-20160816; b=DByfFbnOmSstdh2hu0jcbZmyb704GuKFllWutXGF0ayTtxZAscjsWDmPcqPyHNYAr0 bSLRYjhCNLwXhwcQK+xLBqRHVZ1B5flGkvgKGrUL2+Q9o8bRnxu8ld6J4sLJHu6/wAGE mB61ABJGtpK291iCjKzMIXZssLu1jOHJc4w3FtiqQn3kBSXYWX1I622kfC/ocjHTZpJe x66IB2ZAB5aio6GMzQk0hX4Fk+CL2jbgKgzzKEBBrWutPd28vIouqSgZLTaXYAPuOSNC rqNBCBbTleIOgFL1oXe1abTPLCmxPLWk4xZJSJjUKUEQIZImSwe1MromNDdO685KSyjX hdiA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:cc:to:from :subject:arc-authentication-results; bh=xLZRs8SeMOch95CoKfI1Xxx4r1IzTXtxGw+TnS5IioE=; b=Erje97TmudLrkmAz1sE5h5FxfP8V3G2JPmqCJdWIfhJbs3d9ji9p/iG/pqT6lXhGaS il1Sb8TarEpougpqYiXTcwxSja2gHO/u9w/mlJy1aFrxc5SUGMM6hcHX8j9FsfWHFCoW EIrmHTWpkmBNrWH2K9hqUfBguxMJdbV2nHAOkcRozlthsxJxG2oFP7DldMenV9liVqoZ few6yAqaiKSyX6M2f/43jP5tOhx/pzESQfSNTkWp8/i5FaIA8iugHUaYFxGilPkHdP1F 5QxU5TJqP85kK8hBDDwDEDljp/WAmhiudNRChCTLGNE8TAJ3ghSV1GRJZ0t5xzjLI6HA /8vw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id v11si1903656pgf.326.2018.01.05.17.18.24; Fri, 05 Jan 2018 17:18:27 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753639AbeAFBSW (ORCPT + 28 others); Fri, 5 Jan 2018 20:18:22 -0500 Received: from mga03.intel.com ([134.134.136.65]:7023 "EHLO mga03.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753608AbeAFBSS (ORCPT ); Fri, 5 Jan 2018 20:18:18 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Jan 2018 17:18:17 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.46,320,1511856000"; d="scan'208";a="7614989" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.16]) by fmsmga007.fm.intel.com with ESMTP; 05 Jan 2018 17:18:17 -0800 Subject: [PATCH 02/18] Documentation: document nospec helpers From: Dan Williams To: linux-kernel@vger.kernel.org Cc: Mark Rutland , linux-arch@vger.kernel.org, Jonathan Corbet , peterz@infradead.org, netdev@vger.kernel.org, Will Deacon , gregkh@linuxfoundation.org, tglx@linutronix.de, torvalds@linux-foundation.org, alan@linux.intel.com Date: Fri, 05 Jan 2018 17:10:03 -0800 Message-ID: <151520100323.32271.8384226462583945132.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <151520099201.32271.4677179499894422956.stgit@dwillia2-desk3.amr.corp.intel.com> References: <151520099201.32271.4677179499894422956.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.17.1-9-g687f MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Mark Rutland Document the rationale and usage of the new nospec*() helpers. Signed-off-by: Mark Rutland Signed-off-by: Will Deacon Cc: Dan Williams Cc: Jonathan Corbet Cc: Peter Zijlstra Signed-off-by: Dan Williams --- Documentation/speculation.txt | 166 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 166 insertions(+) create mode 100644 Documentation/speculation.txt diff --git a/Documentation/speculation.txt b/Documentation/speculation.txt new file mode 100644 index 000000000000..748fcd4dcda4 --- /dev/null +++ b/Documentation/speculation.txt @@ -0,0 +1,166 @@ +This document explains potential effects of speculation, and how undesirable +effects can be mitigated portably using common APIs. + +=========== +Speculation +=========== + +To improve performance and minimize average latencies, many contemporary CPUs +employ speculative execution techniques such as branch prediction, performing +work which may be discarded at a later stage. + +Typically speculative execution cannot be observed from architectural state, +such as the contents of registers. However, in some cases it is possible to +observe its impact on microarchitectural state, such as the presence or +absence of data in caches. Such state may form side-channels which can be +observed to extract secret information. + +For example, in the presence of branch prediction, it is possible for bounds +checks to be ignored by code which is speculatively executed. Consider the +following code: + + int load_array(int *array, unsigned int idx) { + if (idx >= MAX_ARRAY_ELEMS) + return 0; + else + return array[idx]; + } + +Which, on arm64, may be compiled to an assembly sequence such as: + + CMP , #MAX_ARRAY_ELEMS + B.LT less + MOV , #0 + RET + less: + LDR , [, ] + RET + +It is possible that a CPU mis-predicts the conditional branch, and +speculatively loads array[idx], even if idx >= MAX_ARRAY_ELEMS. This value +will subsequently be discarded, but the speculated load may affect +microarchitectural state which can be subsequently measured. + +More complex sequences involving multiple dependent memory accesses may result +in sensitive information being leaked. Consider the following code, building on +the prior example: + + int load_dependent_arrays(int *arr1, int *arr2, int idx) { + int val1, val2, + + val1 = load_array(arr1, idx); + val2 = load_array(arr2, val1); + + return val2; + } + +Under speculation, the first call to load_array() may return the value of an +out-of-bounds address, while the second call will influence microarchitectural +state dependent on this value. This may provide an arbitrary read primitive. + +==================================== +Mitigating speculation side-channels +==================================== + +The kernel provides a generic API to ensure that bounds checks are respected +even under speculation. Architectures which are affected by speculation-based +side-channels are expected to implement these primitives. + +The following helpers found in can be used to prevent +information from being leaked via side-channels. + +* nospec_ptr(ptr, lo, hi) + + Returns a sanitized pointer that is bounded by the [lo, hi) interval. When + ptr < lo, or ptr >= hi, NULL is returned. Prevents an out-of-bounds pointer + being propagated to code which is speculatively executed. + + This is expected to be used by code which computes pointers to data + structures, where part of the address (such as an array index) may be + user-controlled. + + This can be used to protect the earlier load_array() example: + + int load_array(int *array, unsigned int idx) + { + int *elem; + + if ((elem = nospec_ptr(array + idx, array, array + MAX_ARRAY_ELEMS))) + return *elem; + else + return 0; + } + + This can also be used in situations where multiple fields on a structure are + accessed: + + struct foo array[SIZE]; + int a, b; + + void do_thing(int idx) + { + struct foo *elem; + + if ((elem = nospec_ptr(array + idx, array, array + SIZE)) { + a = elem->field_a; + b = elem->field_b; + } + } + + It is imperative that the returned pointer is used. Pointers which are + generated separately are subject to a number of potential CPU and compiler + optimizations, and may still be used speculatively. For example, this means + that the following sequence is unsafe: + + struct foo array[SIZE]; + int a, b; + + void do_thing(int idx) + { + if (nospec_ptr(array + idx, array, array + SIZE) != NULL) { + // unsafe as wrong pointer is used + a = array[idx].field_a; + b = array[idx].field_b; + } + } + + Similarly, it is unsafe to compare the returned pointer with other pointers, + as this may permit the compiler to substitute one pointer with another, + permitting speculation. For example, the following sequence is unsafe: + + struct foo array[SIZE]; + int a, b; + + void do_thing(int idx) + { + struct foo *elem = nospec_ptr(array + idx, array, array + size); + + // unsafe due to pointer substitution + if (elem == &array[idx]) { + a = elem->field_a; + b = elem->field_b; + } + } + +* nospec_array_ptr(arr, idx, sz) + + Returns a sanitized pointer to arr[idx] only if idx falls in the [0, sz) + interval. When idx < 0 or idx > sz, NULL is returned. Prevents an + out-of-bounds pointer being propagated to code which is speculatively + executed. + + This is a convenience function which wraps nospec_ptr(), and has the same + caveats w.r.t. the use of the returned pointer. + + For example, this may be used as follows: + + int load_array(int *array, unsigned int idx) + { + int *elem; + + if ((elem = nospec_array_ptr(array, idx, MAX_ARRAY_ELEMS))) + return *elem; + else + return 0; + } + From patchwork Sat Jan 6 01:10:09 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 123595 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp40697qgn; Fri, 5 Jan 2018 17:18:33 -0800 (PST) X-Google-Smtp-Source: ACJfBou39faF94SUgRLigPKTAX8oOzrjW7ad4VShosEpWcAggTaTHn6Phbv0rq+Ai1Fo7YzTLlob X-Received: by 10.84.133.226 with SMTP id f89mr4809444plf.407.1515201513175; Fri, 05 Jan 2018 17:18:33 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515201513; cv=none; d=google.com; s=arc-20160816; b=00FDFYyT/0TiEg2poJGXVYVvW0N5A7xzPhWVqT6toeIXdZ8jqPg+v3sdHiOOxIlhba X8G4LGfdHIMyfbD0YhxK1+Gg3kPbW8JD0aeL4Lj/XETtYyLBb0erH3jWoTwW/unHLyWq 7ucSamUR14GUjSIIC/RnmOUvk+gsZNAnloJ2y1c5FZncj+pLoO6KuaMXkkCHsmWRe6A6 bHqfG5t5w8/eMuKSQQnVuD5GhC95oRRssPZ6FhiZqUvIk9dWAGbpbp1mhNppWxqLMtDV LIlqYT6xAPl5v1RwBlg+Uku71Tafe1pWKvhOpKBByQBvKKiCjcSnyV1OJXk9my9UhP5u eFRA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:cc:to:from :subject:arc-authentication-results; bh=O3BUN/+LDrqnRPf6nu5PF+cb+Z+R4oaAWxmNEYoItf4=; b=qdyjTYIk8zdEZ8a4v3BgZgA/wXJT76Vit2w/CUxVp78Jaa1Jj4B83bV+AWjdOHvn6s AUOER93p3gZROD/6rctyfVif8szrHojskKd6xSiNtr1a/qsTInXVxAIecaP+Ir/qrFqs 0ktrfc2uivl0oEHptC+hp5/KMEmBUuVB0xrcPDgnF49v45mWngthrFyzB/amdSFW7T9L M/OgyWuOnFVihvksXK0wYyM+f/UEJld5I1W+VTig5m4Y6OlsvJAgXsieBvHqfI7zwFdc Fne0ye7iHsoQY36BS6RErETVwkYxypr67ms5RASq7ewmvdO3h4KeqPEnoF+1JSMDGPEO 2w3g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id v11si1903656pgf.326.2018.01.05.17.18.32; Fri, 05 Jan 2018 17:18:33 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753673AbeAFBSa (ORCPT + 28 others); Fri, 5 Jan 2018 20:18:30 -0500 Received: from mga06.intel.com ([134.134.136.31]:15560 "EHLO mga06.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753640AbeAFBSY (ORCPT ); Fri, 5 Jan 2018 20:18:24 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Jan 2018 17:18:23 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.46,320,1511856000"; d="scan'208";a="192645547" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.16]) by fmsmga005.fm.intel.com with ESMTP; 05 Jan 2018 17:18:22 -0800 Subject: [PATCH 03/18] arm64: implement nospec_ptr() From: Dan Williams To: linux-kernel@vger.kernel.org Cc: Mark Rutland , linux-arch@vger.kernel.org, peterz@infradead.org, netdev@vger.kernel.org, Will Deacon , gregkh@linuxfoundation.org, tglx@linutronix.de, torvalds@linux-foundation.org, alan@linux.intel.com Date: Fri, 05 Jan 2018 17:10:09 -0800 Message-ID: <151520100890.32271.10501551400360324504.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <151520099201.32271.4677179499894422956.stgit@dwillia2-desk3.amr.corp.intel.com> References: <151520099201.32271.4677179499894422956.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.17.1-9-g687f MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Mark Rutland This patch implements nospec_ptr() for arm64, following the recommended architectural sequence. Signed-off-by: Mark Rutland Signed-off-by: Will Deacon Cc: Dan Williams Cc: Peter Zijlstra Signed-off-by: Dan Williams --- arch/arm64/include/asm/barrier.h | 55 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/arch/arm64/include/asm/barrier.h b/arch/arm64/include/asm/barrier.h index 77651c49ef44..b4819f6a0e5c 100644 --- a/arch/arm64/include/asm/barrier.h +++ b/arch/arm64/include/asm/barrier.h @@ -40,6 +40,61 @@ #define dma_rmb() dmb(oshld) #define dma_wmb() dmb(oshst) +#define __load_no_speculate_n(ptr, lo, hi, failval, cmpptr, w, sz) \ +({ \ + typeof(*ptr) __nln_val; \ + typeof(*ptr) __failval = \ + (typeof(*ptr))(unsigned long)(failval); \ + \ + asm volatile ( \ + " cmp %[c], %[l]\n" \ + " ccmp %[c], %[h], 2, cs\n" \ + " b.cs 1f\n" \ + " ldr" #sz " %" #w "[v], %[p]\n" \ + "1: csel %" #w "[v], %" #w "[v], %" #w "[f], cc\n" \ + " hint #0x14 // CSDB\n" \ + : [v] "=&r" (__nln_val) \ + : [p] "m" (*(ptr)), [l] "r" (lo), [h] "r" (hi), \ + [f] "rZ" (__failval), [c] "r" (cmpptr) \ + : "cc"); \ + \ + __nln_val; \ +}) + +#define __load_no_speculate(ptr, lo, hi, failval, cmpptr) \ +({ \ + typeof(*(ptr)) __nl_val; \ + \ + switch (sizeof(__nl_val)) { \ + case 1: \ + __nl_val = __load_no_speculate_n(ptr, lo, hi, failval, \ + cmpptr, w, b); \ + break; \ + case 2: \ + __nl_val = __load_no_speculate_n(ptr, lo, hi, failval, \ + cmpptr, w, h); \ + break; \ + case 4: \ + __nl_val = __load_no_speculate_n(ptr, lo, hi, failval, \ + cmpptr, w, ); \ + break; \ + case 8: \ + __nl_val = __load_no_speculate_n(ptr, lo, hi, failval, \ + cmpptr, x, ); \ + break; \ + default: \ + BUILD_BUG(); \ + } \ + \ + __nl_val; \ +}) + +#define nospec_ptr(ptr, lo, hi) \ +({ \ + typeof(ptr) __np_ptr = (ptr); \ + __load_no_speculate(&__np_ptr, lo, hi, 0, __np_ptr); \ +}) + #define __smp_mb() dmb(ish) #define __smp_rmb() dmb(ishld) #define __smp_wmb() dmb(ishst) From patchwork Sat Jan 6 01:10:14 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 123596 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp40779qgn; Fri, 5 Jan 2018 17:18:40 -0800 (PST) X-Google-Smtp-Source: ACJfBosVIhq0dYC4b8z/dsVcAq8OwJL4QXPYY5joiSy6krqhLH9ogq9L7701lkCL4gBv8OgPquLC X-Received: by 10.84.246.195 with SMTP id j3mr5012580plt.350.1515201520124; Fri, 05 Jan 2018 17:18:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515201520; cv=none; d=google.com; s=arc-20160816; b=F+1ShBOjEae1Bo91bLQI6Cm/xvh93iOEuKRIHPfPK9ogDQoQi+63WNPddxv2wQ0iNj cTo4k72IQb0qMuBKzGMURQWh3GQ468Q+MtiIJsIf89RH1gArl+wNDH6UyxmCO5o4LiKu I8mbiLBaw59KP4lLI9n9cLKgfDnoLhbqhsoW/tnTN435ZkrlGFkM+o3Hsj4zDd3m8VhR BMWQ7FTok3rcBuAlHrrpOCiD03ydOL9k/l3SCrXRULJmKVS9N0nz/3cq1MbMlqIojRDF 6qmStnzFEL4cFjRFH4c93kKpoigPv/9lBfb/EXQ6ZvRS8OV8O0FtuZGr1cP5gwi8GV83 01Gg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:cc:to:from :subject:arc-authentication-results; bh=g1gi9Cjzcdgn3nDLkHl+niAekyOhnHw8vy3RsjsVl04=; b=rQhKouvbU0dHv1EnhJxyviHILkXZWa1QxBAKxkwf1uM32A/jnJuNKYdheivMffoL0m 1TZx+ySQ0xdMAGmRHu0mnhOVogXxkrs9xNPN59PD1tzCMoX6BMss1rXjldt+VWFLKUVh Ka/f+jdnutYJzbl6LkWwyji74S8x8hW7PihOu9CaLX4FYjrUBimeDYuknCJEz0siLKL4 JYe2pDfN5ZXkn8hFQz6KgAFR2/KV8bIi15CBaUECIu5kNFmt0u1zFhHE8K6Vy8WqAnQJ jEijpHLgSOrvBOJn8W9VCUv3FAX5qwjbOJOVbv2KG8qV0AkMjLTK6+BVkon6y0jOKJlr uLVw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e3si4870271pfg.23.2018.01.05.17.18.39; Fri, 05 Jan 2018 17:18:40 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753702AbeAFBSf (ORCPT + 28 others); Fri, 5 Jan 2018 20:18:35 -0500 Received: from mga04.intel.com ([192.55.52.120]:26995 "EHLO mga04.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753674AbeAFBSc (ORCPT ); Fri, 5 Jan 2018 20:18:32 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Jan 2018 17:18:32 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.46,320,1511856000"; d="scan'208";a="24777435" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.16]) by orsmga002.jf.intel.com with ESMTP; 05 Jan 2018 17:18:29 -0800 Subject: [PATCH 04/18] arm: implement nospec_ptr() From: Dan Williams To: linux-kernel@vger.kernel.org Cc: Mark Rutland , linux-arch@vger.kernel.org, peterz@infradead.org, netdev@vger.kernel.org, gregkh@linuxfoundation.org, tglx@linutronix.de, torvalds@linux-foundation.org, alan@linux.intel.com Date: Fri, 05 Jan 2018 17:10:14 -0800 Message-ID: <151520101463.32271.9320376934909324865.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <151520099201.32271.4677179499894422956.stgit@dwillia2-desk3.amr.corp.intel.com> References: <151520099201.32271.4677179499894422956.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.17.1-9-g687f MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Mark Rutland This patch implements nospec_ptr() for arm, following the recommended architectural sequences for the arm and thumb instruction sets. Signed-off-by: Mark Rutland Signed-off-by: Dan Williams --- arch/arm/include/asm/barrier.h | 75 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+) diff --git a/arch/arm/include/asm/barrier.h b/arch/arm/include/asm/barrier.h index 40f5c410fd8c..6384c90e4b72 100644 --- a/arch/arm/include/asm/barrier.h +++ b/arch/arm/include/asm/barrier.h @@ -37,6 +37,81 @@ #define dmb(x) __asm__ __volatile__ ("" : : : "memory") #endif +#ifdef CONFIG_THUMB2_KERNEL +#define __load_no_speculate_n(ptr, lo, hi, failval, cmpptr, sz) \ +({ \ + typeof(*ptr) __nln_val; \ + typeof(*ptr) __failval = \ + (typeof(*ptr)(unsigned long)(failval)); \ + \ + asm volatile ( \ + " cmp %[c], %[l]\n" \ + " it hs\n" \ + " cmphs %[h], %[c]\n" \ + " blo 1f\n" \ + " ld" #sz " %[v], %[p]\n" \ + "1: it lo\n" \ + " movlo %[v], %[f]\n" \ + " .inst 0xf3af8014 @ CSDB\n" \ + : [v] "=&r" (__nln_val) \ + : [p] "m" (*(ptr)), [l] "r" (lo), [h] "r" (hi), \ + [f] "r" (__failval), [c] "r" (cmpptr) \ + : "cc"); \ + \ + __nln_val; \ +}) +#else +#define __load_no_speculate_n(ptr, lo, hi, failval, cmpptr, sz) \ +({ \ + typeof(*ptr) __nln_val; \ + typeof(*ptr) __failval = \ + (typeof(*ptr)(unsigned long)(failval)); \ + \ + asm volatile ( \ + " cmp %[c], %[l]\n" \ + " cmphs %[h], %[c]\n" \ + " ldr" #sz "hi %[v], %[p]\n" \ + " movls %[v], %[f]\n" \ + " .inst 0xe320f014 @ CSDB\n" \ + : [v] "=&r" (__nln_val) \ + : [p] "m" (*(ptr)), [l] "r" (lo), [h] "r" (hi), \ + [f] "r" (__failval), [c] "r" (cmpptr) \ + : "cc"); \ + \ + __nln_val; \ +}) +#endif + +#define __load_no_speculate(ptr, lo, hi, failval, cmpptr) \ +({ \ + typeof(*(ptr)) __nl_val; \ + \ + switch (sizeof(__nl_val)) { \ + case 1: \ + __nl_val = __load_no_speculate_n(ptr, lo, hi, failval, \ + cmpptr, b); \ + break; \ + case 2: \ + __nl_val = __load_no_speculate_n(ptr, lo, hi, failval, \ + cmpptr, h); \ + break; \ + case 4: \ + __nl_val = __load_no_speculate_n(ptr, lo, hi, failval, \ + cmpptr, ); \ + break; \ + default: \ + BUILD_BUG(); \ + } \ + \ + __nl_val; \ +}) + +#define nospec_ptr(ptr, lo, hi) \ +({ \ + typeof(ptr) __np_ptr = (ptr); \ + __load_no_speculate(&__np_ptr, lo, hi, 0, __np_ptr); \ +}) + #ifdef CONFIG_ARM_HEAVY_MB extern void (*soc_mb)(void); extern void arm_heavy_mb(void);