From patchwork Fri Jan 5 14:57:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 123527 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp910814qgn; Fri, 5 Jan 2018 06:58:10 -0800 (PST) X-Google-Smtp-Source: ACJfBovcGudZdDdMZkfFa/K/43G57I4WXaZM4tDj+j2+FEg3wsqgTp4m5g7ogAiPc5c96GcMf3A5 X-Received: by 10.98.185.16 with SMTP id z16mr3275573pfe.140.1515164290041; Fri, 05 Jan 2018 06:58:10 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515164290; cv=none; d=google.com; s=arc-20160816; b=hQBDl2bzgYDwcfLO5CmG3UH10pNqYp6L5va3YS0Z6BOtYS6nE2hGB7ffizrLXRSEiM 2GZPcLrVny98qByC776Qnvp2V2+ETWL3JUr/nr0lOKDJZaIsk0Nj7z1ECxUSCLU7StYz SpMaY+fULtV5s4kD/FeoFVi9LeeLhPcgy9qd12s0zdqUG1drzi4vGESIm8PpkH+UBUV0 z6Ox0bU32d0VVrED+MhL7cGpMl+S0ke9HvBBqNtRbAY4rbDgQIl8MtkWUSjiFAfZgH+H DyYv8WwC/dzi4Ukc5ZlVlh4fSL7qBu7AooSSqpnN3zv80WUfMjHU2z/z50Bl0Pev7oLl VJLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=+wgyfcCU94Fe9SH2HhvsI3zrsOGGDGykF3c59TBqu7c=; b=B+IhYI7W0PYAzI8N+690wu0WlctfKfjeAA7Z4mAYBDjW24E3rDkZH0ZbRiDf6K/4nO acLrzWUy4w0t9oTZtmWsate7c5p7mBYxPIUmObAM36dSRmxIidypJEhAwd5v1cFWozsR FQXZLIYjeTfmedlGTsrHVLX9Aw/oCRJth1Zp986+uRU/pSB13MPkvkaKuu6HvaL3sAox t59bszhVfk61HtIV99IgZPbCKV2wQRe8YcAO0DXogqt1Cyd9Il/M+NB2MXFP5vCQXu37 qxvYZw4pQKU/DPbC1zaYUbTMr9ZPb4NhFbadu7G/xVBx+fj1Vt5FRHsvSCo+SFc1XUmd hwtg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 33si4077516ply.308.2018.01.05.06.58.09; Fri, 05 Jan 2018 06:58:10 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752189AbeAEO6G (ORCPT + 26 others); Fri, 5 Jan 2018 09:58:06 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:46054 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751961AbeAEO6D (ORCPT ); Fri, 5 Jan 2018 09:58:03 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8EE201596; Fri, 5 Jan 2018 06:58:03 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 9F2403F581; Fri, 5 Jan 2018 06:58:01 -0800 (PST) From: Mark Rutland To: linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org Cc: dan.j.williams@intel.com, elena.reshetova@intel.com, corbet@lwn.net, alan@linux.intel.com, peterz@infradead.org, will.deacon@arm.com, gregkh@linuxfoundation.org, tglx@linutronix.de, Mark Rutland Subject: [RFCv2 1/4] asm-generic/barrier: add generic nospec helpers Date: Fri, 5 Jan 2018 14:57:47 +0000 Message-Id: <20180105145750.53294-2-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180105145750.53294-1-mark.rutland@arm.com> References: <20180105145750.53294-1-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Under speculation, CPUs may mis-predict branches in bounds checks. Thus, memory accesses under a bounds check may be speculated even if the bounds check fails, providing a primitive for building a side channel. This patch adds helpers which can be used to inhibit the use of out-of-bounds pointers under speculation. A generic implementation is provided for compatibility, but does not guarantee safety under speculation. Architectures are expected to override these helpers as necessary. Signed-off-by: Mark Rutland Signed-off-by: Will Deacon Cc: Daniel Willams Cc: Peter Zijlstra --- include/asm-generic/barrier.h | 68 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+) Dan, I've reworked this so that nospec_ptr() can take an arch-specific barrier sequence. I believe that for x86 you just need to implement __nospec_barrier() as osb(). Mark. -- 2.11.0 diff --git a/include/asm-generic/barrier.h b/include/asm-generic/barrier.h index fe297b599b0a..91c3071f49e5 100644 --- a/include/asm-generic/barrier.h +++ b/include/asm-generic/barrier.h @@ -54,6 +54,74 @@ #define read_barrier_depends() do { } while (0) #endif +/* + * Inhibit subsequent speculative memory accesses. + * + * Architectures with a suitable memory barrier should provide an + * implementation. This is non-portable, and generic code should use + * nospec_ptr(). + */ +#ifndef __nospec_barrier +#define __nospec_barrier() do { } while (0) +#endif + +/** + * nospec_ptr() - Ensure a pointer is bounded, even under speculation. + * + * @ptr: the pointer to test + * @lo: the lower valid bound for @ptr, inclusive + * @hi: the upper valid bound for @ptr, exclusive + * + * If @ptr falls in the interval [@lo, @i), returns @ptr, otherwise returns + * NULL. + * + * Architectures which do not provide __nospec_barrier() should override this + * to ensure that ptr falls in the [lo, hi) interval both under architectural + * execution and under speculation, preventing propagation of an out-of-bounds + * pointer to code which is speculatively executed. + */ +#ifndef nospec_ptr +#define nospec_ptr(ptr, lo, hi) \ +({ \ + typeof (ptr) __ret; \ + typeof (ptr) __ptr = (ptr); \ + typeof (ptr) __lo = (lo); \ + typeof (ptr) __hi = (hi); \ + \ + __ret = (__lo <= __ptr && __ptr < __hi) ? __ptr : NULL; \ + \ + __nospec_barrier(); \ + \ + __ret; \ +}) +#endif + +/** + * nospec_array_ptr - Generate a pointer to an array element, ensuring the + * pointer is bounded under speculation. + * + * @arr: the base of the array + * @idx: the index of the element + * @sz: the number of elements in the array + * + * If @idx falls in the interval [0, @sz), returns the pointer to @arr[@idx], + * otherwise returns NULL. + * + * This is a wrapper around nospec_ptr(), provided for convenience. + * Architectures should implement nospec_ptr() to ensure this is the case + * under speculation. + */ +#define nospec_array_ptr(arr, idx, sz) \ +({ \ + typeof(*(arr)) *__arr = (arr); \ + typeof(idx) __idx = (idx); \ + typeof(sz) __sz = (sz); \ + \ + nospec_ptr(__arr + __idx, __arr, __arr + __sz); \ +}) + +#undef __nospec_barrier + #ifndef __smp_mb #define __smp_mb() mb() #endif From patchwork Fri Jan 5 14:57:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 123528 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp910860qgn; Fri, 5 Jan 2018 06:58:13 -0800 (PST) X-Google-Smtp-Source: ACJfBouIspYejIMqwZD/QQ//rdaVW200ZttBU9GDAxm49IFFxMMdCd5skbT1eMKYmL8ukx5ofzgJ X-Received: by 10.101.82.138 with SMTP id y10mr2766065pgp.165.1515164293719; Fri, 05 Jan 2018 06:58:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515164293; cv=none; d=google.com; s=arc-20160816; b=L0IdtJV6aqmBhYgfXALx9sRlEapNRN+zAqhroBfdxO1v+2aLpr2QWaLt9fVHR9TC6x xKAW9yKlQrl/hcx16OFAFPAEd9g4MBFlF7MhL7px1wCqPCeLiBPuWHJUWimH3xLEmFBw FD/Y+AEyvbTJd60/z37lq74ARnhk0KfdReJKcYMeHlcF6foX1dyCCMGkguuCVwdoybgY Y9FR9Hmr1WvxmF3+sr0Ux/FHi3TnRJYmROh47ZIxl1KhcvxFy4fvB7l3aZjnCCWLMgPI el9c5z01a9gnOSzrUWTrs8qBI/KgtVjM/UKkEjdG873yK+oe933cT38RT6hkgmeKb3q5 fyNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=4fy7ceuDtKNdGCY8zd3wm/eUWR452Osl0ALudt+piUU=; b=vzpFQYAjv/7Qzo5Pb+0Wf5mMX41hoRdtaDneXuCcXJG5hLl4vzSfjBiVC+tTCEX5MN 53SwQTpcT9exuw+jpmDYzMqhBseQXoDaG5Vpn/VQuSV/C9JmMFGIaZrZlhDLoFS8epJF KJH9qdbx+rdbbKpOw4+l4BNf2/HANqJZmPrWwDVhcDY6z3+d76ESL+D3Q2ks5LCm+mE6 sJudAN+oyQYU3pyD2MRYWmWhmuwTj/WapFHvuaMSNIlHDwY/FVkgIGJe1S5uVY8MliQm P13KPI0qDFrcyrMcEHA9ftDdZk7vW+qdill6xwR4gFBADQ9mRUwIThoXUiPv0JobhisE YJ3w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 33si4077516ply.308.2018.01.05.06.58.13; Fri, 05 Jan 2018 06:58:13 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752211AbeAEO6J (ORCPT + 26 others); Fri, 5 Jan 2018 09:58:09 -0500 Received: from foss.arm.com ([217.140.101.70]:46070 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752191AbeAEO6G (ORCPT ); Fri, 5 Jan 2018 09:58:06 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 36C8215BE; Fri, 5 Jan 2018 06:58:06 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 473E83F581; Fri, 5 Jan 2018 06:58:04 -0800 (PST) From: Mark Rutland To: linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org Cc: dan.j.williams@intel.com, elena.reshetova@intel.com, corbet@lwn.net, alan@linux.intel.com, peterz@infradead.org, will.deacon@arm.com, gregkh@linuxfoundation.org, tglx@linutronix.de, Mark Rutland Subject: [RFCv2 2/4] Documentation: document nospec helpers Date: Fri, 5 Jan 2018 14:57:48 +0000 Message-Id: <20180105145750.53294-3-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180105145750.53294-1-mark.rutland@arm.com> References: <20180105145750.53294-1-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Document the rationale and usage of the new nospec*() helpers. Signed-off-by: Mark Rutland Signed-off-by: Will Deacon Cc: Dan Williams Cc: Jonathan Corbet Cc: Peter Zijlstra --- Documentation/speculation.txt | 166 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 166 insertions(+) create mode 100644 Documentation/speculation.txt -- 2.11.0 diff --git a/Documentation/speculation.txt b/Documentation/speculation.txt new file mode 100644 index 000000000000..748fcd4dcda4 --- /dev/null +++ b/Documentation/speculation.txt @@ -0,0 +1,166 @@ +This document explains potential effects of speculation, and how undesirable +effects can be mitigated portably using common APIs. + +=========== +Speculation +=========== + +To improve performance and minimize average latencies, many contemporary CPUs +employ speculative execution techniques such as branch prediction, performing +work which may be discarded at a later stage. + +Typically speculative execution cannot be observed from architectural state, +such as the contents of registers. However, in some cases it is possible to +observe its impact on microarchitectural state, such as the presence or +absence of data in caches. Such state may form side-channels which can be +observed to extract secret information. + +For example, in the presence of branch prediction, it is possible for bounds +checks to be ignored by code which is speculatively executed. Consider the +following code: + + int load_array(int *array, unsigned int idx) { + if (idx >= MAX_ARRAY_ELEMS) + return 0; + else + return array[idx]; + } + +Which, on arm64, may be compiled to an assembly sequence such as: + + CMP , #MAX_ARRAY_ELEMS + B.LT less + MOV , #0 + RET + less: + LDR , [, ] + RET + +It is possible that a CPU mis-predicts the conditional branch, and +speculatively loads array[idx], even if idx >= MAX_ARRAY_ELEMS. This value +will subsequently be discarded, but the speculated load may affect +microarchitectural state which can be subsequently measured. + +More complex sequences involving multiple dependent memory accesses may result +in sensitive information being leaked. Consider the following code, building on +the prior example: + + int load_dependent_arrays(int *arr1, int *arr2, int idx) { + int val1, val2, + + val1 = load_array(arr1, idx); + val2 = load_array(arr2, val1); + + return val2; + } + +Under speculation, the first call to load_array() may return the value of an +out-of-bounds address, while the second call will influence microarchitectural +state dependent on this value. This may provide an arbitrary read primitive. + +==================================== +Mitigating speculation side-channels +==================================== + +The kernel provides a generic API to ensure that bounds checks are respected +even under speculation. Architectures which are affected by speculation-based +side-channels are expected to implement these primitives. + +The following helpers found in can be used to prevent +information from being leaked via side-channels. + +* nospec_ptr(ptr, lo, hi) + + Returns a sanitized pointer that is bounded by the [lo, hi) interval. When + ptr < lo, or ptr >= hi, NULL is returned. Prevents an out-of-bounds pointer + being propagated to code which is speculatively executed. + + This is expected to be used by code which computes pointers to data + structures, where part of the address (such as an array index) may be + user-controlled. + + This can be used to protect the earlier load_array() example: + + int load_array(int *array, unsigned int idx) + { + int *elem; + + if ((elem = nospec_ptr(array + idx, array, array + MAX_ARRAY_ELEMS))) + return *elem; + else + return 0; + } + + This can also be used in situations where multiple fields on a structure are + accessed: + + struct foo array[SIZE]; + int a, b; + + void do_thing(int idx) + { + struct foo *elem; + + if ((elem = nospec_ptr(array + idx, array, array + SIZE)) { + a = elem->field_a; + b = elem->field_b; + } + } + + It is imperative that the returned pointer is used. Pointers which are + generated separately are subject to a number of potential CPU and compiler + optimizations, and may still be used speculatively. For example, this means + that the following sequence is unsafe: + + struct foo array[SIZE]; + int a, b; + + void do_thing(int idx) + { + if (nospec_ptr(array + idx, array, array + SIZE) != NULL) { + // unsafe as wrong pointer is used + a = array[idx].field_a; + b = array[idx].field_b; + } + } + + Similarly, it is unsafe to compare the returned pointer with other pointers, + as this may permit the compiler to substitute one pointer with another, + permitting speculation. For example, the following sequence is unsafe: + + struct foo array[SIZE]; + int a, b; + + void do_thing(int idx) + { + struct foo *elem = nospec_ptr(array + idx, array, array + size); + + // unsafe due to pointer substitution + if (elem == &array[idx]) { + a = elem->field_a; + b = elem->field_b; + } + } + +* nospec_array_ptr(arr, idx, sz) + + Returns a sanitized pointer to arr[idx] only if idx falls in the [0, sz) + interval. When idx < 0 or idx > sz, NULL is returned. Prevents an + out-of-bounds pointer being propagated to code which is speculatively + executed. + + This is a convenience function which wraps nospec_ptr(), and has the same + caveats w.r.t. the use of the returned pointer. + + For example, this may be used as follows: + + int load_array(int *array, unsigned int idx) + { + int *elem; + + if ((elem = nospec_array_ptr(array, idx, MAX_ARRAY_ELEMS))) + return *elem; + else + return 0; + } + From patchwork Fri Jan 5 14:57:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 123529 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp910905qgn; Fri, 5 Jan 2018 06:58:16 -0800 (PST) X-Google-Smtp-Source: ACJfBotUiFEde5oqbILDa0UT6C2fwlPFnAe92Keak0IMI3yrUAuhS9epHH/gNyX+GkoaEJszlv23 X-Received: by 10.101.92.66 with SMTP id v2mr2821012pgr.37.1515164296542; Fri, 05 Jan 2018 06:58:16 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515164296; cv=none; d=google.com; s=arc-20160816; b=VmRg3nwGaXxVRZ8grrzQ5eGQOEB9OVl3Un54C6oGP087LHk/ittLkjrHuDKIy2Urze KuvHp1ZbRsH0nllK+o2w+IRe9LQg6OUGHksXv1nxshATggYyyIKR+GotbMkXzYJRAdDF /4M06smO2dYdNIYXyujV63I41yXhYCQsA2B6Zg7UWqweTqcg91VNPYRZDrBirKA6VzaK viVtB+bk5TYikr/uivzaIZVOXA/ofjxg3/qJsYwJ6T4Yg6rx221WTy/aplVJr2SNLfIp X5sey+vWIlN0reYj4nli4+LQjO//9+pl9HhiPcWnGvjnBqijidLV6faZ+HtYfLEAUwVl y/oA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=MnZKqmMPmBImvWxbqYQitaeW6PpeTXGCjZlv633zhyQ=; b=d15A01IWu0oJLukk0to4yNFg5Ji/rqoW2poafY2cBU5tSxHlYH2m8VLvPlR60/zX3X MLHQeUuku4Hlt5q5sPX+fAkWU9478m4NtULC1ybdvYLBc6XhSuHWcgU6vkFKZow9dS4X P1dqCzZmecNqh2Qc9zhch6WHNMbpIYjFDIZOG6/Or2hwTHV6k2AKUVgT10qLo0YbGZEy g8hzX5+W6hxw4OpYDhn27hmeMLAX1QcSR0EcaE1MhOSV3mS6ZwsjjXDqPWPd+71d1Ve1 mSScPm9oZ6jshDFj9AEi/ZmWdz8HwcN260/Pn+k8a80KKqOsnyWwsKrjyuw88GofhLAW Scdw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 33si4077516ply.308.2018.01.05.06.58.16; Fri, 05 Jan 2018 06:58:16 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752240AbeAEO6O (ORCPT + 26 others); Fri, 5 Jan 2018 09:58:14 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:46086 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752195AbeAEO6J (ORCPT ); Fri, 5 Jan 2018 09:58:09 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D2EFE15BF; Fri, 5 Jan 2018 06:58:08 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id E375E3F581; Fri, 5 Jan 2018 06:58:06 -0800 (PST) From: Mark Rutland To: linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org Cc: dan.j.williams@intel.com, elena.reshetova@intel.com, corbet@lwn.net, alan@linux.intel.com, peterz@infradead.org, will.deacon@arm.com, gregkh@linuxfoundation.org, tglx@linutronix.de, Mark Rutland Subject: [RFCv2 3/4] arm64: implement nospec_ptr() Date: Fri, 5 Jan 2018 14:57:49 +0000 Message-Id: <20180105145750.53294-4-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180105145750.53294-1-mark.rutland@arm.com> References: <20180105145750.53294-1-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org This patch implements nospec_ptr() for arm64, following the recommended architectural sequence. Signed-off-by: Mark Rutland Signed-off-by: Will Deacon Cc: Dan Williams Cc: Peter Zijlstra --- arch/arm64/include/asm/barrier.h | 55 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) -- 2.11.0 diff --git a/arch/arm64/include/asm/barrier.h b/arch/arm64/include/asm/barrier.h index 77651c49ef44..b4819f6a0e5c 100644 --- a/arch/arm64/include/asm/barrier.h +++ b/arch/arm64/include/asm/barrier.h @@ -40,6 +40,61 @@ #define dma_rmb() dmb(oshld) #define dma_wmb() dmb(oshst) +#define __load_no_speculate_n(ptr, lo, hi, failval, cmpptr, w, sz) \ +({ \ + typeof(*ptr) __nln_val; \ + typeof(*ptr) __failval = \ + (typeof(*ptr))(unsigned long)(failval); \ + \ + asm volatile ( \ + " cmp %[c], %[l]\n" \ + " ccmp %[c], %[h], 2, cs\n" \ + " b.cs 1f\n" \ + " ldr" #sz " %" #w "[v], %[p]\n" \ + "1: csel %" #w "[v], %" #w "[v], %" #w "[f], cc\n" \ + " hint #0x14 // CSDB\n" \ + : [v] "=&r" (__nln_val) \ + : [p] "m" (*(ptr)), [l] "r" (lo), [h] "r" (hi), \ + [f] "rZ" (__failval), [c] "r" (cmpptr) \ + : "cc"); \ + \ + __nln_val; \ +}) + +#define __load_no_speculate(ptr, lo, hi, failval, cmpptr) \ +({ \ + typeof(*(ptr)) __nl_val; \ + \ + switch (sizeof(__nl_val)) { \ + case 1: \ + __nl_val = __load_no_speculate_n(ptr, lo, hi, failval, \ + cmpptr, w, b); \ + break; \ + case 2: \ + __nl_val = __load_no_speculate_n(ptr, lo, hi, failval, \ + cmpptr, w, h); \ + break; \ + case 4: \ + __nl_val = __load_no_speculate_n(ptr, lo, hi, failval, \ + cmpptr, w, ); \ + break; \ + case 8: \ + __nl_val = __load_no_speculate_n(ptr, lo, hi, failval, \ + cmpptr, x, ); \ + break; \ + default: \ + BUILD_BUG(); \ + } \ + \ + __nl_val; \ +}) + +#define nospec_ptr(ptr, lo, hi) \ +({ \ + typeof(ptr) __np_ptr = (ptr); \ + __load_no_speculate(&__np_ptr, lo, hi, 0, __np_ptr); \ +}) + #define __smp_mb() dmb(ish) #define __smp_rmb() dmb(ishld) #define __smp_wmb() dmb(ishst) From patchwork Fri Jan 5 14:57:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 123530 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp911195qgn; Fri, 5 Jan 2018 06:58:32 -0800 (PST) X-Google-Smtp-Source: ACJfBosW8PAgU+GY6tdhFkOWPDMzcsm5PO6pvewSWB3caamVUiTTUmj6Kjux/1QHVsy8caxNEBLR X-Received: by 10.159.241.140 with SMTP id s12mr3355844plr.297.1515164312511; Fri, 05 Jan 2018 06:58:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515164312; cv=none; d=google.com; s=arc-20160816; b=gKtrodMSBf/jNiiYrjneooqjZ9aLa7DmjQbFfEh55WhjeVxGH27Zejl1z6y4tVVEGR KdkcROLvz3Px1KYQD7//sOt4vkgafdBRlUV+nFlG/Y7cGzBMFFCeJNiyBtzGeKykFM5u z2U5O7hOr2TfwYYUDfZp9ctQZCRoD8kDeCOKab3ypwrTpSmaqQE1yB4R1sdEcnsppqmJ KZK3UDsyME8adM4vNmLkveDwOdrLZPaxpA5grSUmiditMr7rTKxisGB5uGb738toRnN0 Ljsp9NYfEzk8o+zGBjM0mhAxqD0IoeYoAc2X7MirjxRP4xyW0G6eNKI+xqRI5RWXg4hg nIiQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=acDhIT+STMBo9is2Ni5mTehLq9lIFU4oTA/Bek7nWaU=; b=T0bA6wbacdMZdy67nTX8DJTzv1ce7ZLWY+G6EUr6UGUYYozGTkuTbqIglXLXv+mlu8 pOE0OVSVC6WOKwYMZT5QnBjM4UzGbfbSRZaTPJC7oac+LOhvAWp5Lccn3/aZJjL6pUZC yhCuPv7dw35L3UcQTyftv4ny3PgB/KRRaOrm+ooMtl/AUZX57rWeuiIUJN7zJcEIaICE GxloD0nUJCcPa0Dtw4cc8tH/ElFpjC5E+ikMu73Orre+Cjq7UhqGJVS6VnZ1bkBDOJt+ gqEB9Sc/v3d1OKd6LcSAIQ6D3qWMKcf/cqG2DhjckxJdYbtDKDjbuNisGNKKG/wHdG/I VjHQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id n17si3624402pgc.536.2018.01.05.06.58.32; Fri, 05 Jan 2018 06:58:32 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752277AbeAEO6a (ORCPT + 26 others); Fri, 5 Jan 2018 09:58:30 -0500 Received: from foss.arm.com ([217.140.101.70]:46092 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752214AbeAEO6M (ORCPT ); Fri, 5 Jan 2018 09:58:12 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 82CEA1610; Fri, 5 Jan 2018 06:58:11 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 9366A3F581; Fri, 5 Jan 2018 06:58:09 -0800 (PST) From: Mark Rutland To: linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org Cc: dan.j.williams@intel.com, elena.reshetova@intel.com, corbet@lwn.net, alan@linux.intel.com, peterz@infradead.org, will.deacon@arm.com, gregkh@linuxfoundation.org, tglx@linutronix.de, Mark Rutland Subject: [RFCv2 4/4] bpf: inhibit speculated out-of-bounds pointers Date: Fri, 5 Jan 2018 14:57:50 +0000 Message-Id: <20180105145750.53294-5-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180105145750.53294-1-mark.rutland@arm.com> References: <20180105145750.53294-1-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Note: this patch is an *example* use of the nospec API. It is understood that this is incomplete, etc. Under speculation, CPUs may mis-predict branches in bounds checks. Thus, memory accesses under a bounds check may be speculated even if the bounds check fails, providing a primitive for building a side channel. The EBPF map code has a number of such bounds-checks accesses in map_lookup_elem implementations. This patch modifies these to use the nospec helpers to inhibit such side channels. The JITted lookup_elem implementations remain potentially vulnerable, and are disabled (with JITted code falling back to the C implementations). Signed-off-by: Mark Rutland Signed-off-by: Will Deacon Cc: Dan Williams Cc: Peter Zijlstra --- kernel/bpf/arraymap.c | 20 +++++++++++++------- kernel/bpf/cpumap.c | 5 ++--- kernel/bpf/devmap.c | 3 ++- kernel/bpf/sockmap.c | 3 ++- 4 files changed, 19 insertions(+), 12 deletions(-) -- 2.11.0 diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c index 7c25426d3cf5..deaad334a100 100644 --- a/kernel/bpf/arraymap.c +++ b/kernel/bpf/arraymap.c @@ -117,15 +117,20 @@ static void *array_map_lookup_elem(struct bpf_map *map, void *key) { struct bpf_array *array = container_of(map, struct bpf_array, map); u32 index = *(u32 *)key; + void *ptr, *high; if (unlikely(index >= array->map.max_entries)) return NULL; - return array->value + array->elem_size * index; + ptr = array->value + array->elem_size * index; + high = array->value + array->elem_size * array->map.max_entries; + + return nospec_ptr(ptr, array->value, high); } /* emit BPF instructions equivalent to C code of array_map_lookup_elem() */ -static u32 array_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf) +static u32 __maybe_unused array_map_gen_lookup(struct bpf_map *map, + struct bpf_insn *insn_buf) { struct bpf_insn *insn = insn_buf; u32 elem_size = round_up(map->value_size, 8); @@ -153,11 +158,14 @@ static void *percpu_array_map_lookup_elem(struct bpf_map *map, void *key) { struct bpf_array *array = container_of(map, struct bpf_array, map); u32 index = *(u32 *)key; + void __percpu *pptr; if (unlikely(index >= array->map.max_entries)) return NULL; - return this_cpu_ptr(array->pptrs[index]); + pptr = nospec_array_ptr(array->pptrs, index, array->map.max_entries); + + return this_cpu_ptr(pptr); } int bpf_percpu_array_copy(struct bpf_map *map, void *key, void *value) @@ -302,7 +310,6 @@ const struct bpf_map_ops array_map_ops = { .map_lookup_elem = array_map_lookup_elem, .map_update_elem = array_map_update_elem, .map_delete_elem = array_map_delete_elem, - .map_gen_lookup = array_map_gen_lookup, }; const struct bpf_map_ops percpu_array_map_ops = { @@ -610,8 +617,8 @@ static void *array_of_map_lookup_elem(struct bpf_map *map, void *key) return READ_ONCE(*inner_map); } -static u32 array_of_map_gen_lookup(struct bpf_map *map, - struct bpf_insn *insn_buf) +static u32 __maybe_unused array_of_map_gen_lookup(struct bpf_map *map, + struct bpf_insn *insn_buf) { u32 elem_size = round_up(map->value_size, 8); struct bpf_insn *insn = insn_buf; @@ -644,5 +651,4 @@ const struct bpf_map_ops array_of_maps_map_ops = { .map_fd_get_ptr = bpf_map_fd_get_ptr, .map_fd_put_ptr = bpf_map_fd_put_ptr, .map_fd_sys_lookup_elem = bpf_map_fd_sys_lookup_elem, - .map_gen_lookup = array_of_map_gen_lookup, }; diff --git a/kernel/bpf/cpumap.c b/kernel/bpf/cpumap.c index ce5b669003b2..6769a0e30c8c 100644 --- a/kernel/bpf/cpumap.c +++ b/kernel/bpf/cpumap.c @@ -551,13 +551,12 @@ void cpu_map_free(struct bpf_map *map) struct bpf_cpu_map_entry *__cpu_map_lookup_elem(struct bpf_map *map, u32 key) { struct bpf_cpu_map *cmap = container_of(map, struct bpf_cpu_map, map); - struct bpf_cpu_map_entry *rcpu; if (key >= map->max_entries) return NULL; - rcpu = READ_ONCE(cmap->cpu_map[key]); - return rcpu; + return READ_ONCE(*nospec_array_ptr(cmap->cpu_map, key, + map->max_entries)); } static void *cpu_map_lookup_elem(struct bpf_map *map, void *key) diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index ebdef54bf7df..5a1050d270a0 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -254,7 +254,8 @@ struct net_device *__dev_map_lookup_elem(struct bpf_map *map, u32 key) if (key >= map->max_entries) return NULL; - dev = READ_ONCE(dtab->netdev_map[key]); + dev = READ_ONCE(*nospec_array_ptr(dtab->netdev_map, key, + map->max_entries)); return dev ? dev->dev : NULL; } diff --git a/kernel/bpf/sockmap.c b/kernel/bpf/sockmap.c index 5ee2e41893d9..e912de3cd4ce 100644 --- a/kernel/bpf/sockmap.c +++ b/kernel/bpf/sockmap.c @@ -630,7 +630,8 @@ struct sock *__sock_map_lookup_elem(struct bpf_map *map, u32 key) if (key >= map->max_entries) return NULL; - return READ_ONCE(stab->sock_map[key]); + return READ_ONCE(*nospec_array_ptr(stab->sock_map, key, + map->max_entries)); } static int sock_map_delete_elem(struct bpf_map *map, void *key)