From patchwork Wed Jan 3 22:38:24 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 123362 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp10722915qgn; Wed, 3 Jan 2018 14:39:27 -0800 (PST) X-Google-Smtp-Source: ACJfBouiuMrlZoIGdLcwv3thJsFbzySKzzpUDO2Cr1l9CiHhpwdv2E5Ka1swDlEvAInNu8Wf7rqm X-Received: by 10.159.242.131 with SMTP id u3mr2693322plr.442.1515019167134; Wed, 03 Jan 2018 14:39:27 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515019167; cv=none; d=google.com; s=arc-20160816; b=fRg075ewGBcDmjVC6UAfP0PP5mTKwjniPXV5eiBpTcDhg/F+N8rMzfNRrE2XJIQSGg t1/cQqMNQJG5BQPLdmMr6cIue7CEx0U8c8GObc9l3JklSd5A2iIJVygLuInHVqz0VY44 7vBstmjiQfiiUbKA+TkSYjE13l4Nwkf+JLGvkmitW4PxR+rVoIcbkQjyKUPDTzO27jZ7 mPcxjRRbqtDaGGbK7dXiilxU6/eGzybpKA2/+j3JBAbK4nZG1jn4gVcRBvQddYugauZI 7hStKZZSg/AAdnV3f1aw0v5jvlP1gvA0Sao2WvXBDQdCvxwX+GcyD+w10JGnUysqgBCk pcjg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=HTylki0zXrryBetIITY6CoddAugxsVyrvUeqlT9MAZY=; b=VtKtbdZqfPruH7nmb0FmkaxytsMH72n/bed1R/QKbYPDqGpduDw6CxYdsYl+0grtK3 5ejPW2JIdDsuRkJ0gyg2WjDPCjCp4W3P3O3yc/mjoNB3/tZwsz7TgxIB0rrdUyWCk8IS Aqu8sxmNGqxVk/R4gMvlmOTGeJGISWfnvcAS97AjE3ZMUByzGO81DkTlGgeYYyMHbxqG oS/TvvaSDC+DxSSg/y4vt4bYE30Mm7XCUccrSqbN9VE9pUb+zeHmbjPako8Bnp/Orgio jtiNYL90GVJIBvhFQaLu4KCFuYHWSN1jbosM+B5NA/GvRWClw2f/Qoj/h/gDrTHbcI9J v/Sg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id m13si1195963pgs.37.2018.01.03.14.39.26; Wed, 03 Jan 2018 14:39:27 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751545AbeACWjX (ORCPT + 28 others); Wed, 3 Jan 2018 17:39:23 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:55498 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751274AbeACWif (ORCPT ); Wed, 3 Jan 2018 17:38:35 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 657AB1529; Wed, 3 Jan 2018 14:38:35 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 79FB93F24A; Wed, 3 Jan 2018 14:38:34 -0800 (PST) From: Mark Rutland To: linux-kernel@vger.kernel.org Cc: linux-arch@vger.kernel.org, Mark Rutland , Will Deacon Subject: [RFC PATCH 1/4] asm-generic/barrier: add generic nospec helpers Date: Wed, 3 Jan 2018 22:38:24 +0000 Message-Id: <20180103223827.39601-2-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180103223827.39601-1-mark.rutland@arm.com> References: <20180103223827.39601-1-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Under speculation, CPUs may mis-predict branches in bounds checks. Thus, memory accesses under a bounds check may be speculated even if the bounds check fails, providing a primitive for building a side channel. This patch adds helpers which can be used to inhibit the use of out-of-bounds pointers and/or valeus read from these under speculation. A generic implementation is provided for compatibility, but does not guarantee safety under speculation. Architectures are expected to override these helpers as necessary. Signed-off-by: Mark Rutland Signed-off-by: Will Deacon --- include/asm-generic/barrier.h | 76 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) -- 2.11.0 diff --git a/include/asm-generic/barrier.h b/include/asm-generic/barrier.h index fe297b599b0a..5eba6ae0c34e 100644 --- a/include/asm-generic/barrier.h +++ b/include/asm-generic/barrier.h @@ -54,6 +54,82 @@ #define read_barrier_depends() do { } while (0) #endif +/** + * nospec_ptr() - Ensure a pointer is bounded, even under speculation. + * + * @ptr: the pointer to test + * @lo: the lower valid bound for @ptr, inclusive + * @hi: the upper valid bound for @ptr, exclusive + * + * If @ptr falls in the interval [@lo, @i), returns @ptr, otherwise returns + * NULL. + * + * Architectures should override this to ensure that ptr falls in the [lo, hi) + * interval both under architectural execution and under speculation, + * preventing propagation of an out-of-bounds pointer to code which is + * speculatively executed. + */ +#ifndef nospec_ptr +#define nospec_ptr(ptr, lo, hi) \ +({ \ + typeof (ptr) __ptr = (ptr); \ + typeof (ptr) __lo = (lo); \ + typeof (ptr) __hi = (hi); \ + \ + (__lo <= __ptr && __ptr < __hi) ? __ptr : NULL; \ +}) +#endif + +/** + * nospec_load() - Load a pointer, respecting bounds under speculation + * + * @ptr: the pointer to load + * @lo: the lower valid bound for @ptr, inclusive + * @hi: the upper valid bound for @ptr, exclusive + * + * If @ptr falls in the interval [@lo, @hi), returns the value at @ptr, + * otherwise returns (typeof(*ptr))0. + * + * Architectures should override this to ensure that ptr falls in the [lo, hi) + * interval both under architectural execution and under speculation, + * preventing speculative out-of-bounds reads. + */ +#ifndef nospec_load +#define nospec_load(ptr, lo, hi) \ +({ \ + typeof (ptr) __ptr = (ptr); \ + typeof (ptr) __lo = (lo); \ + typeof (ptr) __hi = (hi); \ + \ + (__lo <= __ptr && __ptr <= __hi) ? \ + *__ptr : \ + (typeof(*__ptr))(unsigned long)0; \ +}) +#endif + +/** + * nospec_array_load - Load an array entry, respecting bounds under speculation + * + * @arr: the base of the array + * @idx: the index of the element to load + * @sz: the number of elements in the array + * + * If @idx falls in the interval [0, @sz), returns the value at @arr[@idx], + * otherwise returns (typeof(*ptr))0. + * + * This is a wrapper around nospec_load(), provided for convenience. + * Architectures should implement nospec_load() to ensure this is the case + * under speculation. + */ +#define nospec_array_load(arr, idx, sz) \ +({ \ + typeof(*(arr)) *__arr = arr; \ + typeof(idx) __idx = idx; \ + typeof(sz) __sz = __sz; \ + \ + nospec_load(__arr + __idx, __arr, __arr + __sz); \ +}) + #ifndef __smp_mb #define __smp_mb() mb() #endif From patchwork Wed Jan 3 22:38:25 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 123360 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp10722553qgn; Wed, 3 Jan 2018 14:39:00 -0800 (PST) X-Google-Smtp-Source: ACJfBoume3kHUqpCv0eLi5zE9rWaITACXIgMspE4ETkgg4hKDbwJ7qtLhJ770FGt4xcIHS6uQnTl X-Received: by 10.84.198.67 with SMTP id o61mr2750835pld.229.1515019140422; Wed, 03 Jan 2018 14:39:00 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515019140; cv=none; d=google.com; s=arc-20160816; b=JJRCPGfY8k/a5YCjH8eUdLcybYqjggzF0jEmXuHXnDrU3F1aVOjK1TWcHsYKSmM/Dw CCofWnEMXy8gFgQ4WPM7Y4dLCq54cUxb5AsLeTtepDJ8qy06SsoWvLQxOWZsL8MtW9eb JiPVq6OgrGdFLAaApr0joy7T5yda5n136HiIQJPnCqoRXsjhOpBI4esj+QILXmAJGkfm dpoWQyoUeZhd4bjnSy/31+IxsEQe4EU9FzjHgbsMFAWenf3fLJoFRiIb/diYAtrYcdYZ 8eajiwQ1qCF2qdQ2iWWxXD0V998wvIox1Wpkei5rGKHnk+yoJjhN2JuyPh5CUgIH3VEQ acXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=jxS4YClD+1AU65lUv//CQpvaInYJey3daUvbSf8jOpE=; b=WuURpCQ2sD4V5bcOUngL/OhIqDgYRznzCwwEbRFJyd9WMXJP2TaH6QPmhmkFIc2HYc 06tmEyHKDFFjCA0RIW+NzPF1YB8sJHI+WNenoZcsgJz5XqBQA6zxw2i5zFxW0hICyxW1 88jfq+NKs4wDOlo2NfaeOdysgPwYvbwtwrMtJU+JmahyO2YtJMwS2BZ96t+pMvm+j8Ni cdt5cHhpwLlR/0YGNLqbQxY7hK6tdj9YiZ7gg+0OLnyEHNp7fim8wx3/8K8aSE2uGe6i SVykrlXykk3tGoUHbY3lOZsYQaUCsRmM/+GouY6HYiMS+3RSIKSzs6Y3B8rQuY1r2twu hPzw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id h34si1309456pld.202.2018.01.03.14.39.00; Wed, 03 Jan 2018 14:39:00 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751403AbeACWim (ORCPT + 28 others); Wed, 3 Jan 2018 17:38:42 -0500 Received: from foss.arm.com ([217.140.101.70]:55506 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751302AbeACWij (ORCPT ); Wed, 3 Jan 2018 17:38:39 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EFB291529; Wed, 3 Jan 2018 14:38:38 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 107B63F24A; Wed, 3 Jan 2018 14:38:37 -0800 (PST) From: Mark Rutland To: linux-kernel@vger.kernel.org Cc: linux-arch@vger.kernel.org, Mark Rutland , Will Deacon Subject: [RFC PATCH 2/4] Documentation: document nospec helpers Date: Wed, 3 Jan 2018 22:38:25 +0000 Message-Id: <20180103223827.39601-3-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180103223827.39601-1-mark.rutland@arm.com> References: <20180103223827.39601-1-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Document the rationale and usage of the new nospec*() helpers. Signed-off-by: Mark Rutland Signed-off-by: Will Deacon --- Documentation/speculation.txt | 99 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 99 insertions(+) create mode 100644 Documentation/speculation.txt -- 2.11.0 diff --git a/Documentation/speculation.txt b/Documentation/speculation.txt new file mode 100644 index 000000000000..0bec4ed5ac29 --- /dev/null +++ b/Documentation/speculation.txt @@ -0,0 +1,99 @@ +This document explains potential effects of speculation, and how undesirable +effects can be mitigated portably using common APIs. + +=========== +Speculation +=========== + +To improve performance and minimize average latencies, many contemporary CPUs +employ speculative execution techniques such as branch prediction, performing +work which may be discarded at a later stage. + +Typically speculative execution cannot be observed from architectural state, +such as the contents of registers. However, in some cases it is possible to +observe its impact on microarchitectural state, such as the presence or +absence of data in caches. Such state may form side-channels which can be +observed to extract secret information. + +For example, in the presence of branch prediction, it is possible for bounds +checks to be ignored by code which is speculatively executed. Consider the +following code: + + int load_array(int *array, unsigned int idx) { + if (idx >= MAX_ARRAY_ELEMS) + return 0; + else + return array[idx]; + } + +Which, on arm64, may be compiled to an assembly sequence such as: + + CMP , #MAX_ARRAY_ELEMS + B.LT less + MOV , #0 + RET + less: + LDR , [, ] + RET + +It is possible that a CPU mis-predicts the conditional branch, and +speculatively loads array[idx], even if idx >= MAX_ARRAY_ELEMS. This value +will subsequently be discarded, but the speculated load may affect +microarchitectural state which can be subsequently measured. + +More complex sequences involving multiple dependent memory accesses may result +in sensitive information being leaked. Consider the following code, building on +the prior example: + + int load_dependent_arrays(int *arr1, int *arr2, int idx) { + int val1, val2, + + val1 = load_array(arr1, idx); + val2 = load_array(arr2, val1); + + return val2; + } + +Under speculation, the first call to load_array() may return the value of an +out-of-bounds address, while the second call will influence microarchitectural +state dependent on this value. This may provide an arbitrary read primitive. + +==================================== +Mitigating speculation side-channels +==================================== + +The kernel provides a generic API to ensure that bounds checks are respected +even under speculation. Architectures which are affected by speculation-based +side-channels are expected to implement these primitives. + +The following helpers found in can be used to prevent +information from being leaked via side-channels. + +* nospec_load(ptr, lo, hi) + + Returns the data at *ptr only if ptr falls in the [lo, hi) interval. When + ptr < lo or ptr >= hi, typeof(*ptr)0 is returned, even under speculation. + + This does not prevent an out-of-bounds load from being speculated, but does + prevent its value from influencing code which is subsequently speculated, + preventing the value from being leaked. + +* nospec_array_load(arr, idx, sz) + + Returns the data at arr[idx] only if idx falls in the [0, sz) interval. When + idx < 0 or idx > sz, typeof(*arr)0 is returned, even under speculation. + + This is a wrapper around nospec_load() provided for convenience. + +* nospec_ptr(ptr, lo, hi) + + Returns a sanitized pointer that is bounded by the [lo, hi) interval, even + under speculation. If ptr < lo, or ptr >= hi, NULL is returned. + + This is expected to be used by code which computes a pointer to an element + of a data structure, or where multiple fields of a data structure will be + accessed. + + Note that it is not safe to compare the returned value to the original + pointer, as compiler optimizations may infer that the original unsanitized + pointer is safe to use when the two compare equal. From patchwork Wed Jan 3 22:38:26 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 123361 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp10722732qgn; Wed, 3 Jan 2018 14:39:13 -0800 (PST) X-Google-Smtp-Source: ACJfBot2Dc7JYHJ9XoWg7NuHuRjifTFulGkxi3ZwYkuM7gd16ZykssBpxZlZYHopn+KdCoAxtaqY X-Received: by 10.84.143.70 with SMTP id 64mr2711171ply.277.1515019153610; Wed, 03 Jan 2018 14:39:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515019153; cv=none; d=google.com; s=arc-20160816; b=v81rwPITlWk5RCMccXPvKDxnnAOJFScsdZRlWPaVwXgxoZ+R5EdXGJVKdilj2uUE0C ihuyk52uM6dzwNC4ZHG0U/sdGUA8DCiDrhLPsSmBInb+byb1txagcKzw2PFRE8Co8x79 B0KurKimoqvAlNP0bAL+7IaU5f0mxubmZWJ6uPRhdTTMAzgOSEoZWdRLuz31baDPu/j7 sLnAhkes1YqPdBgQuDVjOWS1HjYapJQrjRHLyNG1crNC8NZIvwO+wnl/9wrc62o7w1i2 qNYTXAvnYww7VFbkPbY0WYcZFL6CkCNFtzd131BkfGkZDQ5kLU0TMhf0P2eNYfcaD9s/ gr6A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=bNqK77d6y+pOSHFNdx0aabB6V5rfZW85vv/sPI/2NEw=; b=pqKP//cxZZ8l0UGa7/9acsX9WlgFY6faL1c+0kK6+EM4P7nKoXoC8Sm5IgUmaSNjfB /HS/BpX6d2JUpp6i42togQhgcE5kQ8D86fOWOlF4mkMjsrd7s1rNdVUuggmQ9M6lgiV2 vDvesca/a7wsuQvxQy3L1FUHWjXwlxq5ckuiTgyYdCyJkgT9GmSUQC8eH66Pb9gElPfg kOPyr070LahswsMpDdQHJEW7gWhc/trUvaqVwUStwA1hT6jqvGcawdS8c13OxvyA/n/J QIDFsKlcmB4IeLoN8hbyWxcPEySGUFLT52qnTO+aZ+Da27iA63+KqOCWTAajduTTVwT2 +hHA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id k135si1193854pgc.32.2018.01.03.14.39.13; Wed, 03 Jan 2018 14:39:13 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751505AbeACWjJ (ORCPT + 28 others); Wed, 3 Jan 2018 17:39:09 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:55510 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751078AbeACWil (ORCPT ); Wed, 3 Jan 2018 17:38:41 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C8C1E1596; Wed, 3 Jan 2018 14:38:40 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id DD9813F24A; Wed, 3 Jan 2018 14:38:39 -0800 (PST) From: Mark Rutland To: linux-kernel@vger.kernel.org Cc: linux-arch@vger.kernel.org, Mark Rutland , Will Deacon Subject: [RFC PATCH 3/4] arm64: implement nospec_{load,ptr}() Date: Wed, 3 Jan 2018 22:38:26 +0000 Message-Id: <20180103223827.39601-4-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180103223827.39601-1-mark.rutland@arm.com> References: <20180103223827.39601-1-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org This patch implements nospec_load() and nospec_ptr() for arm64, following the recommended architectural sequence. Signed-off-by: Mark Rutland Signed-off-by: Will Deacon --- arch/arm64/include/asm/barrier.h | 61 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 61 insertions(+) -- 2.11.0 diff --git a/arch/arm64/include/asm/barrier.h b/arch/arm64/include/asm/barrier.h index 77651c49ef44..5d17919e2688 100644 --- a/arch/arm64/include/asm/barrier.h +++ b/arch/arm64/include/asm/barrier.h @@ -40,6 +40,67 @@ #define dma_rmb() dmb(oshld) #define dma_wmb() dmb(oshst) +#define __load_no_speculate_n(ptr, lo, hi, failval, cmpptr, w, sz) \ +({ \ + typeof(*ptr) __nln_val; \ + typeof(*ptr) __failval = \ + (typeof(*ptr))(unsigned long)(failval); \ + \ + asm volatile ( \ + " cmp %[c], %[l]\n" \ + " ccmp %[c], %[h], 2, cs\n" \ + " b.cs 1f\n" \ + " ldr" #sz " %" #w "[v], %[p]\n" \ + "1: csel %" #w "[v], %" #w "[v], %" #w "[f], cc\n" \ + " hint #0x14 // CSDB\n" \ + : [v] "=&r" (__nln_val) \ + : [p] "m" (*(ptr)), [l] "r" (lo), [h] "r" (hi), \ + [f] "rZ" (__failval), [c] "r" (cmpptr) \ + : "cc"); \ + \ + __nln_val; \ +}) + +#define __load_no_speculate(ptr, lo, hi, failval, cmpptr) \ +({ \ + typeof(*(ptr)) __nl_val; \ + \ + switch (sizeof(__nl_val)) { \ + case 1: \ + __nl_val = __load_no_speculate_n(ptr, lo, hi, failval, \ + cmpptr, w, b); \ + break; \ + case 2: \ + __nl_val = __load_no_speculate_n(ptr, lo, hi, failval, \ + cmpptr, w, h); \ + break; \ + case 4: \ + __nl_val = __load_no_speculate_n(ptr, lo, hi, failval, \ + cmpptr, w, ); \ + break; \ + case 8: \ + __nl_val = __load_no_speculate_n(ptr, lo, hi, failval, \ + cmpptr, x, ); \ + break; \ + default: \ + BUILD_BUG(); \ + } \ + \ + __nl_val; \ +}) + +#define nospec_load(ptr, lo, hi) \ +({ \ + typeof(ptr) __nl_ptr = (ptr); \ + __load_no_speculate(__nl_ptr, lo, hi, 0, __nl_ptr); \ +}) + +#define nospec_ptr(ptr, lo, hi) \ +({ \ + typeof(ptr) __np_ptr = (ptr); \ + __load_no_speculate(&__np_ptr, lo, hi, 0, __np_ptr); \ +}) + #define __smp_mb() dmb(ish) #define __smp_rmb() dmb(ishld) #define __smp_wmb() dmb(ishst) From patchwork Wed Jan 3 22:38:27 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 123359 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp10722390qgn; Wed, 3 Jan 2018 14:38:49 -0800 (PST) X-Google-Smtp-Source: ACJfBouJsh/8OPX67vGEiKYPyiTCEXonrats1jp0IYVFJ4aHoxYkAF0Yi1kbn86HtO8SWE/l+OK8 X-Received: by 10.99.117.24 with SMTP id q24mr2366892pgc.407.1515019129238; Wed, 03 Jan 2018 14:38:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515019129; cv=none; d=google.com; s=arc-20160816; b=jeP7mnPHQqNi3L87CkjBdY2MPPmZBS0g73rqogdOmpFHE03KytXw4gnau5mb0lQUpO xRT/YjQyIZp4axS8q479eYxkpMPi1/g5LIM6rb/dYKtqHWnvmWd4amvf9lqE5w8A4cTZ b8UWyP44nPZJyzA3DWKPjrLjFPGFkjpa9EE3GEnBqQYq+wG2reqaN38Ll2AYy4Hf3SvF j+Y8yNzjPiIQIYV2ph03lqrHezveNpFk+NHWZkLfuREuz71V4RwAid901P+iSV8vwFIL 7DDU2B/9wtqhgwSTySbjL5D/9kwH7GmTSBPNi8nOV3wtdP02VAJol7rICdNgsbfFT7B3 ysuQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=2Y9/ghBRtRySqdQWscxufUkql09b7WG1fBnSsENsZns=; b=0bePzbXOc3iuRSIOv9CRbXuTaUlPAuXsWDzJOpSEVIvmDE5NysyUEZ2wvDA+ZX7R/p 0/RaUw1FpS0VpT6Xqn0ByCmesweda17u24AwppRNQ2hBc2J4bow8DYSPLMP0vh5ke4er F4HFImLtGXKSwqt9gciObJzeejXeOSA80yjox8j+lCwHQppzBCar+M4sC6A+9+7Iq5HB x0rQ34B+EXFpDTI3RbZSDHRb/pZelnySw+8wU0f5pXdS8MWlYaB142FC7A7PJpVQHWWc iN4n6vxZSpsUOBxJMGhlU52qK8lYw4LkpyL4bvnTFrH5Y7xMQMo5cJjCNgXQjkp1jSrR ueeA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id h34si1309456pld.202.2018.01.03.14.38.48; Wed, 03 Jan 2018 14:38:49 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751487AbeACWip (ORCPT + 28 others); Wed, 3 Jan 2018 17:38:45 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:55514 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751302AbeACWim (ORCPT ); Wed, 3 Jan 2018 17:38:42 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 767C11529; Wed, 3 Jan 2018 14:38:42 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 8B8A73F24A; Wed, 3 Jan 2018 14:38:41 -0800 (PST) From: Mark Rutland To: linux-kernel@vger.kernel.org Cc: linux-arch@vger.kernel.org, Mark Rutland , Will Deacon Subject: [RFC PATCH 4/4] bpf: inhibit speculated out-of-bounds pointers Date: Wed, 3 Jan 2018 22:38:27 +0000 Message-Id: <20180103223827.39601-5-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180103223827.39601-1-mark.rutland@arm.com> References: <20180103223827.39601-1-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Under speculation, CPUs may mis-predict branches in bounds checks. Thus, memory accesses under a bounds check may be speculated even if the bounds check fails, providing a primitive for building a side channel. The EBPF map code has a number of such bounds-checks accesses in map_lookup_elem implementations. This patch modifies these to use the nospec helpers to inhibit such side channels. The JITted lookup_elem implementations remain potentially vulnerable, and are disabled (with JITted code falling back to the C implementations). Signed-off-by: Mark Rutland Signed-off-by: Will Deacon --- kernel/bpf/arraymap.c | 21 ++++++++++++++------- kernel/bpf/cpumap.c | 8 +++++--- kernel/bpf/devmap.c | 6 +++++- kernel/bpf/sockmap.c | 6 +++++- 4 files changed, 29 insertions(+), 12 deletions(-) -- 2.11.0 diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c index 7c25426d3cf5..5090636da2c1 100644 --- a/kernel/bpf/arraymap.c +++ b/kernel/bpf/arraymap.c @@ -117,15 +117,20 @@ static void *array_map_lookup_elem(struct bpf_map *map, void *key) { struct bpf_array *array = container_of(map, struct bpf_array, map); u32 index = *(u32 *)key; + void *ptr, *high; if (unlikely(index >= array->map.max_entries)) return NULL; - return array->value + array->elem_size * index; + ptr = array->value + array->elem_size * index; + high = array->value + array->elem_size * array->map.max_entries; + + return nospec_ptr(ptr, array->value, high); } /* emit BPF instructions equivalent to C code of array_map_lookup_elem() */ -static u32 array_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf) +static u32 __maybe_unused array_map_gen_lookup(struct bpf_map *map, + struct bpf_insn *insn_buf) { struct bpf_insn *insn = insn_buf; u32 elem_size = round_up(map->value_size, 8); @@ -153,11 +158,15 @@ static void *percpu_array_map_lookup_elem(struct bpf_map *map, void *key) { struct bpf_array *array = container_of(map, struct bpf_array, map); u32 index = *(u32 *)key; + void __percpu **pptrs, __percpu **high; if (unlikely(index >= array->map.max_entries)) return NULL; - return this_cpu_ptr(array->pptrs[index]); + pptrs = array->pptrs + index; + high = array->pptrs + array->map.max_entries; + + return this_cpu_ptr(nospec_load(pptrs, array->pptrs, high)); } int bpf_percpu_array_copy(struct bpf_map *map, void *key, void *value) @@ -302,7 +311,6 @@ const struct bpf_map_ops array_map_ops = { .map_lookup_elem = array_map_lookup_elem, .map_update_elem = array_map_update_elem, .map_delete_elem = array_map_delete_elem, - .map_gen_lookup = array_map_gen_lookup, }; const struct bpf_map_ops percpu_array_map_ops = { @@ -610,8 +618,8 @@ static void *array_of_map_lookup_elem(struct bpf_map *map, void *key) return READ_ONCE(*inner_map); } -static u32 array_of_map_gen_lookup(struct bpf_map *map, - struct bpf_insn *insn_buf) +static u32 __maybe_unused array_of_map_gen_lookup(struct bpf_map *map, + struct bpf_insn *insn_buf) { u32 elem_size = round_up(map->value_size, 8); struct bpf_insn *insn = insn_buf; @@ -644,5 +652,4 @@ const struct bpf_map_ops array_of_maps_map_ops = { .map_fd_get_ptr = bpf_map_fd_get_ptr, .map_fd_put_ptr = bpf_map_fd_put_ptr, .map_fd_sys_lookup_elem = bpf_map_fd_sys_lookup_elem, - .map_gen_lookup = array_of_map_gen_lookup, }; diff --git a/kernel/bpf/cpumap.c b/kernel/bpf/cpumap.c index ce5b669003b2..52831b101d35 100644 --- a/kernel/bpf/cpumap.c +++ b/kernel/bpf/cpumap.c @@ -551,13 +551,15 @@ void cpu_map_free(struct bpf_map *map) struct bpf_cpu_map_entry *__cpu_map_lookup_elem(struct bpf_map *map, u32 key) { struct bpf_cpu_map *cmap = container_of(map, struct bpf_cpu_map, map); - struct bpf_cpu_map_entry *rcpu; + struct bpf_cpu_map_entry **ptr, **high; if (key >= map->max_entries) return NULL; - rcpu = READ_ONCE(cmap->cpu_map[key]); - return rcpu; + ptr = cmap->cpu_map + key; + high = cmap->cpu_map + map->max_entries; + + return READ_ONCE(*nospec_ptr(ptr, cmap->cpu_map, high)); } static void *cpu_map_lookup_elem(struct bpf_map *map, void *key) diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index ebdef54bf7df..23b2b0547304 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -250,11 +250,15 @@ struct net_device *__dev_map_lookup_elem(struct bpf_map *map, u32 key) { struct bpf_dtab *dtab = container_of(map, struct bpf_dtab, map); struct bpf_dtab_netdev *dev; + struct bpf_dtab_netdev **ptr, **high; if (key >= map->max_entries) return NULL; - dev = READ_ONCE(dtab->netdev_map[key]); + ptr = dtab->netdev_map + key; + high = dtab->netdev_map + map->max_entries; + + dev = READ_ONCE(*nospec_ptr(ptr, dtab->netdev_map, high)); return dev ? dev->dev : NULL; } diff --git a/kernel/bpf/sockmap.c b/kernel/bpf/sockmap.c index 5ee2e41893d9..ea59f6737751 100644 --- a/kernel/bpf/sockmap.c +++ b/kernel/bpf/sockmap.c @@ -626,11 +626,15 @@ static int sock_map_get_next_key(struct bpf_map *map, void *key, void *next_key) struct sock *__sock_map_lookup_elem(struct bpf_map *map, u32 key) { struct bpf_stab *stab = container_of(map, struct bpf_stab, map); + struct sock **ptr, **high; if (key >= map->max_entries) return NULL; - return READ_ONCE(stab->sock_map[key]); + ptr = stab->sock_map + key; + high = stab->sock_map + map->max_entries; + + return READ_ONCE(*nospec_ptr(ptr, stab->sock_map, high)); } static int sock_map_delete_elem(struct bpf_map *map, void *key)