From patchwork Wed Feb 1 15:05:52 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alex_Benn=C3=A9e?= X-Patchwork-Id: 93059 Delivered-To: patch@linaro.org Received: by 10.140.20.99 with SMTP id 90csp2464802qgi; Wed, 1 Feb 2017 07:18:58 -0800 (PST) X-Received: by 10.233.239.143 with SMTP id d137mr3507360qkg.284.1485962338634; Wed, 01 Feb 2017 07:18:58 -0800 (PST) Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id g67si14619568qkf.69.2017.02.01.07.18.58 for (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 01 Feb 2017 07:18:58 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:51495 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cYwgU-0002A4-3n for patch@linaro.org; Wed, 01 Feb 2017 10:18:58 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37378) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cYwUI-0007N3-Br for qemu-devel@nongnu.org; Wed, 01 Feb 2017 10:06:25 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cYwUB-0008Kx-29 for qemu-devel@nongnu.org; Wed, 01 Feb 2017 10:06:22 -0500 Received: from mail-wj0-x230.google.com ([2a00:1450:400c:c01::230]:36047) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1cYwUA-0008Ko-OH for qemu-devel@nongnu.org; Wed, 01 Feb 2017 10:06:14 -0500 Received: by mail-wj0-x230.google.com with SMTP id n2so21241304wjq.3 for ; Wed, 01 Feb 2017 07:06:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ANlYvXF8wBmVdbxibGY0IPvKUX2GJJAGF3NWHyvc0bk=; b=hjrQR3APNDmzmwER0vjND4BYz24Kr+QEScWE7In8MOtyLrhGaNJPfz2f5hy+35fqtB IP8wJFx9pSmJIFvND+tXSpolNkKNMPbG/EG2HIkyIjpP4ooeSMZLeOEYS6NE9/nqS7IC vvCjNv/db4z9HF0CROtCnqRsLhH1c+YHtroGE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ANlYvXF8wBmVdbxibGY0IPvKUX2GJJAGF3NWHyvc0bk=; b=O/UWA6D3VhsUIA/mqch9X/vN3ni3T1MjXpL8CKXLGbJA0HYoicShkoNV+AU/jwqF7S 2ZkDBoWEWlxO0aTOJz/Oh65YfEgq3CDhI107B506TvmxPP/MgStTDQpklMS9esjpmFKN ZamufNneFh5ifWDKEFdH4tt0Jkqrc3c1DoThqVAihUu00TA5aJjAvBmZqoQi4lzWbwSu OlC8IMFZ6l7QHbP7n7dhF1LXOKyA8z8nm6+nqaDK21WFrnx4p6UVRIEoGcoV1i9hWs02 Ccvn9A76uAZGO5ihNy2N7KWLzSq5FZtyewlD2IDIepzbPwfwAyj7iU6plJGasu7g1YTW 0ktA== X-Gm-Message-State: AIkVDXKZSKrLqFGsJycOSybnKKn7sbAUmzJoMo7F8fyml374yTejMvDE3h+z8VjB3zUOpksD X-Received: by 10.223.154.109 with SMTP id z100mr3007119wrb.145.1485961573430; Wed, 01 Feb 2017 07:06:13 -0800 (PST) Received: from zen.linaro.local ([81.128.185.34]) by smtp.gmail.com with ESMTPSA id c133sm29961157wmd.13.2017.02.01.07.06.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 01 Feb 2017 07:06:07 -0800 (PST) Received: from zen.linaroharston (localhost [127.0.0.1]) by zen.linaro.local (Postfix) with ESMTP id 3DB773E0F8E; Wed, 1 Feb 2017 15:05:55 +0000 (GMT) From: =?utf-8?q?Alex_Benn=C3=A9e?= To: mttcg@listserver.greensocs.com, qemu-devel@nongnu.org, fred.konrad@greensocs.com, a.rigo@virtualopensystems.com, cota@braap.org, bobby.prani@gmail.com, nikunj@linux.vnet.ibm.com Date: Wed, 1 Feb 2017 15:05:52 +0000 Message-Id: <20170201150553.9381-25-alex.bennee@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20170201150553.9381-1-alex.bennee@linaro.org> References: <20170201150553.9381-1-alex.bennee@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2a00:1450:400c:c01::230 Subject: [Qemu-devel] [PATCH v9 24/25] target-arm: ensure all cross vCPUs TLB flushes complete X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, jan.kiszka@siemens.com, mark.burton@greensocs.com, "open list:ARM" , serge.fdrv@gmail.com, pbonzini@redhat.com, =?utf-8?q?Alex_Benn=C3=A9e?= , bamvor.zhangjian@linaro.org, rth@twiddle.net Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Previously flushes on other vCPUs would only get serviced when they exited their TranslationBlocks. While this isn't overly problematic it violates the semantics of TLB flush from the point of view of source vCPU. To solve this we call the cputlb *_all_cpus_synced() functions to do the flushes and ask it to ensure all flushes are completed before we start the next instruction. As this involves exiting the cpu_loop we need to ensure the PC is saved before the tlb helper functions are called. Signed-off-by: Alex Bennée Reviewed-by: Richard Henderson --- v8 - fixup merge for bitmap based API - fixup new _synced API v9 - fixup merge conflicts --- target/arm/helper.c | 221 +++++++++++++++++++++++----------------------------- 1 file changed, 98 insertions(+), 123 deletions(-) -- 2.11.0 diff --git a/target/arm/helper.c b/target/arm/helper.c index 599eca872a..4ff9d26c8a 100644 --- a/target/arm/helper.c +++ b/target/arm/helper.c @@ -536,41 +536,33 @@ static void tlbimvaa_write(CPUARMState *env, const ARMCPRegInfo *ri, static void tlbiall_is_write(CPUARMState *env, const ARMCPRegInfo *ri, uint64_t value) { - CPUState *other_cs; + CPUState *cs = ENV_GET_CPU(env); - CPU_FOREACH(other_cs) { - tlb_flush(other_cs); - } + tlb_flush_all_cpus_synced(cs); } static void tlbiasid_is_write(CPUARMState *env, const ARMCPRegInfo *ri, uint64_t value) { - CPUState *other_cs; + CPUState *cs = ENV_GET_CPU(env); - CPU_FOREACH(other_cs) { - tlb_flush(other_cs); - } + tlb_flush_all_cpus_synced(cs); } static void tlbimva_is_write(CPUARMState *env, const ARMCPRegInfo *ri, uint64_t value) { - CPUState *other_cs; + CPUState *cs = ENV_GET_CPU(env); - CPU_FOREACH(other_cs) { - tlb_flush_page(other_cs, value & TARGET_PAGE_MASK); - } + tlb_flush_page_all_cpus_synced(cs, value & TARGET_PAGE_MASK); } static void tlbimvaa_is_write(CPUARMState *env, const ARMCPRegInfo *ri, uint64_t value) { - CPUState *other_cs; + CPUState *cs = ENV_GET_CPU(env); - CPU_FOREACH(other_cs) { - tlb_flush_page(other_cs, value & TARGET_PAGE_MASK); - } + tlb_flush_page_all_cpus_synced(cs, value & TARGET_PAGE_MASK); } static void tlbiall_nsnh_write(CPUARMState *env, const ARMCPRegInfo *ri, @@ -587,14 +579,12 @@ static void tlbiall_nsnh_write(CPUARMState *env, const ARMCPRegInfo *ri, static void tlbiall_nsnh_is_write(CPUARMState *env, const ARMCPRegInfo *ri, uint64_t value) { - CPUState *other_cs; + CPUState *cs = ENV_GET_CPU(env); - CPU_FOREACH(other_cs) { - tlb_flush_by_mmuidx(other_cs, - (1 << ARMMMUIdx_S12NSE1) | - (1 << ARMMMUIdx_S12NSE0) | - (1 << ARMMMUIdx_S2NS)); - } + tlb_flush_by_mmuidx_all_cpus_synced(cs, + (1 << ARMMMUIdx_S12NSE1) | + (1 << ARMMMUIdx_S12NSE0) | + (1 << ARMMMUIdx_S2NS)); } static void tlbiipas2_write(CPUARMState *env, const ARMCPRegInfo *ri, @@ -621,7 +611,7 @@ static void tlbiipas2_write(CPUARMState *env, const ARMCPRegInfo *ri, static void tlbiipas2_is_write(CPUARMState *env, const ARMCPRegInfo *ri, uint64_t value) { - CPUState *other_cs; + CPUState *cs = ENV_GET_CPU(env); uint64_t pageaddr; if (!arm_feature(env, ARM_FEATURE_EL2) || !(env->cp15.scr_el3 & SCR_NS)) { @@ -630,9 +620,8 @@ static void tlbiipas2_is_write(CPUARMState *env, const ARMCPRegInfo *ri, pageaddr = sextract64(value << 12, 0, 40); - CPU_FOREACH(other_cs) { - tlb_flush_page_by_mmuidx(other_cs, pageaddr, (1 << ARMMMUIdx_S2NS)); - } + tlb_flush_page_by_mmuidx_all_cpus_synced(cs, pageaddr, + (1 << ARMMMUIdx_S2NS)); } static void tlbiall_hyp_write(CPUARMState *env, const ARMCPRegInfo *ri, @@ -646,11 +635,9 @@ static void tlbiall_hyp_write(CPUARMState *env, const ARMCPRegInfo *ri, static void tlbiall_hyp_is_write(CPUARMState *env, const ARMCPRegInfo *ri, uint64_t value) { - CPUState *other_cs; + CPUState *cs = ENV_GET_CPU(env); - CPU_FOREACH(other_cs) { - tlb_flush_by_mmuidx(other_cs, (1 << ARMMMUIdx_S1E2)); - } + tlb_flush_by_mmuidx_all_cpus_synced(cs, (1 << ARMMMUIdx_S1E2)); } static void tlbimva_hyp_write(CPUARMState *env, const ARMCPRegInfo *ri, @@ -665,12 +652,11 @@ static void tlbimva_hyp_write(CPUARMState *env, const ARMCPRegInfo *ri, static void tlbimva_hyp_is_write(CPUARMState *env, const ARMCPRegInfo *ri, uint64_t value) { - CPUState *other_cs; + CPUState *cs = ENV_GET_CPU(env); uint64_t pageaddr = value & ~MAKE_64BIT_MASK(0, 12); - CPU_FOREACH(other_cs) { - tlb_flush_page_by_mmuidx(other_cs, pageaddr, (1 << ARMMMUIdx_S1E2)); - } + tlb_flush_page_by_mmuidx_all_cpus_synced(cs, pageaddr, + (1 << ARMMMUIdx_S1E2)); } static const ARMCPRegInfo cp_reginfo[] = { @@ -1339,14 +1325,16 @@ static const ARMCPRegInfo v7_cp_reginfo[] = { static const ARMCPRegInfo v7mp_cp_reginfo[] = { /* 32 bit TLB invalidates, Inner Shareable */ { .name = "TLBIALLIS", .cp = 15, .opc1 = 0, .crn = 8, .crm = 3, .opc2 = 0, - .type = ARM_CP_NO_RAW, .access = PL1_W, .writefn = tlbiall_is_write }, + .type = ARM_CP_NO_RAW | ARM_CP_EXIT_PC, .access = PL1_W, + .writefn = tlbiall_is_write }, { .name = "TLBIMVAIS", .cp = 15, .opc1 = 0, .crn = 8, .crm = 3, .opc2 = 1, - .type = ARM_CP_NO_RAW, .access = PL1_W, .writefn = tlbimva_is_write }, + .type = ARM_CP_NO_RAW | ARM_CP_EXIT_PC, .access = PL1_W, + .writefn = tlbimva_is_write }, { .name = "TLBIASIDIS", .cp = 15, .opc1 = 0, .crn = 8, .crm = 3, .opc2 = 2, - .type = ARM_CP_NO_RAW, .access = PL1_W, + .type = ARM_CP_NO_RAW | ARM_CP_EXIT_PC, .access = PL1_W, .writefn = tlbiasid_is_write }, { .name = "TLBIMVAAIS", .cp = 15, .opc1 = 0, .crn = 8, .crm = 3, .opc2 = 3, - .type = ARM_CP_NO_RAW, .access = PL1_W, + .type = ARM_CP_NO_RAW | ARM_CP_EXIT_PC, .access = PL1_W, .writefn = tlbimvaa_is_write }, REGINFO_SENTINEL }; @@ -2861,8 +2849,7 @@ static CPAccessResult aa64_cacheop_access(CPUARMState *env, static void tlbi_aa64_vmalle1_write(CPUARMState *env, const ARMCPRegInfo *ri, uint64_t value) { - ARMCPU *cpu = arm_env_get_cpu(env); - CPUState *cs = CPU(cpu); + CPUState *cs = ENV_GET_CPU(env); if (arm_is_secure_below_el3(env)) { tlb_flush_by_mmuidx(cs, @@ -2878,19 +2865,17 @@ static void tlbi_aa64_vmalle1_write(CPUARMState *env, const ARMCPRegInfo *ri, static void tlbi_aa64_vmalle1is_write(CPUARMState *env, const ARMCPRegInfo *ri, uint64_t value) { + CPUState *cs = ENV_GET_CPU(env); bool sec = arm_is_secure_below_el3(env); - CPUState *other_cs; - CPU_FOREACH(other_cs) { - if (sec) { - tlb_flush_by_mmuidx(other_cs, - (1 << ARMMMUIdx_S1SE1) | - (1 << ARMMMUIdx_S1SE0)); - } else { - tlb_flush_by_mmuidx(other_cs, - (1 << ARMMMUIdx_S12NSE1) | - (1 << ARMMMUIdx_S12NSE0)); - } + if (sec) { + tlb_flush_by_mmuidx_all_cpus_synced(cs, + (1 << ARMMMUIdx_S1SE1) | + (1 << ARMMMUIdx_S1SE0)); + } else { + tlb_flush_by_mmuidx_all_cpus_synced(cs, + (1 << ARMMMUIdx_S12NSE1) | + (1 << ARMMMUIdx_S12NSE0)); } } @@ -2947,46 +2932,40 @@ static void tlbi_aa64_alle1is_write(CPUARMState *env, const ARMCPRegInfo *ri, * stage 2 translations, whereas most other scopes only invalidate * stage 1 translations. */ + CPUState *cs = ENV_GET_CPU(env); bool sec = arm_is_secure_below_el3(env); bool has_el2 = arm_feature(env, ARM_FEATURE_EL2); - CPUState *other_cs; - - CPU_FOREACH(other_cs) { - if (sec) { - tlb_flush_by_mmuidx(other_cs, - (1 << ARMMMUIdx_S1SE1) | - (1 << ARMMMUIdx_S1SE0)); - } else if (has_el2) { - tlb_flush_by_mmuidx(other_cs, - (1 << ARMMMUIdx_S12NSE1) | - (1 << ARMMMUIdx_S12NSE0) | - (1 << ARMMMUIdx_S2NS)); - } else { - tlb_flush_by_mmuidx(other_cs, - (1 << ARMMMUIdx_S12NSE1) | - (1 << ARMMMUIdx_S12NSE0)); - } + + if (sec) { + tlb_flush_by_mmuidx_all_cpus_synced(cs, + (1 << ARMMMUIdx_S1SE1) | + (1 << ARMMMUIdx_S1SE0)); + } else if (has_el2) { + tlb_flush_by_mmuidx_all_cpus_synced(cs, + (1 << ARMMMUIdx_S12NSE1) | + (1 << ARMMMUIdx_S12NSE0) | + (1 << ARMMMUIdx_S2NS)); + } else { + tlb_flush_by_mmuidx_all_cpus_synced(cs, + (1 << ARMMMUIdx_S12NSE1) | + (1 << ARMMMUIdx_S12NSE0)); } } static void tlbi_aa64_alle2is_write(CPUARMState *env, const ARMCPRegInfo *ri, uint64_t value) { - CPUState *other_cs; + CPUState *cs = ENV_GET_CPU(env); - CPU_FOREACH(other_cs) { - tlb_flush_by_mmuidx(other_cs, (1 << ARMMMUIdx_S1E2)); - } + tlb_flush_by_mmuidx_all_cpus_synced(cs, (1 << ARMMMUIdx_S1E2)); } static void tlbi_aa64_alle3is_write(CPUARMState *env, const ARMCPRegInfo *ri, uint64_t value) { - CPUState *other_cs; + CPUState *cs = ENV_GET_CPU(env); - CPU_FOREACH(other_cs) { - tlb_flush_by_mmuidx(other_cs, (1 << ARMMMUIdx_S1E3)); - } + tlb_flush_by_mmuidx_all_cpus_synced(cs, (1 << ARMMMUIdx_S1E3)); } static void tlbi_aa64_vae1_write(CPUARMState *env, const ARMCPRegInfo *ri, @@ -3043,43 +3022,40 @@ static void tlbi_aa64_vae3_write(CPUARMState *env, const ARMCPRegInfo *ri, static void tlbi_aa64_vae1is_write(CPUARMState *env, const ARMCPRegInfo *ri, uint64_t value) { + ARMCPU *cpu = arm_env_get_cpu(env); + CPUState *cs = CPU(cpu); bool sec = arm_is_secure_below_el3(env); - CPUState *other_cs; uint64_t pageaddr = sextract64(value << 12, 0, 56); - CPU_FOREACH(other_cs) { - if (sec) { - tlb_flush_page_by_mmuidx(other_cs, pageaddr, - (1 << ARMMMUIdx_S1SE1) | - (1 << ARMMMUIdx_S1SE0)); - } else { - tlb_flush_page_by_mmuidx(other_cs, pageaddr, - (1 << ARMMMUIdx_S12NSE1) | - (1 << ARMMMUIdx_S12NSE0)); - } + if (sec) { + tlb_flush_page_by_mmuidx_all_cpus_synced(cs, pageaddr, + (1 << ARMMMUIdx_S1SE1) | + (1 << ARMMMUIdx_S1SE0)); + } else { + tlb_flush_page_by_mmuidx_all_cpus_synced(cs, pageaddr, + (1 << ARMMMUIdx_S12NSE1) | + (1 << ARMMMUIdx_S12NSE0)); } } static void tlbi_aa64_vae2is_write(CPUARMState *env, const ARMCPRegInfo *ri, uint64_t value) { - CPUState *other_cs; + CPUState *cs = ENV_GET_CPU(env); uint64_t pageaddr = sextract64(value << 12, 0, 56); - CPU_FOREACH(other_cs) { - tlb_flush_page_by_mmuidx(other_cs, pageaddr, (1 << ARMMMUIdx_S1E2)); - } + tlb_flush_page_by_mmuidx_all_cpus_synced(cs, pageaddr, + (1 << ARMMMUIdx_S1E2)); } static void tlbi_aa64_vae3is_write(CPUARMState *env, const ARMCPRegInfo *ri, uint64_t value) { - CPUState *other_cs; + CPUState *cs = ENV_GET_CPU(env); uint64_t pageaddr = sextract64(value << 12, 0, 56); - CPU_FOREACH(other_cs) { - tlb_flush_page_by_mmuidx(other_cs, pageaddr, (1 << ARMMMUIdx_S1E3)); - } + tlb_flush_page_by_mmuidx_all_cpus_synced(cs, pageaddr, + (1 << ARMMMUIdx_S1E3)); } static void tlbi_aa64_ipas2e1_write(CPUARMState *env, const ARMCPRegInfo *ri, @@ -3107,7 +3083,7 @@ static void tlbi_aa64_ipas2e1_write(CPUARMState *env, const ARMCPRegInfo *ri, static void tlbi_aa64_ipas2e1is_write(CPUARMState *env, const ARMCPRegInfo *ri, uint64_t value) { - CPUState *other_cs; + CPUState *cs = ENV_GET_CPU(env); uint64_t pageaddr; if (!arm_feature(env, ARM_FEATURE_EL2) || !(env->cp15.scr_el3 & SCR_NS)) { @@ -3116,9 +3092,8 @@ static void tlbi_aa64_ipas2e1is_write(CPUARMState *env, const ARMCPRegInfo *ri, pageaddr = sextract64(value << 12, 0, 48); - CPU_FOREACH(other_cs) { - tlb_flush_page_by_mmuidx(other_cs, pageaddr, (1 << ARMMMUIdx_S2NS)); - } + tlb_flush_page_by_mmuidx_all_cpus_synced(cs, pageaddr, + (1 << ARMMMUIdx_S2NS)); } static CPAccessResult aa64_zva_access(CPUARMState *env, const ARMCPRegInfo *ri, @@ -3275,27 +3250,27 @@ static const ARMCPRegInfo v8_cp_reginfo[] = { /* TLBI operations */ { .name = "TLBI_VMALLE1IS", .state = ARM_CP_STATE_AA64, .opc0 = 1, .opc1 = 0, .crn = 8, .crm = 3, .opc2 = 0, - .access = PL1_W, .type = ARM_CP_NO_RAW, + .access = PL1_W, .type = ARM_CP_NO_RAW | ARM_CP_EXIT_PC, .writefn = tlbi_aa64_vmalle1is_write }, { .name = "TLBI_VAE1IS", .state = ARM_CP_STATE_AA64, .opc0 = 1, .opc1 = 0, .crn = 8, .crm = 3, .opc2 = 1, - .access = PL1_W, .type = ARM_CP_NO_RAW, + .access = PL1_W, .type = ARM_CP_NO_RAW | ARM_CP_EXIT_PC, .writefn = tlbi_aa64_vae1is_write }, { .name = "TLBI_ASIDE1IS", .state = ARM_CP_STATE_AA64, .opc0 = 1, .opc1 = 0, .crn = 8, .crm = 3, .opc2 = 2, - .access = PL1_W, .type = ARM_CP_NO_RAW, + .access = PL1_W, .type = ARM_CP_NO_RAW | ARM_CP_EXIT_PC, .writefn = tlbi_aa64_vmalle1is_write }, { .name = "TLBI_VAAE1IS", .state = ARM_CP_STATE_AA64, .opc0 = 1, .opc1 = 0, .crn = 8, .crm = 3, .opc2 = 3, - .access = PL1_W, .type = ARM_CP_NO_RAW, + .access = PL1_W, .type = ARM_CP_NO_RAW | ARM_CP_EXIT_PC, .writefn = tlbi_aa64_vae1is_write }, { .name = "TLBI_VALE1IS", .state = ARM_CP_STATE_AA64, .opc0 = 1, .opc1 = 0, .crn = 8, .crm = 3, .opc2 = 5, - .access = PL1_W, .type = ARM_CP_NO_RAW, + .access = PL1_W, .type = ARM_CP_NO_RAW | ARM_CP_EXIT_PC, .writefn = tlbi_aa64_vae1is_write }, { .name = "TLBI_VAALE1IS", .state = ARM_CP_STATE_AA64, .opc0 = 1, .opc1 = 0, .crn = 8, .crm = 3, .opc2 = 7, - .access = PL1_W, .type = ARM_CP_NO_RAW, + .access = PL1_W, .type = ARM_CP_NO_RAW | ARM_CP_EXIT_PC, .writefn = tlbi_aa64_vae1is_write }, { .name = "TLBI_VMALLE1", .state = ARM_CP_STATE_AA64, .opc0 = 1, .opc1 = 0, .crn = 8, .crm = 7, .opc2 = 0, @@ -3323,19 +3298,19 @@ static const ARMCPRegInfo v8_cp_reginfo[] = { .writefn = tlbi_aa64_vae1_write }, { .name = "TLBI_IPAS2E1IS", .state = ARM_CP_STATE_AA64, .opc0 = 1, .opc1 = 4, .crn = 8, .crm = 0, .opc2 = 1, - .access = PL2_W, .type = ARM_CP_NO_RAW, + .access = PL2_W, .type = ARM_CP_NO_RAW | ARM_CP_EXIT_PC, .writefn = tlbi_aa64_ipas2e1is_write }, { .name = "TLBI_IPAS2LE1IS", .state = ARM_CP_STATE_AA64, .opc0 = 1, .opc1 = 4, .crn = 8, .crm = 0, .opc2 = 5, - .access = PL2_W, .type = ARM_CP_NO_RAW, + .access = PL2_W, .type = ARM_CP_NO_RAW | ARM_CP_EXIT_PC, .writefn = tlbi_aa64_ipas2e1is_write }, { .name = "TLBI_ALLE1IS", .state = ARM_CP_STATE_AA64, .opc0 = 1, .opc1 = 4, .crn = 8, .crm = 3, .opc2 = 4, - .access = PL2_W, .type = ARM_CP_NO_RAW, + .access = PL2_W, .type = ARM_CP_NO_RAW | ARM_CP_EXIT_PC, .writefn = tlbi_aa64_alle1is_write }, { .name = "TLBI_VMALLS12E1IS", .state = ARM_CP_STATE_AA64, .opc0 = 1, .opc1 = 4, .crn = 8, .crm = 3, .opc2 = 6, - .access = PL2_W, .type = ARM_CP_NO_RAW, + .access = PL2_W, .type = ARM_CP_NO_RAW | ARM_CP_EXIT_PC, .writefn = tlbi_aa64_alle1is_write }, { .name = "TLBI_IPAS2E1", .state = ARM_CP_STATE_AA64, .opc0 = 1, .opc1 = 4, .crn = 8, .crm = 4, .opc2 = 1, @@ -3351,7 +3326,7 @@ static const ARMCPRegInfo v8_cp_reginfo[] = { .writefn = tlbi_aa64_alle1_write }, { .name = "TLBI_VMALLS12E1", .state = ARM_CP_STATE_AA64, .opc0 = 1, .opc1 = 4, .crn = 8, .crm = 7, .opc2 = 6, - .access = PL2_W, .type = ARM_CP_NO_RAW, + .access = PL2_W, .type = ARM_CP_NO_RAW | ARM_CP_EXIT_PC, .writefn = tlbi_aa64_alle1is_write }, #ifndef CONFIG_USER_ONLY /* 64 bit address translation operations */ @@ -3397,7 +3372,7 @@ static const ARMCPRegInfo v8_cp_reginfo[] = { { .name = "TLBIMVALIS", .cp = 15, .opc1 = 0, .crn = 8, .crm = 3, .opc2 = 5, .type = ARM_CP_NO_RAW, .access = PL1_W, .writefn = tlbimva_is_write }, { .name = "TLBIMVAALIS", .cp = 15, .opc1 = 0, .crn = 8, .crm = 3, .opc2 = 7, - .type = ARM_CP_NO_RAW, .access = PL1_W, + .type = ARM_CP_NO_RAW | ARM_CP_EXIT_PC, .access = PL1_W, .writefn = tlbimvaa_is_write }, { .name = "TLBIMVAL", .cp = 15, .opc1 = 0, .crn = 8, .crm = 7, .opc2 = 5, .type = ARM_CP_NO_RAW, .access = PL1_W, .writefn = tlbimva_write }, @@ -3408,7 +3383,7 @@ static const ARMCPRegInfo v8_cp_reginfo[] = { .writefn = tlbimva_hyp_write }, { .name = "TLBIMVALHIS", .cp = 15, .opc1 = 4, .crn = 8, .crm = 3, .opc2 = 5, - .type = ARM_CP_NO_RAW, .access = PL2_W, + .type = ARM_CP_NO_RAW | ARM_CP_EXIT_PC, .access = PL2_W, .writefn = tlbimva_hyp_is_write }, { .name = "TLBIIPAS2", .cp = 15, .opc1 = 4, .crn = 8, .crm = 4, .opc2 = 1, @@ -3416,7 +3391,7 @@ static const ARMCPRegInfo v8_cp_reginfo[] = { .writefn = tlbiipas2_write }, { .name = "TLBIIPAS2IS", .cp = 15, .opc1 = 4, .crn = 8, .crm = 0, .opc2 = 1, - .type = ARM_CP_NO_RAW, .access = PL2_W, + .type = ARM_CP_NO_RAW | ARM_CP_EXIT_PC, .access = PL2_W, .writefn = tlbiipas2_is_write }, { .name = "TLBIIPAS2L", .cp = 15, .opc1 = 4, .crn = 8, .crm = 4, .opc2 = 5, @@ -3424,7 +3399,7 @@ static const ARMCPRegInfo v8_cp_reginfo[] = { .writefn = tlbiipas2_write }, { .name = "TLBIIPAS2LIS", .cp = 15, .opc1 = 4, .crn = 8, .crm = 0, .opc2 = 5, - .type = ARM_CP_NO_RAW, .access = PL2_W, + .type = ARM_CP_NO_RAW | ARM_CP_EXIT_PC, .access = PL2_W, .writefn = tlbiipas2_is_write }, /* 32 bit cache operations */ { .name = "ICIALLUIS", .cp = 15, .opc1 = 0, .crn = 7, .crm = 1, .opc2 = 0, @@ -3764,7 +3739,7 @@ static const ARMCPRegInfo el2_cp_reginfo[] = { .writefn = tlbiall_nsnh_write }, { .name = "TLBIALLNSNHIS", .cp = 15, .opc1 = 4, .crn = 8, .crm = 3, .opc2 = 4, - .type = ARM_CP_NO_RAW, .access = PL2_W, + .type = ARM_CP_NO_RAW | ARM_CP_EXIT_PC, .access = PL2_W, .writefn = tlbiall_nsnh_is_write }, { .name = "TLBIALLH", .cp = 15, .opc1 = 4, .crn = 8, .crm = 7, .opc2 = 0, .type = ARM_CP_NO_RAW, .access = PL2_W, @@ -3776,7 +3751,7 @@ static const ARMCPRegInfo el2_cp_reginfo[] = { .type = ARM_CP_NO_RAW, .access = PL2_W, .writefn = tlbimva_hyp_write }, { .name = "TLBIMVAHIS", .cp = 15, .opc1 = 4, .crn = 8, .crm = 3, .opc2 = 1, - .type = ARM_CP_NO_RAW, .access = PL2_W, + .type = ARM_CP_NO_RAW | ARM_CP_EXIT_PC, .access = PL2_W, .writefn = tlbimva_hyp_is_write }, { .name = "TLBI_ALLE2", .state = ARM_CP_STATE_AA64, .opc0 = 1, .opc1 = 4, .crn = 8, .crm = 7, .opc2 = 0, @@ -3792,15 +3767,15 @@ static const ARMCPRegInfo el2_cp_reginfo[] = { .writefn = tlbi_aa64_vae2_write }, { .name = "TLBI_ALLE2IS", .state = ARM_CP_STATE_AA64, .opc0 = 1, .opc1 = 4, .crn = 8, .crm = 3, .opc2 = 0, - .access = PL2_W, .type = ARM_CP_NO_RAW, + .access = PL2_W, .type = ARM_CP_NO_RAW | ARM_CP_EXIT_PC, .writefn = tlbi_aa64_alle2is_write }, { .name = "TLBI_VAE2IS", .state = ARM_CP_STATE_AA64, .opc0 = 1, .opc1 = 4, .crn = 8, .crm = 3, .opc2 = 1, - .type = ARM_CP_NO_RAW, .access = PL2_W, + .type = ARM_CP_NO_RAW | ARM_CP_EXIT_PC, .access = PL2_W, .writefn = tlbi_aa64_vae2is_write }, { .name = "TLBI_VALE2IS", .state = ARM_CP_STATE_AA64, .opc0 = 1, .opc1 = 4, .crn = 8, .crm = 3, .opc2 = 5, - .access = PL2_W, .type = ARM_CP_NO_RAW, + .access = PL2_W, .type = ARM_CP_NO_RAW | ARM_CP_EXIT_PC, .writefn = tlbi_aa64_vae2is_write }, #ifndef CONFIG_USER_ONLY /* Unlike the other EL2-related AT operations, these must @@ -3987,15 +3962,15 @@ static const ARMCPRegInfo el3_cp_reginfo[] = { .resetvalue = 0 }, { .name = "TLBI_ALLE3IS", .state = ARM_CP_STATE_AA64, .opc0 = 1, .opc1 = 6, .crn = 8, .crm = 3, .opc2 = 0, - .access = PL3_W, .type = ARM_CP_NO_RAW, + .access = PL3_W, .type = ARM_CP_NO_RAW | ARM_CP_EXIT_PC, .writefn = tlbi_aa64_alle3is_write }, { .name = "TLBI_VAE3IS", .state = ARM_CP_STATE_AA64, .opc0 = 1, .opc1 = 6, .crn = 8, .crm = 3, .opc2 = 1, - .access = PL3_W, .type = ARM_CP_NO_RAW, + .access = PL3_W, .type = ARM_CP_NO_RAW | ARM_CP_EXIT_PC, .writefn = tlbi_aa64_vae3is_write }, { .name = "TLBI_VALE3IS", .state = ARM_CP_STATE_AA64, .opc0 = 1, .opc1 = 6, .crn = 8, .crm = 3, .opc2 = 5, - .access = PL3_W, .type = ARM_CP_NO_RAW, + .access = PL3_W, .type = ARM_CP_NO_RAW | ARM_CP_EXIT_PC, .writefn = tlbi_aa64_vae3is_write }, { .name = "TLBI_ALLE3", .state = ARM_CP_STATE_AA64, .opc0 = 1, .opc1 = 6, .crn = 8, .crm = 7, .opc2 = 0,