From patchwork Mon Mar 14 12:52:59 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laszlo Ersek X-Patchwork-Id: 63808 Delivered-To: patch@linaro.org Received: by 10.112.199.169 with SMTP id jl9csp17692lbc; Mon, 14 Mar 2016 05:53:12 -0700 (PDT) X-Received: by 10.66.220.7 with SMTP id ps7mr37885136pac.58.1457959992336; Mon, 14 Mar 2016 05:53:12 -0700 (PDT) Return-Path: Received: from ml01.01.org (ml01.01.org. [198.145.21.10]) by mx.google.com with ESMTPS id tf6si7283991pac.233.2016.03.14.05.53.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 14 Mar 2016 05:53:12 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of edk2-devel-bounces@lists.01.org designates 198.145.21.10 as permitted sender) client-ip=198.145.21.10; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of edk2-devel-bounces@lists.01.org designates 198.145.21.10 as permitted sender) smtp.mailfrom=edk2-devel-bounces@lists.01.org Received: from [127.0.0.1] (localhost [IPv6:::1]) by ml01.01.org (Postfix) with ESMTP id 8EA821A1FA1; Mon, 14 Mar 2016 05:53:29 -0700 (PDT) X-Original-To: edk2-devel@ml01.01.org Delivered-To: edk2-devel@ml01.01.org Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 98CD61A1FA1 for ; Mon, 14 Mar 2016 05:53:28 -0700 (PDT) Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26]) by mx1.redhat.com (Postfix) with ESMTPS id 56E32165ED7; Mon, 14 Mar 2016 12:53:10 +0000 (UTC) Received: from lacos-laptop-7.usersys.redhat.com (ovpn-113-101.phx2.redhat.com [10.3.113.101]) by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u2ECr3Zn003103; Mon, 14 Mar 2016 08:53:08 -0400 From: Laszlo Ersek To: edk2-devel@ml01.01.org Date: Mon, 14 Mar 2016 13:52:59 +0100 Message-Id: <1457959980-29438-3-git-send-email-lersek@redhat.com> In-Reply-To: <1457959980-29438-1-git-send-email-lersek@redhat.com> References: <56E6B2D9.5010507@redhat.com> <1457959980-29438-1-git-send-email-lersek@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.26 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Mon, 14 Mar 2016 12:53:10 +0000 (UTC) Cc: Marcel Apfelbaum , Jordan Justen Subject: [edk2] [wave 2 PATCH 2/3] OvmfPkg: PlatformPei: determine the 64-bit PCI host aperture for X64 DXE X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: edk2-devel-bounces@lists.01.org Sender: "edk2-devel" The main observation about the 64-bit PCI host aperture is that it is the highest part of the useful address space. It impacts the top of the GCD memory space map, and, consequently, our maximum address width calculation for the CPU HOB too. Thus, modify the GetFirstNonAddress() function to consider the following areas above the high RAM, while calculating the first non-address (i.e., the highest inclusive address, plus one): - the memory hotplug area (optional, the size comes from QEMU), - the 64-bit PCI host aperture (we set a default size). While computing the first non-address, capture the base and the size of the 64-bit PCI host aperture at once in PCDs, since they are natural parts of the calculation. (Similarly to how PcdPciMmio32* are not rewritten on the S3 resume path (see the InitializePlatform() -> MemMapInitialization() condition), nor are PcdPciMmio64*. Only the core PciHostBridgeDxe driver consumes them, through our PciHostBridgeLib instance.) Set 32GB as the default size for the aperture. Issue#59 mentions the NVIDIA Tesla K80 as an assignable device. According to nvidia.com, these cards may have 24GB of memory (probably 16GB + 8GB BARs). As a strictly experimental feature, the user can specify the size of the aperture (in MB) as well, with the QEMU option -fw_cfg name=opt/ovmf/X-PciMmio64Mb,string=65536 The "X-" prefix follows the QEMU tradition (spelled "x-" there), meaning that the property is experimental, unstable, and might go away any time. Gerd has proposed heuristics for sizing the aperture automatically (based on 1GB page support and PCPU address width), but such should be delayed to a later patch (which may very well back out "X-PciMmio64Mb" then). For "everyday" guests, the 32GB default for the aperture size shouldn't impact the PEI memory demand (the size of the page tables that the DXE IPL PEIM builds). Namely, we've never reported narrower than 36-bit addresses; the DXE IPL PEIM has always built page tables for 64GB at least. For the aperture to bump the address width above 36 bits, either the guest must have quite a bit of memory itself (in which case the additional PEI memory demand shouldn't matter), or the user must specify a large aperture manually with "X-PciMmio64Mb" (and then he or she is also responsible for giving enough RAM to the VM, to satisfy the PEI memory demand). Cc: Gerd Hoffmann Cc: Jordan Justen Cc: Marcel Apfelbaum Cc: Thomas Lamprecht Ref: https://github.com/tianocore/edk2/issues/59 Ref: http://www.nvidia.com/object/tesla-servers.html Contributed-under: TianoCore Contribution Agreement 1.0 Signed-off-by: Laszlo Ersek --- OvmfPkg/OvmfPkg.dec | 5 + OvmfPkg/OvmfPkgIa32X64.dsc | 2 + OvmfPkg/OvmfPkgX64.dsc | 2 + OvmfPkg/PlatformPei/PlatformPei.inf | 2 + OvmfPkg/PlatformPei/MemDetect.c | 110 ++++++++++++++++++++ 5 files changed, 121 insertions(+) -- 1.8.3.1 _______________________________________________ edk2-devel mailing list edk2-devel@lists.01.org https://lists.01.org/mailman/listinfo/edk2-devel diff --git a/OvmfPkg/OvmfPkg.dec b/OvmfPkg/OvmfPkg.dec index a7d887ad353c..56bd511bb0f8 100644 --- a/OvmfPkg/OvmfPkg.dec +++ b/OvmfPkg/OvmfPkg.dec @@ -127,14 +127,19 @@ [PcdsDynamic, PcdsDynamicEx] gUefiOvmfPkgTokenSpaceGuid.PcdQemuSmbiosValidated|FALSE|BOOLEAN|0x21 ## The 32-bit MMIO aperture shared by all PCI root bridges. # gUefiOvmfPkgTokenSpaceGuid.PcdPciMmio32Base|0x0|UINT64|0x24 gUefiOvmfPkgTokenSpaceGuid.PcdPciMmio32Size|0x0|UINT64|0x25 + ## The 64-bit MMIO aperture shared by all PCI root bridges. + # + gUefiOvmfPkgTokenSpaceGuid.PcdPciMmio64Base|0x0|UINT64|0x26 + gUefiOvmfPkgTokenSpaceGuid.PcdPciMmio64Size|0x0|UINT64|0x27 + [PcdsFeatureFlag] gUefiOvmfPkgTokenSpaceGuid.PcdSecureBootEnable|FALSE|BOOLEAN|3 gUefiOvmfPkgTokenSpaceGuid.PcdQemuBootOrderPciTranslation|TRUE|BOOLEAN|0x1c gUefiOvmfPkgTokenSpaceGuid.PcdQemuBootOrderMmioTranslation|FALSE|BOOLEAN|0x1d ## This feature flag enables SMM/SMRAM support. Note that it also requires # such support from the underlying QEMU instance; if that support is not diff --git a/OvmfPkg/OvmfPkgIa32X64.dsc b/OvmfPkg/OvmfPkgIa32X64.dsc index e9ffcb418522..eb16b9fb82ee 100644 --- a/OvmfPkg/OvmfPkgIa32X64.dsc +++ b/OvmfPkg/OvmfPkgIa32X64.dsc @@ -463,14 +463,16 @@ [PcdsDynamicDefault] gEfiMdeModulePkgTokenSpaceGuid.PcdFlashNvStorageFtwSpareBase|0 gEfiMdeModulePkgTokenSpaceGuid.PcdPciDisableBusEnumeration|FALSE gEfiMdeModulePkgTokenSpaceGuid.PcdVideoHorizontalResolution|800 gEfiMdeModulePkgTokenSpaceGuid.PcdVideoVerticalResolution|600 gUefiOvmfPkgTokenSpaceGuid.PcdOvmfHostBridgePciDevId|0 gUefiOvmfPkgTokenSpaceGuid.PcdPciMmio32Base|0x0 gUefiOvmfPkgTokenSpaceGuid.PcdPciMmio32Size|0x0 + gUefiOvmfPkgTokenSpaceGuid.PcdPciMmio64Base|0x0 + gUefiOvmfPkgTokenSpaceGuid.PcdPciMmio64Size|0x800000000 gEfiMdePkgTokenSpaceGuid.PcdPlatformBootTimeOut|0 # Set video resolution for text setup. gEfiMdeModulePkgTokenSpaceGuid.PcdSetupVideoHorizontalResolution|640 gEfiMdeModulePkgTokenSpaceGuid.PcdSetupVideoVerticalResolution|480 diff --git a/OvmfPkg/OvmfPkgX64.dsc b/OvmfPkg/OvmfPkgX64.dsc index c983a544d5f3..2d955b3df5f5 100644 --- a/OvmfPkg/OvmfPkgX64.dsc +++ b/OvmfPkg/OvmfPkgX64.dsc @@ -462,14 +462,16 @@ [PcdsDynamicDefault] gEfiMdeModulePkgTokenSpaceGuid.PcdFlashNvStorageFtwSpareBase|0 gEfiMdeModulePkgTokenSpaceGuid.PcdPciDisableBusEnumeration|FALSE gEfiMdeModulePkgTokenSpaceGuid.PcdVideoHorizontalResolution|800 gEfiMdeModulePkgTokenSpaceGuid.PcdVideoVerticalResolution|600 gUefiOvmfPkgTokenSpaceGuid.PcdOvmfHostBridgePciDevId|0 gUefiOvmfPkgTokenSpaceGuid.PcdPciMmio32Base|0x0 gUefiOvmfPkgTokenSpaceGuid.PcdPciMmio32Size|0x0 + gUefiOvmfPkgTokenSpaceGuid.PcdPciMmio64Base|0x0 + gUefiOvmfPkgTokenSpaceGuid.PcdPciMmio64Size|0x800000000 gEfiMdePkgTokenSpaceGuid.PcdPlatformBootTimeOut|0 # Set video resolution for text setup. gEfiMdeModulePkgTokenSpaceGuid.PcdSetupVideoHorizontalResolution|640 gEfiMdeModulePkgTokenSpaceGuid.PcdSetupVideoVerticalResolution|480 diff --git a/OvmfPkg/PlatformPei/PlatformPei.inf b/OvmfPkg/PlatformPei/PlatformPei.inf index 6dc5ff079f53..5b643a358f1f 100644 --- a/OvmfPkg/PlatformPei/PlatformPei.inf +++ b/OvmfPkg/PlatformPei/PlatformPei.inf @@ -75,14 +75,16 @@ [Pcd] gUefiOvmfPkgTokenSpaceGuid.PcdOvmfLockBoxStorageSize gUefiOvmfPkgTokenSpaceGuid.PcdGuidedExtractHandlerTableSize gUefiOvmfPkgTokenSpaceGuid.PcdOvmfHostBridgePciDevId gUefiOvmfPkgTokenSpaceGuid.PcdPciIoBase gUefiOvmfPkgTokenSpaceGuid.PcdPciIoSize gUefiOvmfPkgTokenSpaceGuid.PcdPciMmio32Base gUefiOvmfPkgTokenSpaceGuid.PcdPciMmio32Size + gUefiOvmfPkgTokenSpaceGuid.PcdPciMmio64Base + gUefiOvmfPkgTokenSpaceGuid.PcdPciMmio64Size gUefiOvmfPkgTokenSpaceGuid.PcdOvmfDecompressionScratchEnd gUefiOvmfPkgTokenSpaceGuid.PcdQ35TsegMbytes gEfiIntelFrameworkModulePkgTokenSpaceGuid.PcdS3AcpiReservedMemorySize gEfiMdePkgTokenSpaceGuid.PcdGuidedExtractHandlerTableAddress gEfiMdeModulePkgTokenSpaceGuid.PcdVariableStoreSize gEfiMdeModulePkgTokenSpaceGuid.PcdFlashNvStorageFtwSpareSize gEfiMdeModulePkgTokenSpaceGuid.PcdFlashNvStorageVariableSize diff --git a/OvmfPkg/PlatformPei/MemDetect.c b/OvmfPkg/PlatformPei/MemDetect.c index 286f6914a702..ed13c57beb2f 100644 --- a/OvmfPkg/PlatformPei/MemDetect.c +++ b/OvmfPkg/PlatformPei/MemDetect.c @@ -28,14 +28,15 @@ Module Name: #include #include #include #include #include #include #include +#include #include "Platform.h" #include "Cmos.h" UINT8 mPhysMemAddressWidth; UINT32 @@ -93,16 +94,125 @@ GetSystemMemorySizeAbove4gb ( STATIC UINT64 GetFirstNonAddress ( VOID ) { UINT64 FirstNonAddress; + UINT64 Pci64Base, Pci64Size; + CHAR8 MbString[7 + 1]; + EFI_STATUS Status; + FIRMWARE_CONFIG_ITEM FwCfgItem; + UINTN FwCfgSize; + UINT64 HotPlugMemoryEnd; FirstNonAddress = BASE_4GB + GetSystemMemorySizeAbove4gb (); + + // + // If DXE is 32-bit, then we're done; PciBusDxe will degrade 64-bit MMIO + // resources to 32-bit anyway. See DegradeResource() in + // "PciResourceSupport.c". + // +#ifdef MDE_CPU_IA32 + if (!FeaturePcdGet (PcdDxeIplSwitchToLongMode)) { + return FirstNonAddress; + } +#endif + + // + // Otherwise, in order to calculate the highest address plus one, we must + // consider the 64-bit PCI host aperture too. Fetch the default size. + // + Pci64Size = PcdGet64 (PcdPciMmio64Size); + + // + // See if the user specified the number of megabytes for the 64-bit PCI host + // aperture. The number of non-NUL characters in MbString allows for + // 9,999,999 MB, which is approximately 10 TB. + // + // As signaled by the "X-" prefix, this knob is experimental, and might go + // away at any time. + // + Status = QemuFwCfgFindFile ("opt/ovmf/X-PciMmio64Mb", &FwCfgItem, + &FwCfgSize); + if (!EFI_ERROR (Status)) { + if (FwCfgSize >= sizeof MbString) { + DEBUG ((EFI_D_WARN, + "%a: ignoring malformed 64-bit PCI host aperture size from fw_cfg\n", + __FUNCTION__)); + } else { + QemuFwCfgSelectItem (FwCfgItem); + QemuFwCfgReadBytes (FwCfgSize, MbString); + MbString[FwCfgSize] = '\0'; + Pci64Size = LShiftU64 (AsciiStrDecimalToUint64 (MbString), 20); + } + } + + if (Pci64Size == 0) { + if (mBootMode != BOOT_ON_S3_RESUME) { + DEBUG ((EFI_D_INFO, "%a: disabling 64-bit PCI host aperture\n", + __FUNCTION__)); + PcdSet64 (PcdPciMmio64Size, 0); + } + + // + // There's nothing more to do; the amount of memory above 4GB fully + // determines the highest address plus one. The memory hotplug area (see + // below) plays no role for the firmware in this case. + // + return FirstNonAddress; + } + + // + // The "etc/reserved-memory-end" fw_cfg file, when present, contains an + // absolute, exclusive end address for the memory hotplug area. This area + // starts right at the end of the memory above 4GB. The 64-bit PCI host + // aperture must be placed above it. + // + Status = QemuFwCfgFindFile ("etc/reserved-memory-end", &FwCfgItem, + &FwCfgSize); + if (!EFI_ERROR (Status) && FwCfgSize == sizeof HotPlugMemoryEnd) { + QemuFwCfgSelectItem (FwCfgItem); + QemuFwCfgReadBytes (FwCfgSize, &HotPlugMemoryEnd); + + ASSERT (HotPlugMemoryEnd >= FirstNonAddress); + FirstNonAddress = HotPlugMemoryEnd; + } + + // + // SeaBIOS aligns both boundaries of the 64-bit PCI host aperture to 1GB, so + // that the host can map it with 1GB hugepages. Follow suit. + // + Pci64Base = ALIGN_VALUE (FirstNonAddress, (UINT64)SIZE_1GB); + Pci64Size = ALIGN_VALUE (Pci64Size, (UINT64)SIZE_1GB); + + // + // The 64-bit PCI host aperture should also be "naturally" aligned. The + // alignment is determined by rounding the size of the aperture down to the + // next smaller or equal power of two. That is, align the aperture by the + // largest BAR size that can fit into it. + // + Pci64Base = ALIGN_VALUE (Pci64Base, GetPowerOfTwo64 (Pci64Size)); + + if (mBootMode != BOOT_ON_S3_RESUME) { + // + // The core PciHostBridgeDxe driver will automatically add this range to + // the GCD memory space map through our PciHostBridgeLib instance; here we + // only need to set the PCDs. + // + PcdSet64 (PcdPciMmio64Base, Pci64Base); + PcdSet64 (PcdPciMmio64Size, Pci64Size); + DEBUG ((EFI_D_INFO, "%a: Pci64Base=0x%Lx Pci64Size=0x%Lx\n", + __FUNCTION__, Pci64Base, Pci64Size)); + } + + // + // The useful address space ends with the 64-bit PCI host aperture. + // + FirstNonAddress = Pci64Base + Pci64Size; return FirstNonAddress; } /** Initialize the mPhysMemAddressWidth variable, based on guest RAM size. **/