[v4,1/8] platform/x86/intel: Add Primary to Sideband (P2SB) bridge support

From: Jonathan Yong <jonathan.yong@intel.com>

There are already two and at least one more user is coming which
require an access to Primary to Sideband (P2SB) bridge in order
to get IO or MMIO BAR hidden by BIOS.

Create a library to access P2SB for x86 devices.

Background information
======================
Note, the term "bridge" is used in the documentation and it has nothing
to do with a PCI (host) bridge as per the PCI specifications.

The P2SB is an interesting device by its nature and hardware design.
First of all, it has several devices in the hardware behind it. These
devices may or may not be represented as ACPI devices by a firmware.

It also has a hardwired (to 0s) the least significant bits of the
base address register which is represented by the only 64-bit BAR0.
It means that OS mustn't reallocate the BAR.

On top of that in some cases P2SB is represented by function 0 on PCI
slot (in terms of B:D.F) and according to the PCI specification any
other function can't be seen until function 0 is present and visible.

In the PCI configuration space of P2SB device the full 32-bit register
is allocated for the only purpose of hiding the entire P2SB device. As
per [3]:

  3.1.39 P2SB Control (P2SBC)—Offset E0h

  Hide Device (HIDE): When this bit is set, the P2SB will return 1s on
  any PCI Configuration Read on IOSF-P. All other transactions including
  PCI Configuration Writes on IOSF-P are unaffected by this. This does
  not affect reads performed on the IOSF-SB interface.

This doesn't prevent MMIO accesses, although preventing the OS from
assigning these addresses. The firmware on the affected platforms marks
the region as unusable (by cutting it off from the PCI host bridge
resources) as depicted in the Apollo Lake example below:

  PCI host bridge to bus 0000:00
  pci_bus 0000:00: root bus resource [io  0x0070-0x0077]
  pci_bus 0000:00: root bus resource [io  0x0000-0x006f window]
  pci_bus 0000:00: root bus resource [io  0x0078-0x0cf7 window]
  pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
  pci_bus 0000:00: root bus resource [mem 0x7c000001-0x7fffffff window]
  pci_bus 0000:00: root bus resource [mem 0x7b800001-0x7bffffff window]
  pci_bus 0000:00: root bus resource [mem 0x80000000-0xcfffffff window]
  pci_bus 0000:00: root bus resource [mem 0xe0000000-0xefffffff window]
  pci_bus 0000:00: root bus resource [bus 00-ff]

The P2SB 16MB BAR is located at 0xd0000000-0xd0ffffff memory window.

The generic solution
====================
The generic solution for all cases when we need to access to the information
behind P2SB device is a library code where users ask for necessary resources
by demand and hence those users take care of not being run on the systems
where this access is not required.

The library provides the p2sb_bar() API to retrieve the MMIO of the BAR0 of
the device from P2SB device slot.

P2SB unconditional unhiding awareness
=====================================
Technically it's possible to unhide the P2SB device and devices on
the same PCI slot and access them at any time as needed. But there are
several potential issues with that:

 - the systems were never tested against such configuration and hence
   nobody knows what kind of bugs it may bring, especially when we talk
   about SPI NOR case which contains Intel FirmWare Image (IFWI) code
   (including BIOS) and already known to be problematic in the past for
   end users

 - the PCI by its nature is a hotpluggable bus and in case somebody
   attaches a driver to the functions of a P2SB slot device(s) the
   end user experience and system behaviour can be unpredictable

 - the kernel code would need some ugly hacks (or code looking as an
   ugly hack) under arch/x86/pci in order to enable these devices on
   only selected platforms (which may include CPU ID table followed by
   a potentially growing number of DMI strings

The future improvements
=======================
The future improvements with this code may go in order to gain some kind
of cache, if it's possible at all, to prevent unhiding and hiding many
times to take static information that may be saved once per boot.

Links
=====
[1]: https://lab.whitequark.org/notes/2017-11-08/accessing-intel-ich-pch-gpios/
[2]: https://cdrdv2.intel.com/v1/dl/getContent/332690?wapkw=332690
[3]: https://cdrdv2.intel.com/v1/dl/getContent/332691?wapkw=332691
[4]: https://medium.com/@jacksonchen_43335/bios-gpio-p2sb-70e9b829b403

Signed-off-by: Jonathan Yong <jonathan.yong@intel.com>
Co-developed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Acked-by: Hans de Goede <hdegoede@redhat.com>
Acked-by: Linus Walleij <linus.walleij@linaro.org>
---
 drivers/platform/x86/intel/Kconfig     |  12 +
 drivers/platform/x86/intel/Makefile    |   1 +
 drivers/platform/x86/intel/p2sb.c      | 299 +++++++++++++++++++++++++
 include/linux/platform_data/x86/p2sb.h |  27 +++
 4 files changed, 339 insertions(+)
 create mode 100644 drivers/platform/x86/intel/p2sb.c
 create mode 100644 include/linux/platform_data/x86/p2sb.h

Hi Andy,

On 1/31/22 16:13, Andy Shevchenko wrote:
> From: Jonathan Yong <jonathan.yong@intel.com>
> 
> There are already two and at least one more user is coming which
> require an access to Primary to Sideband (P2SB) bridge in order
> to get IO or MMIO BAR hidden by BIOS.
> 
> Create a library to access P2SB for x86 devices.
> 
> Background information
> ======================
> Note, the term "bridge" is used in the documentation and it has nothing
> to do with a PCI (host) bridge as per the PCI specifications.
> 
> The P2SB is an interesting device by its nature and hardware design.
> First of all, it has several devices in the hardware behind it. These
> devices may or may not be represented as ACPI devices by a firmware.
> 
> It also has a hardwired (to 0s) the least significant bits of the
> base address register which is represented by the only 64-bit BAR0.
> It means that OS mustn't reallocate the BAR.
> 
> On top of that in some cases P2SB is represented by function 0 on PCI
> slot (in terms of B:D.F) and according to the PCI specification any
> other function can't be seen until function 0 is present and visible.
> 
> In the PCI configuration space of P2SB device the full 32-bit register
> is allocated for the only purpose of hiding the entire P2SB device. As
> per [3]:
> 
>   3.1.39 P2SB Control (P2SBC)—Offset E0h
> 
>   Hide Device (HIDE): When this bit is set, the P2SB will return 1s on
>   any PCI Configuration Read on IOSF-P. All other transactions including
>   PCI Configuration Writes on IOSF-P are unaffected by this. This does
>   not affect reads performed on the IOSF-SB interface.
> 
> This doesn't prevent MMIO accesses, although preventing the OS from
> assigning these addresses. The firmware on the affected platforms marks
> the region as unusable (by cutting it off from the PCI host bridge
> resources) as depicted in the Apollo Lake example below:
> 
>   PCI host bridge to bus 0000:00
>   pci_bus 0000:00: root bus resource [io  0x0070-0x0077]
>   pci_bus 0000:00: root bus resource [io  0x0000-0x006f window]
>   pci_bus 0000:00: root bus resource [io  0x0078-0x0cf7 window]
>   pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
>   pci_bus 0000:00: root bus resource [mem 0x7c000001-0x7fffffff window]
>   pci_bus 0000:00: root bus resource [mem 0x7b800001-0x7bffffff window]
>   pci_bus 0000:00: root bus resource [mem 0x80000000-0xcfffffff window]
>   pci_bus 0000:00: root bus resource [mem 0xe0000000-0xefffffff window]
>   pci_bus 0000:00: root bus resource [bus 00-ff]
> 
> The P2SB 16MB BAR is located at 0xd0000000-0xd0ffffff memory window.
> 
> The generic solution
> ====================
> The generic solution for all cases when we need to access to the information
> behind P2SB device is a library code where users ask for necessary resources
> by demand and hence those users take care of not being run on the systems
> where this access is not required.
> 
> The library provides the p2sb_bar() API to retrieve the MMIO of the BAR0 of
> the device from P2SB device slot.
> 
> P2SB unconditional unhiding awareness
> =====================================
> Technically it's possible to unhide the P2SB device and devices on
> the same PCI slot and access them at any time as needed. But there are
> several potential issues with that:
> 
>  - the systems were never tested against such configuration and hence
>    nobody knows what kind of bugs it may bring, especially when we talk
>    about SPI NOR case which contains Intel FirmWare Image (IFWI) code
>    (including BIOS) and already known to be problematic in the past for
>    end users
> 
>  - the PCI by its nature is a hotpluggable bus and in case somebody
>    attaches a driver to the functions of a P2SB slot device(s) the
>    end user experience and system behaviour can be unpredictable
> 
>  - the kernel code would need some ugly hacks (or code looking as an
>    ugly hack) under arch/x86/pci in order to enable these devices on
>    only selected platforms (which may include CPU ID table followed by
>    a potentially growing number of DMI strings
> 
> The future improvements
> =======================
> The future improvements with this code may go in order to gain some kind
> of cache, if it's possible at all, to prevent unhiding and hiding many
> times to take static information that may be saved once per boot.
> 
> Links
> =====
> [1]: https://lab.whitequark.org/notes/2017-11-08/accessing-intel-ich-pch-gpios/
> [2]: https://cdrdv2.intel.com/v1/dl/getContent/332690?wapkw=332690
> [3]: https://cdrdv2.intel.com/v1/dl/getContent/332691?wapkw=332691
> [4]: https://medium.com/@jacksonchen_43335/bios-gpio-p2sb-70e9b829b403
> 
> Signed-off-by: Jonathan Yong <jonathan.yong@intel.com>
> Co-developed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
> Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
> Acked-by: Hans de Goede <hdegoede@redhat.com>
> Acked-by: Linus Walleij <linus.walleij@linaro.org>
> ---
>  drivers/platform/x86/intel/Kconfig     |  12 +
>  drivers/platform/x86/intel/Makefile    |   1 +
>  drivers/platform/x86/intel/p2sb.c      | 299 +++++++++++++++++++++++++
>  include/linux/platform_data/x86/p2sb.h |  27 +++
>  4 files changed, 339 insertions(+)
>  create mode 100644 drivers/platform/x86/intel/p2sb.c
>  create mode 100644 include/linux/platform_data/x86/p2sb.h
> 
> diff --git a/drivers/platform/x86/intel/Kconfig b/drivers/platform/x86/intel/Kconfig
> index 8e65086bb6c8..55b499fb8249 100644
> --- a/drivers/platform/x86/intel/Kconfig
> +++ b/drivers/platform/x86/intel/Kconfig
> @@ -68,6 +68,18 @@ config INTEL_OAKTRAIL
>  	  enable/disable the Camera, WiFi, BT etc. devices. If in doubt, say Y
>  	  here; it will only load on supported platforms.
>  
> +config P2SB
> +	bool "Primary to Sideband (P2SB) bridge access support"
> +	depends on PCI
> +	help
> +	  The Primary to Sideband (P2SB) bridge is an interface to some
> +	  PCI devices connected through it. In particular, SPI NOR controller
> +	  in Intel Apollo Lake SoC is one of such devices.
> +
> +	  The main purpose of this library is to unhide P2SB device in case
> +	  firmware kept it hidden on some platforms in order to access devices
> +	  behind it.
> +
>  config INTEL_BXTWC_PMIC_TMU
>  	tristate "Intel Broxton Whiskey Cove TMU Driver"
>  	depends on INTEL_SOC_PMIC_BXTWC
> diff --git a/drivers/platform/x86/intel/Makefile b/drivers/platform/x86/intel/Makefile
> index 35f2066578b2..6ce944285730 100644
> --- a/drivers/platform/x86/intel/Makefile
> +++ b/drivers/platform/x86/intel/Makefile
> @@ -26,6 +26,7 @@ intel_int0002_vgpio-y			:= int0002_vgpio.o
>  obj-$(CONFIG_INTEL_INT0002_VGPIO)	+= intel_int0002_vgpio.o
>  intel_oaktrail-y			:= oaktrail.o
>  obj-$(CONFIG_INTEL_OAKTRAIL)		+= intel_oaktrail.o
> +obj-$(CONFIG_P2SB)			+= p2sb.o
>  intel_vsec-y				:= vsec.o
>  obj-$(CONFIG_INTEL_VSEC)		+= intel_vsec.o
>  
> diff --git a/drivers/platform/x86/intel/p2sb.c b/drivers/platform/x86/intel/p2sb.c
> new file mode 100644
> index 000000000000..8c12882d0bde
> --- /dev/null
> +++ b/drivers/platform/x86/intel/p2sb.c
> @@ -0,0 +1,299 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Primary to Sideband (P2SB) bridge access support
> + *
> + * Copyright (c) 2017, 2021-2022 Intel Corporation.
> + *
> + * Authors: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
> + *	    Jonathan Yong <jonathan.yong@intel.com>
> + */
> +
> +#include <linux/bitops.h>
> +#include <linux/export.h>
> +#include <linux/pci.h>
> +#include <linux/platform_data/x86/p2sb.h>
> +#include <linux/printk.h>
> +
> +#include <asm/cpu_device_id.h>
> +#include <asm/intel-family.h>
> +
> +#define p2sb_printk(level, bus, devfn, fmt, arg...) \
> +	printk(level "pci %04x:%02x:%02x.%d: " fmt, \
> +	       pci_domain_nr(bus), (bus)->number, PCI_SLOT(devfn), PCI_FUNC(devfn), ##arg)
> +
> +#define p2sb_err(bus, devfn, fmt, arg...)	p2sb_printk(KERN_ERR, (bus), devfn, fmt, ##arg)
> +#define p2sb_info(bus, devfn, fmt, arg...)	p2sb_printk(KERN_INFO, (bus), devfn, fmt, ##arg)
> +
> +#define P2SBC			0xe0
> +#define P2SBC_HIDE		BIT(8)
> +
> +static const struct x86_cpu_id p2sb_cpu_ids[] = {
> +	X86_MATCH_INTEL_FAM6_MODEL(ATOM_GOLDMONT,	PCI_DEVFN(13, 0)),
> +	{}
> +};
> +
> +static int p2sb_get_devfn(unsigned int *devfn)
> +{
> +	const struct x86_cpu_id *id;
> +
> +	id = x86_match_cpu(p2sb_cpu_ids);
> +	if (!id)
> +		return -ENODEV;
> +
> +	*devfn = (unsigned int)id->driver_data;
> +	return 0;
> +}
> +
> +/*
> + * Here is practically a copy'n'paste of __pci_read_base() and Co,
> + * modified to work with PCI bus and devfn instead of PCI device.
> + *
> + * Note, the PCI core doesn't want to have that being refactored
> + * and reused.
> + */
> +static u64 pci_size(u64 base, u64 maxbase, u64 mask)
> +{
> +	u64 size = mask & maxbase;	/* Find the significant bits */
> +	if (!size)
> +		return 0;
> +
> +	/*
> +	 * Get the lowest of them to find the decode size, and from that
> +	 * the extent.
> +	 */
> +	size = size & ~(size-1);
> +
> +	/*
> +	 * base == maxbase can be valid only if the BAR has already been
> +	 * programmed with all 1s.
> +	 */
> +	if (base == maxbase && ((base | (size - 1)) & mask) != mask)
> +		return 0;
> +
> +	return size;
> +}
> +
> +static inline unsigned long decode_bar(u32 bar)
> +{
> +	u32 mem_type;
> +	unsigned long flags;
> +
> +	if ((bar & PCI_BASE_ADDRESS_SPACE) == PCI_BASE_ADDRESS_SPACE_IO) {
> +		flags = bar & ~PCI_BASE_ADDRESS_IO_MASK;
> +		flags |= IORESOURCE_IO;
> +		return flags;
> +	}
> +
> +	flags = bar & ~PCI_BASE_ADDRESS_MEM_MASK;
> +	flags |= IORESOURCE_MEM;
> +	if (flags & PCI_BASE_ADDRESS_MEM_PREFETCH)
> +		flags |= IORESOURCE_PREFETCH;
> +
> +	mem_type = bar & PCI_BASE_ADDRESS_MEM_TYPE_MASK;
> +	switch (mem_type) {
> +	case PCI_BASE_ADDRESS_MEM_TYPE_32:
> +		break;
> +	case PCI_BASE_ADDRESS_MEM_TYPE_1M:
> +		/* 1M mem BAR treated as 32-bit BAR */
> +		break;
> +	case PCI_BASE_ADDRESS_MEM_TYPE_64:
> +		flags |= IORESOURCE_MEM_64;
> +		break;
> +	default:
> +		/* mem unknown type treated as 32-bit BAR */
> +		break;
> +	}
> +	return flags;
> +}
> +
> +/**
> + * __pci_bus_read_base - Read a PCI BAR
> + * @bus: the PCI bus
> + * @devfn: the PCI device and function
> + * @res: resource buffer to be filled in
> + * @pos: BAR position in the config space
> + *
> + * Returns 1 if the BAR is 64-bit, or 0 if 32-bit.
> + * In case of error resulting @res->flags is 0.
> + */
> +static int __pci_bus_read_base(struct pci_bus *bus, unsigned int devfn,
> +			       struct resource *res, unsigned int pos)
> +{

Andy, as discussed already, this is quite a big function to mostly duplicate
from the PCI-core. AFAIK the last status of the discussion surrounding this
was trying to have the P2SB code here create a fake pci_dev struct and use
that with the existing function instead.

Have you given this approach a try ?

Note if that does not work out I'm fine with taking this patch as is.

Regards,

Hans


> +	u32 l = 0, sz = 0, mask = ~0;
> +	u64 l64, sz64, mask64;
> +	struct pci_bus_region region, inverted_region;
> +
> +	pci_bus_read_config_dword(bus, devfn, pos, &l);
> +	pci_bus_write_config_dword(bus, devfn, pos, l | mask);
> +	pci_bus_read_config_dword(bus, devfn, pos, &sz);
> +	pci_bus_write_config_dword(bus, devfn, pos, l);
> +
> +	/*
> +	 * All bits set in sz means the device isn't working properly.
> +	 * If the BAR isn't implemented, all bits must be 0.  If it's a
> +	 * memory BAR or a ROM, bit 0 must be clear; if it's an io BAR, bit
> +	 * 1 must be clear.
> +	 */
> +	if (PCI_POSSIBLE_ERROR(sz))
> +		sz = 0;
> +
> +	/*
> +	 * I don't know how l can have all bits set.  Copied from old code.
> +	 * Maybe it fixes a bug on some ancient platform.
> +	 */
> +	if (PCI_POSSIBLE_ERROR(l))
> +		l = 0;
> +
> +	res->flags = decode_bar(l);
> +	res->flags |= IORESOURCE_SIZEALIGN;
> +	if (res->flags & IORESOURCE_IO) {
> +		l64 = l & PCI_BASE_ADDRESS_IO_MASK;
> +		sz64 = sz & PCI_BASE_ADDRESS_IO_MASK;
> +		mask64 = PCI_BASE_ADDRESS_IO_MASK & (u32)IO_SPACE_LIMIT;
> +	} else {
> +		l64 = l & PCI_BASE_ADDRESS_MEM_MASK;
> +		sz64 = sz & PCI_BASE_ADDRESS_MEM_MASK;
> +		mask64 = (u32)PCI_BASE_ADDRESS_MEM_MASK;
> +	}
> +
> +	if (res->flags & IORESOURCE_MEM_64) {
> +		pci_bus_read_config_dword(bus, devfn, pos + 4, &l);
> +		pci_bus_write_config_dword(bus, devfn, pos + 4, ~0);
> +		pci_bus_read_config_dword(bus, devfn, pos + 4, &sz);
> +		pci_bus_write_config_dword(bus, devfn, pos + 4, l);
> +
> +		l64 |= ((u64)l << 32);
> +		sz64 |= ((u64)sz << 32);
> +		mask64 |= ((u64)~0 << 32);
> +	}
> +
> +	if (!sz64)
> +		goto fail;
> +
> +	sz64 = pci_size(l64, sz64, mask64);
> +	if (!sz64) {
> +		p2sb_info(bus, devfn, FW_BUG "reg 0x%x: invalid BAR (can't size)\n", pos);
> +		goto fail;
> +	}
> +
> +	if (res->flags & IORESOURCE_MEM_64) {
> +		if ((sizeof(pci_bus_addr_t) < 8 || sizeof(resource_size_t) < 8)
> +		    && sz64 > 0x100000000ULL) {
> +			res->flags |= IORESOURCE_UNSET | IORESOURCE_DISABLED;
> +			res->start = 0;
> +			res->end = 0;
> +			p2sb_err(bus, devfn,
> +				 "reg 0x%x: can't handle BAR larger than 4GB (size %#010llx)\n",
> +				 pos, (unsigned long long)sz64);
> +			goto out;
> +		}
> +
> +		if ((sizeof(pci_bus_addr_t) < 8) && l) {
> +			/* Above 32-bit boundary; try to reallocate */
> +			res->flags |= IORESOURCE_UNSET;
> +			res->start = 0;
> +			res->end = sz64 - 1;
> +			p2sb_info(bus, devfn,
> +				  "reg 0x%x: can't handle BAR above 4GB (bus address %#010llx)\n",
> +				  pos, (unsigned long long)l64);
> +			goto out;
> +		}
> +	}
> +
> +	region.start = l64;
> +	region.end = l64 + sz64 - 1;
> +
> +	pcibios_bus_to_resource(bus, res, &region);
> +	pcibios_resource_to_bus(bus, &inverted_region, res);
> +
> +	/*
> +	 * If "A" is a BAR value (a bus address), "bus_to_resource(A)" is
> +	 * the corresponding resource address (the physical address used by
> +	 * the CPU.  Converting that resource address back to a bus address
> +	 * should yield the original BAR value:
> +	 *
> +	 *     resource_to_bus(bus_to_resource(A)) == A
> +	 *
> +	 * If it doesn't, CPU accesses to "bus_to_resource(A)" will not
> +	 * be claimed by the device.
> +	 */
> +	if (inverted_region.start != region.start) {
> +		res->flags |= IORESOURCE_UNSET;
> +		res->start = 0;
> +		res->end = region.end - region.start;
> +		p2sb_info(bus, devfn, "reg 0x%x: initial BAR value %#010llx invalid\n",
> +			  pos, (unsigned long long)region.start);
> +	}
> +
> +	goto out;
> +
> +fail:
> +	res->flags = 0;
> +out:
> +	if (res->flags)
> +		p2sb_info(bus, devfn, "reg 0x%x: %pR\n", pos, res);
> +
> +	return (res->flags & IORESOURCE_MEM_64) ? 1 : 0;
> +}
> +
> +/**
> + * p2sb_bar - Get Primary to Sideband (P2SB) bridge device BAR
> + * @bus: PCI bus to communicate with
> + * @devfn: PCI slot and function to communicate with
> + * @mem: memory resource to be filled in
> + *
> + * The BIOS prevents the P2SB device from being enumerated by the PCI
> + * subsystem, so we need to unhide and hide it back to lookup the BAR.
> + *
> + * if @bus is NULL, the bus 0 in domain 0 will be in use.
> + * If @devfn is 0, it will be replaced by devfn of the P2SB device.
> + *
> + * Caller must provide a valid pointer to @mem.
> + *
> + * Locking is handled by pci_rescan_remove_lock mutex.
> + *
> + * Return:
> + * 0 on success or appropriate errno value on error.
> + */
> +int p2sb_bar(struct pci_bus *bus, unsigned int devfn, struct resource *mem)
> +{
> +	unsigned int devfn_p2sb;
> +	u32 value = P2SBC_HIDE;
> +	int ret;
> +
> +	/* Get devfn for P2SB device itself */
> +	ret = p2sb_get_devfn(&devfn_p2sb);
> +	if (ret)
> +		return ret;
> +
> +	/* if @pdev is NULL, use bus 0 in domain 0 */
> +	bus = bus ?: pci_find_bus(0, 0);
> +
> +	/* If @devfn is 0, replace it with devfn of P2SB device itself */
> +	devfn = devfn ?: devfn_p2sb;
> +
> +	pci_lock_rescan_remove();
> +
> +	/* Unhide the P2SB device, if needed */
> +	pci_bus_read_config_dword(bus, devfn_p2sb, P2SBC, &value);
> +	if ((value & P2SBC_HIDE) == P2SBC_HIDE)
> +		pci_bus_write_config_dword(bus, devfn_p2sb, P2SBC, 0);
> +
> +	/* Read the first BAR of the device in question */
> +	__pci_bus_read_base(bus, devfn, mem, PCI_BASE_ADDRESS_0);
> +
> +	/* Hide the P2SB device, if it was hidden */
> +	if (value & P2SBC_HIDE)
> +		pci_bus_write_config_dword(bus, devfn_p2sb, P2SBC, P2SBC_HIDE);
> +
> +	pci_unlock_rescan_remove();
> +
> +	if (mem->flags == 0) {
> +		p2sb_err(bus, devfn, "Can't read BAR0: %pR\n", mem);
> +		return -ENODEV;
> +	}
> +
> +	p2sb_info(bus, devfn, "BAR: %pR\n", mem);
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(p2sb_bar);
> diff --git a/include/linux/platform_data/x86/p2sb.h b/include/linux/platform_data/x86/p2sb.h
> new file mode 100644
> index 000000000000..2f71de65aee4
> --- /dev/null
> +++ b/include/linux/platform_data/x86/p2sb.h
> @@ -0,0 +1,27 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Primary to Sideband (P2SB) bridge access support
> + */
> +
> +#ifndef _PLATFORM_DATA_X86_P2SB_H
> +#define _PLATFORM_DATA_X86_P2SB_H
> +
> +#include <linux/errno.h>
> +
> +struct pci_bus;
> +struct resource;
> +
> +#if IS_BUILTIN(CONFIG_P2SB)
> +
> +int p2sb_bar(struct pci_bus *bus, unsigned int devfn, struct resource *mem);
> +
> +#else /* CONFIG_P2SB */
> +
> +static inline int p2sb_bar(struct pci_bus *bus, unsigned int devfn, struct resource *mem)
> +{
> +	return -ENODEV;
> +}
> +
> +#endif /* CONFIG_P2SB is not set */
> +
> +#endif /* _PLATFORM_DATA_X86_P2SB_H */

On Mon, Jan 31, 2022 at 05:13:39PM +0200, Andy Shevchenko wrote:
> Background information
> ======================

The wealth of information in the commit message obscures what the
actual problem is, which is actually quite simple:  SoC features
such as GPIO are accessed via a reserved MMIO area, we don't know
its address but can obtain it from the BAR of the P2SB device,
that device is normally hidden so we have to temporarily unhide it.


> On top of that in some cases P2SB is represented by function 0 on PCI
> slot (in terms of B:D.F) and according to the PCI specification any
> other function can't be seen until function 0 is present and visible.

I find that paragraph confusing:  Do multi-function P2SB devices exist?
What are the other functions?  Are they visible but merely not enumerated
because function 0 is not visible?


> P2SB unconditional unhiding awareness
> =====================================
> Technically it's possible to unhide the P2SB device and devices on
> the same PCI slot and access them at any time as needed. But there are
> several potential issues with that:
> 
>  - the systems were never tested against such configuration and hence
>    nobody knows what kind of bugs it may bring, especially when we talk
>    about SPI NOR case which contains Intel FirmWare Image (IFWI) code
>    (including BIOS) and already known to be problematic in the past for
>    end users
> 
>  - the PCI by its nature is a hotpluggable bus and in case somebody
>    attaches a driver to the functions of a P2SB slot device(s) the
>    end user experience and system behaviour can be unpredictable
> 
>  - the kernel code would need some ugly hacks (or code looking as an
>    ugly hack) under arch/x86/pci in order to enable these devices on
>    only selected platforms (which may include CPU ID table followed by
>    a potentially growing number of DMI strings

Honestly I would have taken the step to always expose the device,
identify breakages and then fix those.

We had a similar issue with HD audio controllers on Nvidia GPUs
which were only visible when an HDMI cable was plugged in.
We always expose them since b516ea586d71 and I recall we merely
had a few cases that an audio device was exposed in cases when
the card had no HDMI connectors at all.  So there was a useless
HD audio card visible to the user but no real harm.


> +	pci_lock_rescan_remove();
> +
> +	/* Unhide the P2SB device, if needed */
> +	pci_bus_read_config_dword(bus, devfn_p2sb, P2SBC, &value);
> +	if ((value & P2SBC_HIDE) == P2SBC_HIDE)
> +		pci_bus_write_config_dword(bus, devfn_p2sb, P2SBC, 0);
> +
> +	/* Read the first BAR of the device in question */
> +	__pci_bus_read_base(bus, devfn, mem, PCI_BASE_ADDRESS_0);
> +
> +	/* Hide the P2SB device, if it was hidden */
> +	if (value & P2SBC_HIDE)
> +		pci_bus_write_config_dword(bus, devfn_p2sb, P2SBC, P2SBC_HIDE);
> +
> +	pci_unlock_rescan_remove();

Please add a code comment why you're calling pci_lock_rescan_remove(),
such as:

	/*
	 * Prevent concurrent PCI bus scan from seeing the P2SB device
	 * while it is temporarily exposed.
	 */

Otherwise it looks like you're abusing that lock to prevent multiple
simultaneous RMW operations of the P2SBC_HIDE bit.


I think the first if-clause above can be simplified to

	if (value & P2SBC_HIDE)

I don't understand why one of the two if-clauses adds "== P2SBC_HIDE".


Do you really need all the complicated logic in __pci_bus_read_base()?
For comparison, simatic_ipc_get_membase0() in simatic-ipc.c merely does:

	pci_bus_read_config_dword(bus, p2sb, PCI_BASE_ADDRESS_0, &bar0);

If that's sufficient for simatic-ipc.c, why is the more complicated code
necessary in p2sb.c?


I'm wondering, since you're only retrieving the base address (and thus
temporarily expose the P2SB device) when it's actually needed by a driver,
would there be any harm in keeping the P2SB device exposed indefinitely
from the moment a driver first requests the base address?  I.e., unhide it
but don't hide it again.  That would allow you to call pci_scan_slot() and
pci_bus_add_devices(), thus instantiating a proper pci_dev which you can
access without the __pci_bus_read_base() kludge.


> +	/*
> +	 * I don't know how l can have all bits set.  Copied from old code.
> +	 * Maybe it fixes a bug on some ancient platform.
> +	 */
> +	if (PCI_POSSIBLE_ERROR(l))
> +		l = 0;

l can have all bits set if the device was hot-removed.  That can't happen
with a built-in device such as P2SB.

Thanks,

Lukas

On Thu, May 5, 2022 at 4:55 PM Lukas Wunner <lukas@wunner.de> wrote:
> On Mon, Jan 31, 2022 at 05:13:39PM +0200, Andy Shevchenko wrote:

Thanks for your review, Lukas! My answers below.

...

> > Background information
> > ======================
>
> The wealth of information in the commit message obscures what the
> actual problem is, which is actually quite simple:  SoC features
> such as GPIO are accessed via a reserved MMIO area, we don't know
> its address but can obtain it from the BAR of the P2SB device,
> that device is normally hidden so we have to temporarily unhide it.

Right, but this long commit message was a result of the previous
discussions with Bjorn. If we're ever going to handle something like
this in the PCI core, perhaps he won't be happy if I remove it. Maybe
we can simply state what you wrote as a problem statement and move
this chapter at the end?

> > On top of that in some cases P2SB is represented by function 0 on PCI
> > slot (in terms of B:D.F) and according to the PCI specification any
> > other function can't be seen until function 0 is present and visible.
>
> I find that paragraph confusing:  Do multi-function P2SB devices exist?
> What are the other functions?  Are they visible but merely not enumerated
> because function 0 is not visible?

The case I see is when we want to read the BAR from another slot of a
PCI device, 0 function of which is P2SB. Since P2SB is hidden, the
other device is hidden as well. Any idea how to reformulate this? And
yes, we have this in the existing SoCs.

...

> > P2SB unconditional unhiding awareness
> > =====================================
> > Technically it's possible to unhide the P2SB device and devices on
> > the same PCI slot and access them at any time as needed. But there are
> > several potential issues with that:
> >
> >  - the systems were never tested against such configuration and hence
> >    nobody knows what kind of bugs it may bring, especially when we talk
> >    about SPI NOR case which contains Intel FirmWare Image (IFWI) code
> >    (including BIOS) and already known to be problematic in the past for
> >    end users
> >
> >  - the PCI by its nature is a hotpluggable bus and in case somebody
> >    attaches a driver to the functions of a P2SB slot device(s) the
> >    end user experience and system behaviour can be unpredictable
> >
> >  - the kernel code would need some ugly hacks (or code looking as an
> >    ugly hack) under arch/x86/pci in order to enable these devices on
> >    only selected platforms (which may include CPU ID table followed by
> >    a potentially growing number of DMI strings
>
> Honestly I would have taken the step to always expose the device,
> identify breakages and then fix those.

Taking into account the history of the different issues on Intel Atom
SoCs I would like to be on the safer side, that's why we don't want to
expose it and keep the status quo for now. IIRC Hans and Mika are on
the same page with me on this.

> We had a similar issue with HD audio controllers on Nvidia GPUs
> which were only visible when an HDMI cable was plugged in.
> We always expose them since b516ea586d71 and I recall we merely
> had a few cases that an audio device was exposed in cases when
> the card had no HDMI connectors at all.  So there was a useless
> HD audio card visible to the user but no real harm.

...

> > +     pci_unlock_rescan_remove();
>
> Please add a code comment why you're calling pci_lock_rescan_remove(),

Sure!

> such as:
>
>         /*
>          * Prevent concurrent PCI bus scan from seeing the P2SB device
>          * while it is temporarily exposed.
>          */
>
> Otherwise it looks like you're abusing that lock to prevent multiple
> simultaneous RMW operations of the P2SBC_HIDE bit.
>
>
> I think the first if-clause above can be simplified to
>
>         if (value & P2SBC_HIDE)
>
> I don't understand why one of the two if-clauses adds "== P2SBC_HIDE".

Agree.

...

> Do you really need all the complicated logic in __pci_bus_read_base()?
> For comparison, simatic_ipc_get_membase0() in simatic-ipc.c merely does:
>
>         pci_bus_read_config_dword(bus, p2sb, PCI_BASE_ADDRESS_0, &bar0);
>
> If that's sufficient for simatic-ipc.c, why is the more complicated code
> necessary in p2sb.c?

Since it's a PCI device I want to follow what PCI core does with it.
As I explained somewhere that the current code (actually it's a
simplified version of what is done in PCI core) follows what spec
requires. I would like to be in alignment with the spec, while it
still may work with less code. Besides that, it's theoretically
possible that the base address may be 64-bit in new SoCs, I won't
rewrite code again just because we abused the spec.

> I'm wondering, since you're only retrieving the base address (and thus
> temporarily expose the P2SB device) when it's actually needed by a driver,
> would there be any harm in keeping the P2SB device exposed indefinitely
> from the moment a driver first requests the base address?  I.e., unhide it
> but don't hide it again.  That would allow you to call pci_scan_slot() and
> pci_bus_add_devices(), thus instantiating a proper pci_dev which you can
> access without the __pci_bus_read_base() kludge.

Same as above about permanent unhide awareness.
Don't forget that on the same SoCs but with different BIOSes the GPIO,
for example, may or may not be exposed via ACPI, which means that we
need to take precautions with the possible conflicts in device
enumeration (we obviously prefer the ACPI enumeration over P2SB). If
P2SB is always exposed it's theoretically possible and maybe even
practically that base address is changed (by unbinding/binding cycle),
while GPIO is enumerated via P2SB and hence we may end up with the
completely wrong addresses in GPIO.

...

> > +     /*
> > +      * I don't know how l can have all bits set.  Copied from old code.
> > +      * Maybe it fixes a bug on some ancient platform.
> > +      */
> > +     if (PCI_POSSIBLE_ERROR(l))
> > +             l = 0;
>
> l can have all bits set if the device was hot-removed.  That can't happen
> with a built-in device such as P2SB.

Can be dropped, indeed. But that chicken bit emulates that :-) Anyway,
we unhide the device before looking into it, so we shouldn't have the
surprise "removals".

On Thu, May 05, 2022 at 07:54:49PM +0200, Andy Shevchenko wrote:
> On Thu, May 5, 2022 at 4:55 PM Lukas Wunner <lukas@wunner.de> wrote:
> > On Mon, Jan 31, 2022 at 05:13:39PM +0200, Andy Shevchenko wrote:
> > > Background information
> > > ======================
> >
> > The wealth of information in the commit message obscures what the
> > actual problem is, which is actually quite simple:  SoC features
> > such as GPIO are accessed via a reserved MMIO area, we don't know
> > its address but can obtain it from the BAR of the P2SB device,
> > that device is normally hidden so we have to temporarily unhide it.
> 
> Right, but this long commit message was a result of the previous
> discussions with Bjorn. If we're ever going to handle something like
> this in the PCI core, perhaps he won't be happy if I remove it. Maybe
> we can simply state what you wrote as a problem statement and move
> this chapter at the end?

Yes, feel free to copy-paste the synopsis from my e-mail above
and rephrase as you see fit.


> > > On top of that in some cases P2SB is represented by function 0 on PCI
> > > slot (in terms of B:D.F) and according to the PCI specification any
> > > other function can't be seen until function 0 is present and visible.
> >
> > I find that paragraph confusing:  Do multi-function P2SB devices exist?
> > What are the other functions?  Are they visible but merely not enumerated
> > because function 0 is not visible?
> 
> The case I see is when we want to read the BAR from another slot of a
> PCI device, 0 function of which is P2SB. Since P2SB is hidden, the
> other device is hidden as well. Any idea how to reformulate this? And
> yes, we have this in the existing SoCs.

The spec you linked to in the commit message (for the 100 series chipset)
says that P2SB is located at Device 31 Function 1.

In those chipsets where P2SB is function 0, what kind of devices are
at functions 1 and higher?


> > Do you really need all the complicated logic in __pci_bus_read_base()?
> > For comparison, simatic_ipc_get_membase0() in simatic-ipc.c merely does:
> >
> >         pci_bus_read_config_dword(bus, p2sb, PCI_BASE_ADDRESS_0, &bar0);
> >
> > If that's sufficient for simatic-ipc.c, why is the more complicated code
> > necessary in p2sb.c?
> 
> Since it's a PCI device I want to follow what PCI core does with it.
> As I explained somewhere that the current code (actually it's a
> simplified version of what is done in PCI core) follows what spec
> requires. I would like to be in alignment with the spec, while it
> still may work with less code. Besides that, it's theoretically
> possible that the base address may be 64-bit in new SoCs, I won't
> rewrite code again just because we abused the spec.

So as an alternative to copy-pasting __pci_bus_read_base(),
you could just call pci_scan_single_device().  This will create
a proper pci_dev that you can work with.  Note that no driver will
be bound to the device because of:

  pci_scan_single_device()
    pci_device_add()
      dev->match_driver = false

After you've read the BAR, get rid of the pci_dev with pci_destroy_dev().


> > > +     /*
> > > +      * I don't know how l can have all bits set.  Copied from old code.
> > > +      * Maybe it fixes a bug on some ancient platform.
> > > +      */
> > > +     if (PCI_POSSIBLE_ERROR(l))
> > > +             l = 0;
> >
> > l can have all bits set if the device was hot-removed.  That can't happen
> > with a built-in device such as P2SB.
> 
> Can be dropped, indeed. But that chicken bit emulates that :-) Anyway,
> we unhide the device before looking into it, so we shouldn't have the
> surprise "removals".

pci_lock_rescan_remove() prevents concurrent unhiding as well as
removal via sysfs.

Thanks,

Lukas

On Sun, May 8, 2022 at 9:13 AM Lukas Wunner <lukas@wunner.de> wrote:
> On Thu, May 05, 2022 at 07:54:49PM +0200, Andy Shevchenko wrote:
> > On Thu, May 5, 2022 at 4:55 PM Lukas Wunner <lukas@wunner.de> wrote:
> > > On Mon, Jan 31, 2022 at 05:13:39PM +0200, Andy Shevchenko wrote:

...

> > > > Background information
> > > > ======================
> > >
> > > The wealth of information in the commit message obscures what the
> > > actual problem is, which is actually quite simple:  SoC features
> > > such as GPIO are accessed via a reserved MMIO area, we don't know
> > > its address but can obtain it from the BAR of the P2SB device,
> > > that device is normally hidden so we have to temporarily unhide it.
> >
> > Right, but this long commit message was a result of the previous
> > discussions with Bjorn. If we're ever going to handle something like
> > this in the PCI core, perhaps he won't be happy if I remove it. Maybe
> > we can simply state what you wrote as a problem statement and move
> > this chapter at the end?
>
> Yes, feel free to copy-paste the synopsis from my e-mail above
> and rephrase as you see fit.

Will do.

...

> > > > On top of that in some cases P2SB is represented by function 0 on PCI
> > > > slot (in terms of B:D.F) and according to the PCI specification any
> > > > other function can't be seen until function 0 is present and visible.
> > >
> > > I find that paragraph confusing:  Do multi-function P2SB devices exist?
> > > What are the other functions?  Are they visible but merely not enumerated
> > > because function 0 is not visible?
> >
> > The case I see is when we want to read the BAR from another slot of a
> > PCI device, 0 function of which is P2SB. Since P2SB is hidden, the
> > other device is hidden as well. Any idea how to reformulate this? And
> > yes, we have this in the existing SoCs.
>
> The spec you linked to in the commit message (for the 100 series chipset)
> says that P2SB is located at Device 31 Function 1.
>
> In those chipsets where P2SB is function 0, what kind of devices are
> at functions 1 and higher?

In the Intel Broxton and Apollo Lake cases the P2SB is the function 0
and we want to have a BAR of SPI NOR, which is function 2.

...

> > > Do you really need all the complicated logic in __pci_bus_read_base()?
> > > For comparison, simatic_ipc_get_membase0() in simatic-ipc.c merely does:
> > >
> > >         pci_bus_read_config_dword(bus, p2sb, PCI_BASE_ADDRESS_0, &bar0);
> > >
> > > If that's sufficient for simatic-ipc.c, why is the more complicated code
> > > necessary in p2sb.c?
> >
> > Since it's a PCI device I want to follow what PCI core does with it.
> > As I explained somewhere that the current code (actually it's a
> > simplified version of what is done in PCI core) follows what spec
> > requires. I would like to be in alignment with the spec, while it
> > still may work with less code. Besides that, it's theoretically
> > possible that the base address may be 64-bit in new SoCs, I won't
> > rewrite code again just because we abused the spec.
>
> So as an alternative to copy-pasting __pci_bus_read_base(),
> you could just call pci_scan_single_device().  This will create
> a proper pci_dev that you can work with.  Note that no driver will
> be bound to the device because of:
>
>   pci_scan_single_device()
>     pci_device_add()
>       dev->match_driver = false
>
> After you've read the BAR, get rid of the pci_dev with pci_destroy_dev().

This is pretty nice, if it flies! I definitely try this ASAP (during
working hours).
Thanks for the hint.

...

> > > > +     /*
> > > > +      * I don't know how l can have all bits set.  Copied from old code.
> > > > +      * Maybe it fixes a bug on some ancient platform.
> > > > +      */
> > > > +     if (PCI_POSSIBLE_ERROR(l))
> > > > +             l = 0;
> > >
> > > l can have all bits set if the device was hot-removed.  That can't happen
> > > with a built-in device such as P2SB.
> >
> > Can be dropped, indeed. But that chicken bit emulates that :-) Anyway,
> > we unhide the device before looking into it, so we shouldn't have the
> > surprise "removals".
>
> pci_lock_rescan_remove() prevents concurrent unhiding as well as
> removal via sysfs.

Yep, that's good. In any case this piece of code will be gone if your
above suggestion works, have I got it right?

On Sun, May 08, 2022 at 12:05:53PM +0200, Andy Shevchenko wrote:
> On Sun, May 8, 2022 at 9:13 AM Lukas Wunner <lukas@wunner.de> wrote:
> > pci_lock_rescan_remove() prevents concurrent unhiding as well as
> > removal via sysfs.
> 
> Yep, that's good. In any case this piece of code will be gone if your
> above suggestion works, have I got it right?

Yes.  You just need to make sure that you call pci_scan_single_device()
*after* unhiding the P2SB device so that this check succeeds:

  pci_scan_single_device()
    pci_scan_device()
      pci_bus_read_dev_vendor_id()

Thanks,

Lukas