[0/4,V4] ACPI: Support generic initiator proximity domains

Message ID 20190821145242.2330-1-Jonathan.Cameron@huawei.com
Headers show
Series
  • ACPI: Support generic initiator proximity domains
Related show

Message

Jonathan Cameron Aug. 21, 2019, 2:52 p.m.
This patch set has been sitting around for a long time without significant
review.  I would appreciate it very much if anyone has time to take a look.

One outstanding question to highlight in this series is whether
we should assume all ACPI supporting architectures support Generic
Initiator domains, or whether to introduce an
ARCH_HAS_GENERIC_INITIATOR_DOMAINS entry in Kconfig.

Change since V3.
* Rebase.

Changes since RFC V2.
* RFC dropped as now we have x86 support, so the lack of guards in in the
  ACPI code etc should now be fine.
* Added x86 support.  Note this has only been tested on QEMU as I don't have
  a convenient x86 NUMA machine to play with.  Note that this fitted together
  rather differently form arm64 so I'm particularly interested in feedback
  on the two solutions.

Since RFC V1.
* Fix incorrect interpretation of the ACPI entry noted by Keith Busch
* Use the acpica headers definitions that are now in mmotm.

It's worth noting that, to safely put a given device in a GI node, may
require changes to the existing drivers as it's not unusual to assume
you have local memory or processor core. There may be further constraints
not yet covered by this patch.

Original cover letter...

ACPI 6.3 introduced a new entity that can be part of a NUMA proximity domain.
It may share such a domain with the existing options (memory, CPU etc) but it
may also exist on it's own.

The intent is to allow the description of the NUMA properties (particularly
via HMAT) of accelerators and other initiators of memory activity that are not
the host processor running the operating system.

This patch set introduces 'just enough' to make them work for arm64 and x86.
It should be trivial to support other architectures, I just don't suitable
NUMA systems readily available to test.

There are a few quirks that need to be considered.

1. Fall back nodes
******************

As pre ACPI 6.3 supporting operating systems do not have Generic Initiator
Proximity Domains it is possible to specify, via _PXM in DSDT that another
device is part of such a GI only node.  This currently blows up spectacularly.

Whilst we can obviously 'now' protect against such a situation (see the related
thread on PCI _PXM support and the  threadripper board identified there as
also falling into the  problem of using non existent nodes
https://patchwork.kernel.org/patch/10723311/ ), there is no way to  be sure
we will never have legacy OSes that are not protected  against this.  It would
also be 'non ideal' to fallback to  a default node as there may be a better
(non GI) node to pick  if GI nodes aren't available.

The work around is that we also have a new system wide OSC bit that allows
an operating system to 'announce' that it supports Generic Initiators.  This
allows, the firmware to us DSDT magic to 'move' devices between the nodes
dependent on whether our new nodes are there or not.

2. New ways of assigning a proximity domain for devices
*******************************************************

Until now, the only way firmware could indicate that a particular device
(outside the 'special' set of cpus etc) was to be found in a particular
Proximity Domain by the use of _PXM in DSDT.

That is equally valid with GI domains, but we have new options. The SRAT
affinity structure includes a handle (ACPI or PCI) to identify devices
with the system and specify their proximity domain that way.  If both _PXM
and this are provided, they should give the same answer.

For now this patch set completely ignores that feature as we don't need
it to start the discussion.  It will form a follow up set at some point
(if no one else fancies doing it).
 

Jonathan Cameron (4):
  ACPI: Support Generic Initiator only domains
  arm64: Support Generic Initiator only domains
  x86: Support Generic Initiator only proximity domains
  ACPI: Let ACPI know we support Generic Initiator Affinity Structures

 arch/arm64/kernel/smp.c        |  8 +++++
 arch/x86/include/asm/numa.h    |  2 ++
 arch/x86/kernel/setup.c        |  1 +
 arch/x86/mm/numa.c             | 14 ++++++++
 drivers/acpi/bus.c             |  1 +
 drivers/acpi/numa.c            | 62 +++++++++++++++++++++++++++++++++-
 drivers/base/node.c            |  3 ++
 include/asm-generic/topology.h |  3 ++
 include/linux/acpi.h           |  1 +
 include/linux/nodemask.h       |  1 +
 include/linux/topology.h       |  7 ++++
 11 files changed, 102 insertions(+), 1 deletion(-)

-- 
2.20.1

Comments

Rafael J. Wysocki Sept. 2, 2019, 9:26 p.m. | #1
On Wed, Aug 21, 2019 at 4:53 PM Jonathan Cameron
<Jonathan.Cameron@huawei.com> wrote:
>

> Generic Initiators are a new ACPI concept that allows for the

> description of proximity domains that contain a device which

> performs memory access (such as a network card) but neither

> host CPU nor Memory.

>

> This patch has the parsing code and provides the infrastructure

> for an architecture to associate these new domains with their

> nearest memory processing node.

>

> Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>


Dan, Keith, any comments?

AFAICS this clashes with the series from Dan that rearranges the ACPI
NUMA related code.

> ---

>  drivers/acpi/numa.c            | 62 +++++++++++++++++++++++++++++++++-

>  drivers/base/node.c            |  3 ++

>  include/asm-generic/topology.h |  3 ++

>  include/linux/nodemask.h       |  1 +

>  include/linux/topology.h       |  7 ++++

>  5 files changed, 75 insertions(+), 1 deletion(-)

>

> diff --git a/drivers/acpi/numa.c b/drivers/acpi/numa.c

> index eadbf90e65d1..fe34315a9234 100644

> --- a/drivers/acpi/numa.c

> +++ b/drivers/acpi/numa.c

> @@ -170,6 +170,38 @@ acpi_table_print_srat_entry(struct acpi_subtable_header *header)

>                 }

>                 break;

>

> +       case ACPI_SRAT_TYPE_GENERIC_AFFINITY:

> +       {

> +               struct acpi_srat_generic_affinity *p =

> +                       (struct acpi_srat_generic_affinity *)header;

> +               char name[9] = {};

> +

> +               if (p->device_handle_type == 0) {

> +                       /*

> +                        * For pci devices this may be the only place they

> +                        * are assigned a proximity domain

> +                        */

> +                       pr_debug("SRAT Generic Initiator(Seg:%u BDF:%u) in proximity domain %d %s\n",

> +                                *(u16 *)(&p->device_handle[0]),

> +                                *(u16 *)(&p->device_handle[2]),

> +                                p->proximity_domain,

> +                                (p->flags & ACPI_SRAT_GENERIC_AFFINITY_ENABLED) ?

> +                               "enabled" : "disabled");

> +               } else {

> +                       /*

> +                        * In this case we can rely on the device having a

> +                        * proximity domain reference

> +                        */

> +                       memcpy(name, p->device_handle, 8);

> +                       pr_info("SRAT Generic Initiator(HID=%.8s UID=%.4s) in proximity domain %d %s\n",

> +                               (char *)(&p->device_handle[0]),

> +                               (char *)(&p->device_handle[8]),

> +                               p->proximity_domain,

> +                               (p->flags & ACPI_SRAT_GENERIC_AFFINITY_ENABLED) ?

> +                               "enabled" : "disabled");

> +               }

> +       }

> +       break;

>         default:

>                 pr_warn("Found unsupported SRAT entry (type = 0x%x)\n",

>                         header->type);

> @@ -378,6 +410,32 @@ acpi_parse_gicc_affinity(union acpi_subtable_headers *header,

>         return 0;

>  }

>

> +static int __init

> +acpi_parse_gi_affinity(union acpi_subtable_headers *header,

> +                      const unsigned long end)

> +{

> +       struct acpi_srat_generic_affinity *gi_affinity;

> +       int node;

> +

> +       gi_affinity = (struct acpi_srat_generic_affinity *)header;

> +       if (!gi_affinity)

> +               return -EINVAL;

> +       acpi_table_print_srat_entry(&header->common);

> +

> +       if (!(gi_affinity->flags & ACPI_SRAT_GENERIC_AFFINITY_ENABLED))

> +               return -EINVAL;

> +

> +       node = acpi_map_pxm_to_node(gi_affinity->proximity_domain);

> +       if (node == NUMA_NO_NODE || node >= MAX_NUMNODES) {

> +               pr_err("SRAT: Too many proximity domains.\n");

> +               return -EINVAL;

> +       }

> +       node_set(node, numa_nodes_parsed);

> +       node_set_state(node, N_GENERIC_INITIATOR);

> +

> +       return 0;

> +}

> +

>  static int __initdata parsed_numa_memblks;

>

>  static int __init

> @@ -433,7 +491,7 @@ int __init acpi_numa_init(void)

>

>         /* SRAT: System Resource Affinity Table */

>         if (!acpi_table_parse(ACPI_SIG_SRAT, acpi_parse_srat)) {

> -               struct acpi_subtable_proc srat_proc[3];

> +               struct acpi_subtable_proc srat_proc[4];

>

>                 memset(srat_proc, 0, sizeof(srat_proc));

>                 srat_proc[0].id = ACPI_SRAT_TYPE_CPU_AFFINITY;

> @@ -442,6 +500,8 @@ int __init acpi_numa_init(void)

>                 srat_proc[1].handler = acpi_parse_x2apic_affinity;

>                 srat_proc[2].id = ACPI_SRAT_TYPE_GICC_AFFINITY;

>                 srat_proc[2].handler = acpi_parse_gicc_affinity;

> +               srat_proc[3].id = ACPI_SRAT_TYPE_GENERIC_AFFINITY;

> +               srat_proc[3].handler = acpi_parse_gi_affinity;

>

>                 acpi_table_parse_entries_array(ACPI_SIG_SRAT,

>                                         sizeof(struct acpi_table_srat),

> diff --git a/drivers/base/node.c b/drivers/base/node.c

> index 75b7e6f6535b..6f60689af5f8 100644

> --- a/drivers/base/node.c

> +++ b/drivers/base/node.c

> @@ -980,6 +980,8 @@ static struct node_attr node_state_attr[] = {

>  #endif

>         [N_MEMORY] = _NODE_ATTR(has_memory, N_MEMORY),

>         [N_CPU] = _NODE_ATTR(has_cpu, N_CPU),

> +       [N_GENERIC_INITIATOR] = _NODE_ATTR(has_generic_initiator,

> +                                          N_GENERIC_INITIATOR),

>  };

>

>  static struct attribute *node_state_attrs[] = {

> @@ -991,6 +993,7 @@ static struct attribute *node_state_attrs[] = {

>  #endif

>         &node_state_attr[N_MEMORY].attr.attr,

>         &node_state_attr[N_CPU].attr.attr,

> +       &node_state_attr[N_GENERIC_INITIATOR].attr.attr,

>         NULL

>  };

>

> diff --git a/include/asm-generic/topology.h b/include/asm-generic/topology.h

> index 238873739550..54d0b4176a45 100644

> --- a/include/asm-generic/topology.h

> +++ b/include/asm-generic/topology.h

> @@ -71,6 +71,9 @@

>  #ifndef set_cpu_numa_mem

>  #define set_cpu_numa_mem(cpu, node)

>  #endif

> +#ifndef set_gi_numa_mem

> +#define set_gi_numa_mem(gi, node)

> +#endif

>

>  #endif /* !CONFIG_NUMA || !CONFIG_HAVE_MEMORYLESS_NODES */

>

> diff --git a/include/linux/nodemask.h b/include/linux/nodemask.h

> index 27e7fa36f707..1aebf766fb52 100644

> --- a/include/linux/nodemask.h

> +++ b/include/linux/nodemask.h

> @@ -399,6 +399,7 @@ enum node_states {

>  #endif

>         N_MEMORY,               /* The node has memory(regular, high, movable) */

>         N_CPU,          /* The node has one or more cpus */

> +       N_GENERIC_INITIATOR,    /* The node is a GI only node */

>         NR_NODE_STATES

>  };

>

> diff --git a/include/linux/topology.h b/include/linux/topology.h

> index 47a3e3c08036..2f97754e0508 100644

> --- a/include/linux/topology.h

> +++ b/include/linux/topology.h

> @@ -125,6 +125,13 @@ static inline void set_numa_mem(int node)

>  }

>  #endif

>

> +#ifndef set_gi_numa_mem

> +static inline void set_gi_numa_mem(int gi, int node)

> +{

> +       _node_numa_mem_[gi] = node;

> +}

> +#endif

> +

>  #ifndef node_to_mem_node

>  static inline int node_to_mem_node(int node)

>  {

> --

> 2.20.1

>
Jonathan Cameron Sept. 5, 2019, 9:16 a.m. | #2
On Mon, 2 Sep 2019 23:26:16 +0200
"Rafael J. Wysocki" <rafael@kernel.org> wrote:

> On Wed, Aug 21, 2019 at 4:53 PM Jonathan Cameron

> <Jonathan.Cameron@huawei.com> wrote:

> >

> > Generic Initiators are a new ACPI concept that allows for the

> > description of proximity domains that contain a device which

> > performs memory access (such as a network card) but neither

> > host CPU nor Memory.

> >

> > This patch has the parsing code and provides the infrastructure

> > for an architecture to associate these new domains with their

> > nearest memory processing node.

> >

> > Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>  

> 

> Dan, Keith, any comments?

> 

> AFAICS this clashes with the series from Dan that rearranges the ACPI

> NUMA related code.


Seems that one is going forwards now which is great. I'll rebase this on
top of Dan's series and send a v5 sometime soon.

Thanks,

Jonathan

> 

> > ---

> >  drivers/acpi/numa.c            | 62 +++++++++++++++++++++++++++++++++-

> >  drivers/base/node.c            |  3 ++

> >  include/asm-generic/topology.h |  3 ++

> >  include/linux/nodemask.h       |  1 +

> >  include/linux/topology.h       |  7 ++++

> >  5 files changed, 75 insertions(+), 1 deletion(-)

> >

> > diff --git a/drivers/acpi/numa.c b/drivers/acpi/numa.c

> > index eadbf90e65d1..fe34315a9234 100644

> > --- a/drivers/acpi/numa.c

> > +++ b/drivers/acpi/numa.c

> > @@ -170,6 +170,38 @@ acpi_table_print_srat_entry(struct acpi_subtable_header *header)

> >                 }

> >                 break;

> >

> > +       case ACPI_SRAT_TYPE_GENERIC_AFFINITY:

> > +       {

> > +               struct acpi_srat_generic_affinity *p =

> > +                       (struct acpi_srat_generic_affinity *)header;

> > +               char name[9] = {};

> > +

> > +               if (p->device_handle_type == 0) {

> > +                       /*

> > +                        * For pci devices this may be the only place they

> > +                        * are assigned a proximity domain

> > +                        */

> > +                       pr_debug("SRAT Generic Initiator(Seg:%u BDF:%u) in proximity domain %d %s\n",

> > +                                *(u16 *)(&p->device_handle[0]),

> > +                                *(u16 *)(&p->device_handle[2]),

> > +                                p->proximity_domain,

> > +                                (p->flags & ACPI_SRAT_GENERIC_AFFINITY_ENABLED) ?

> > +                               "enabled" : "disabled");

> > +               } else {

> > +                       /*

> > +                        * In this case we can rely on the device having a

> > +                        * proximity domain reference

> > +                        */

> > +                       memcpy(name, p->device_handle, 8);

> > +                       pr_info("SRAT Generic Initiator(HID=%.8s UID=%.4s) in proximity domain %d %s\n",

> > +                               (char *)(&p->device_handle[0]),

> > +                               (char *)(&p->device_handle[8]),

> > +                               p->proximity_domain,

> > +                               (p->flags & ACPI_SRAT_GENERIC_AFFINITY_ENABLED) ?

> > +                               "enabled" : "disabled");

> > +               }

> > +       }

> > +       break;

> >         default:

> >                 pr_warn("Found unsupported SRAT entry (type = 0x%x)\n",

> >                         header->type);

> > @@ -378,6 +410,32 @@ acpi_parse_gicc_affinity(union acpi_subtable_headers *header,

> >         return 0;

> >  }

> >

> > +static int __init

> > +acpi_parse_gi_affinity(union acpi_subtable_headers *header,

> > +                      const unsigned long end)

> > +{

> > +       struct acpi_srat_generic_affinity *gi_affinity;

> > +       int node;

> > +

> > +       gi_affinity = (struct acpi_srat_generic_affinity *)header;

> > +       if (!gi_affinity)

> > +               return -EINVAL;

> > +       acpi_table_print_srat_entry(&header->common);

> > +

> > +       if (!(gi_affinity->flags & ACPI_SRAT_GENERIC_AFFINITY_ENABLED))

> > +               return -EINVAL;

> > +

> > +       node = acpi_map_pxm_to_node(gi_affinity->proximity_domain);

> > +       if (node == NUMA_NO_NODE || node >= MAX_NUMNODES) {

> > +               pr_err("SRAT: Too many proximity domains.\n");

> > +               return -EINVAL;

> > +       }

> > +       node_set(node, numa_nodes_parsed);

> > +       node_set_state(node, N_GENERIC_INITIATOR);

> > +

> > +       return 0;

> > +}

> > +

> >  static int __initdata parsed_numa_memblks;

> >

> >  static int __init

> > @@ -433,7 +491,7 @@ int __init acpi_numa_init(void)

> >

> >         /* SRAT: System Resource Affinity Table */

> >         if (!acpi_table_parse(ACPI_SIG_SRAT, acpi_parse_srat)) {

> > -               struct acpi_subtable_proc srat_proc[3];

> > +               struct acpi_subtable_proc srat_proc[4];

> >

> >                 memset(srat_proc, 0, sizeof(srat_proc));

> >                 srat_proc[0].id = ACPI_SRAT_TYPE_CPU_AFFINITY;

> > @@ -442,6 +500,8 @@ int __init acpi_numa_init(void)

> >                 srat_proc[1].handler = acpi_parse_x2apic_affinity;

> >                 srat_proc[2].id = ACPI_SRAT_TYPE_GICC_AFFINITY;

> >                 srat_proc[2].handler = acpi_parse_gicc_affinity;

> > +               srat_proc[3].id = ACPI_SRAT_TYPE_GENERIC_AFFINITY;

> > +               srat_proc[3].handler = acpi_parse_gi_affinity;

> >

> >                 acpi_table_parse_entries_array(ACPI_SIG_SRAT,

> >                                         sizeof(struct acpi_table_srat),

> > diff --git a/drivers/base/node.c b/drivers/base/node.c

> > index 75b7e6f6535b..6f60689af5f8 100644

> > --- a/drivers/base/node.c

> > +++ b/drivers/base/node.c

> > @@ -980,6 +980,8 @@ static struct node_attr node_state_attr[] = {

> >  #endif

> >         [N_MEMORY] = _NODE_ATTR(has_memory, N_MEMORY),

> >         [N_CPU] = _NODE_ATTR(has_cpu, N_CPU),

> > +       [N_GENERIC_INITIATOR] = _NODE_ATTR(has_generic_initiator,

> > +                                          N_GENERIC_INITIATOR),

> >  };

> >

> >  static struct attribute *node_state_attrs[] = {

> > @@ -991,6 +993,7 @@ static struct attribute *node_state_attrs[] = {

> >  #endif

> >         &node_state_attr[N_MEMORY].attr.attr,

> >         &node_state_attr[N_CPU].attr.attr,

> > +       &node_state_attr[N_GENERIC_INITIATOR].attr.attr,

> >         NULL

> >  };

> >

> > diff --git a/include/asm-generic/topology.h b/include/asm-generic/topology.h

> > index 238873739550..54d0b4176a45 100644

> > --- a/include/asm-generic/topology.h

> > +++ b/include/asm-generic/topology.h

> > @@ -71,6 +71,9 @@

> >  #ifndef set_cpu_numa_mem

> >  #define set_cpu_numa_mem(cpu, node)

> >  #endif

> > +#ifndef set_gi_numa_mem

> > +#define set_gi_numa_mem(gi, node)

> > +#endif

> >

> >  #endif /* !CONFIG_NUMA || !CONFIG_HAVE_MEMORYLESS_NODES */

> >

> > diff --git a/include/linux/nodemask.h b/include/linux/nodemask.h

> > index 27e7fa36f707..1aebf766fb52 100644

> > --- a/include/linux/nodemask.h

> > +++ b/include/linux/nodemask.h

> > @@ -399,6 +399,7 @@ enum node_states {

> >  #endif

> >         N_MEMORY,               /* The node has memory(regular, high, movable) */

> >         N_CPU,          /* The node has one or more cpus */

> > +       N_GENERIC_INITIATOR,    /* The node is a GI only node */

> >         NR_NODE_STATES

> >  };

> >

> > diff --git a/include/linux/topology.h b/include/linux/topology.h

> > index 47a3e3c08036..2f97754e0508 100644

> > --- a/include/linux/topology.h

> > +++ b/include/linux/topology.h

> > @@ -125,6 +125,13 @@ static inline void set_numa_mem(int node)

> >  }

> >  #endif

> >

> > +#ifndef set_gi_numa_mem

> > +static inline void set_gi_numa_mem(int gi, int node)

> > +{

> > +       _node_numa_mem_[gi] = node;

> > +}

> > +#endif

> > +

> >  #ifndef node_to_mem_node

> >  static inline int node_to_mem_node(int node)

> >  {

> > --

> > 2.20.1

> >  

>