| Message ID | 20170328192330.62599-1-brian.brooks@arm.com |
|---|---|
| State | Superseded |
| Headers | show |
| Series | [API-NEXT,1/4] helper: cuckootable: Specify queue ring_size | expand |
This part generates numerous checkpatch warnings and errors. Please
run checkpatch and correct for v2.
Also, this part introduces a number of errors that result in failure
to compile using clang. Please test with both gcc and clang to ensure
that it compiles cleanly for both (gcc looks fine)
Specific clang issues:
CC odp_packet.lo
odp_packet.c:656:9: error: cast from 'odp_buffer_t' (aka
'_odp_abi_buffer_t *') to 'odp_buffer_hdr_t *' (aka 'struct
odp_buffer_hdr_t *') increases required alignment from 1 to 8
[-Werror,-Wcast-align]
return (odp_buffer_hdr_t *)(_odp_packet_to_buffer(pkt));
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
clang requires an intermediate cast to (void *) to avoid these warnings.
1 error generated.
Makefile:1004: recipe for target 'odp_packet.lo' failed
clang also generates numerous link errors:
CC odp_bench_packet-odp_bench_packet.o
make[3]: *** No rule to make target '../../../lib/libodp-linux.la',
needed by 'odp_bench_packet'.
. CC atomic_main.o
make[5]: *** No rule to make target
'../../../../../lib/libodp-linux.la', needed by 'atomic_main'.
CC classification_main.o
make[5]: *** No rule to make target
'../../../../../lib/libodp-linux.la', needed by 'classification_main'.
...and many others like this.
On Tue, Mar 28, 2017 at 2:23 PM, Brian Brooks <brian.brooks@arm.com> wrote:
> This work derives from Ola Liljedahl's prototype [1] which introduced a
> scalable scheduler design based on primarily lock-free algorithms and
> data structures designed to decrease contention. A thread searches
> through a data structure containing only queues that are both non-empty
> and allowed to be scheduled to that thread. Strict priority scheduling is
> respected, and (W)RR scheduling may be used within queues of the same priority.
> Lastly, pre-scheduling or stashing is not employed since it is optional
> functionality that can be implemented in the application.
>
> In addition to scalable ring buffers, the algorithm also uses unbounded
> concurrent queues. LL/SC and CAS variants exist in cases where absense of
> ABA problem cannot be proved, and also in cases where the compiler's atomic
> built-ins may not be lowered to the desired instruction(s). Finally, a version
> of the algorithm that uses locks is also provided.
>
> See platform/linux-generic/include/odp_config_internal.h for further build
> time configuration.
>
> Use --enable-schedule-scalable to conditionally compile this scheduler
> into the library.
>
> [1] https://lists.linaro.org/pipermail/lng-odp/2016-September/025682.html
>
> Signed-off-by: Brian Brooks <brian.brooks@arm.com>
> Signed-off-by: Kevin Wang <kevin.wang@arm.com>
> Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> Signed-off-by: Ola Liljedahl <ola.liljedahl@arm.com>
> ---
> platform/linux-generic/Makefile.am | 21 +-
> .../include/odp/api/plat/schedule_types.h | 20 +-
> platform/linux-generic/include/odp_atomic16.h | 244 +++
> platform/linux-generic/include/odp_bitset.h | 155 ++
> .../linux-generic/include/odp_config_internal.h | 90 +-
> platform/linux-generic/include/odp_internal.h | 2 +
> platform/linux-generic/include/odp_llqueue.h | 386 ++++
> platform/linux-generic/include/odp_llsc.h | 325 ++++
> .../linux-generic/include/odp_packet_internal.h | 3 +
> .../linux-generic/include/odp_queue_internal.h | 122 +-
> platform/linux-generic/include/odp_schedule_if.h | 166 +-
> .../include/odp_schedule_ordered_internal.h | 150 ++
> platform/linux-generic/m4/odp_schedule.m4 | 55 +-
> platform/linux-generic/odp_classification.c | 4 +-
> platform/linux-generic/odp_packet.c | 5 +
> platform/linux-generic/odp_packet_io.c | 88 +-
> platform/linux-generic/odp_queue.c | 2 +-
> platform/linux-generic/odp_queue_scalable.c | 944 ++++++++++
> platform/linux-generic/odp_schedule_if.c | 36 +-
> platform/linux-generic/odp_schedule_scalable.c | 1959 ++++++++++++++++++++
> .../linux-generic/odp_schedule_scalable_ordered.c | 298 +++
> platform/linux-generic/odp_traffic_mngr.c | 7 +-
> platform/linux-generic/pktio/loop.c | 11 +-
> test/common_plat/performance/odp_sched_latency.c | 69 +-
> test/common_plat/performance/odp_scheduling.c | 12 +-
> .../api/classification/odp_classification_basic.c | 8 +-
> .../classification/odp_classification_test_pmr.c | 42 +-
> .../validation/api/scheduler/scheduler.c | 11 +-
> test/common_plat/validation/api/timer/timer.c | 5 +-
> 29 files changed, 5123 insertions(+), 117 deletions(-)
> create mode 100644 platform/linux-generic/include/odp_atomic16.h
> create mode 100644 platform/linux-generic/include/odp_bitset.h
> create mode 100644 platform/linux-generic/include/odp_llqueue.h
> create mode 100644 platform/linux-generic/include/odp_llsc.h
> create mode 100644 platform/linux-generic/include/odp_schedule_ordered_internal.h
> create mode 100644 platform/linux-generic/odp_queue_scalable.c
> create mode 100644 platform/linux-generic/odp_schedule_scalable.c
> create mode 100644 platform/linux-generic/odp_schedule_scalable_ordered.c
>
> diff --git a/platform/linux-generic/Makefile.am b/platform/linux-generic/Makefile.am
> index 70683cac..8c263b99 100644
> --- a/platform/linux-generic/Makefile.am
> +++ b/platform/linux-generic/Makefile.am
> @@ -151,6 +151,8 @@ noinst_HEADERS = \
> ${srcdir}/include/odp_debug_internal.h \
> ${srcdir}/include/odp_forward_typedefs_internal.h \
> ${srcdir}/include/odp_internal.h \
> + ${srcdir}/include/odp_llqueue.h \
> + ${srcdir}/include/odp_llsc.h \
> ${srcdir}/include/odp_name_table_internal.h \
> ${srcdir}/include/odp_packet_internal.h \
> ${srcdir}/include/odp_packet_io_internal.h \
> @@ -219,13 +221,9 @@ __LIB__libodp_linux_la_SOURCES = \
> pktio/ring.c \
> odp_pkt_queue.c \
> odp_pool.c \
> - odp_queue.c \
> odp_rwlock.c \
> odp_rwlock_recursive.c \
> - odp_schedule.c \
> odp_schedule_if.c \
> - odp_schedule_sp.c \
> - odp_schedule_iquery.c \
> odp_shared_memory.c \
> odp_sorted_list.c \
> odp_spinlock.c \
> @@ -250,6 +248,21 @@ __LIB__libodp_linux_la_SOURCES = \
> arch/@ARCH_DIR@/odp_cpu_arch.c \
> arch/@ARCH_DIR@/odp_sysinfo_parse.c
>
> +if ODP_SCHEDULE_SP
> +__LIB__libodp_linux_la_SOURCES += odp_schedule_sp.c
> +endif
> +
> +if ODP_SCHEDULE_IQUERY
> +__LIB__libodp_linux_la_SOURCES += odp_schedule_iquery.c
> +endif
> +
> +if ODP_SCHEDULE_SCALABLE
> +__LIB__libodp_linux_la_SOURCES += odp_queue_scalable.c odp_schedule_scalable.c \
> + odp_schedule_scalable_ordered.c
> +else
> +__LIB__libodp_linux_la_SOURCES += odp_queue.c odp_schedule.c
> +endif
> +
> if HAVE_PCAP
> __LIB__libodp_linux_la_SOURCES += pktio/pcap.c
> endif
> diff --git a/platform/linux-generic/include/odp/api/plat/schedule_types.h b/platform/linux-generic/include/odp/api/plat/schedule_types.h
> index 535fd6d0..d9bb39a6 100644
> --- a/platform/linux-generic/include/odp/api/plat/schedule_types.h
> +++ b/platform/linux-generic/include/odp/api/plat/schedule_types.h
> @@ -18,6 +18,8 @@
> extern "C" {
> #endif
>
> +#include <odp/api/std_types.h>
> +
> /** @addtogroup odp_scheduler
> * @{
> */
> @@ -27,6 +29,20 @@ extern "C" {
>
> typedef int odp_schedule_prio_t;
>
> +#ifdef ODP_SCHEDULE_SCALABLE
> +
> +#define ODP_SCHED_PRIO_NUM 8
> +
> +#define ODP_SCHED_PRIO_HIGHEST 0
> +
> +#define ODP_SCHED_PRIO_LOWEST (ODP_SCHED_PRIO_NUM - 1)
> +
> +#define ODP_SCHED_PRIO_DEFAULT (ODP_SCHED_PRIO_NUM / 2)
> +
> +#define ODP_SCHED_PRIO_NORMAL ODP_SCHED_PRIO_DEFAULT
> +
> +#else
> +
> #define ODP_SCHED_PRIO_HIGHEST 0
>
> #define ODP_SCHED_PRIO_NORMAL 4
> @@ -35,6 +51,8 @@ typedef int odp_schedule_prio_t;
>
> #define ODP_SCHED_PRIO_DEFAULT ODP_SCHED_PRIO_NORMAL
>
> +#endif
> +
> typedef int odp_schedule_sync_t;
>
> #define ODP_SCHED_SYNC_PARALLEL 0
> @@ -44,7 +62,7 @@ typedef int odp_schedule_sync_t;
> typedef int odp_schedule_group_t;
>
> /* These must be kept in sync with thread_globals_t in odp_thread.c */
> -#define ODP_SCHED_GROUP_INVALID -1
> +#define ODP_SCHED_GROUP_INVALID ((odp_schedule_group_t) -1)
> #define ODP_SCHED_GROUP_ALL 0
> #define ODP_SCHED_GROUP_WORKER 1
> #define ODP_SCHED_GROUP_CONTROL 2
> diff --git a/platform/linux-generic/include/odp_atomic16.h b/platform/linux-generic/include/odp_atomic16.h
> new file mode 100644
> index 00000000..5afd66c5
> --- /dev/null
> +++ b/platform/linux-generic/include/odp_atomic16.h
> @@ -0,0 +1,244 @@
> +/* Copyright (c) 2017, ARM Limited.
> + * All rights reserved.
> + *
> + * SPDX-License-Identifier: BSD-3-Clause
> + */
> +
> +#ifndef _ODP_ATOMIC16_H_
> +#define _ODP_ATOMIC16_H_
> +
> +#include "odp_llsc.h"
> +
> +#if defined __ARM_ARCH && __ARM_ARCH == 8
> +
> +#ifdef __ARM_FEATURE_QRDMX /* Feature only available in v8.1a and beyond */
> +static inline __int128 casp(__int128 *var, __int128 old, __int128 neu, int mo)
> +{
> + if (mo == __ATOMIC_RELAXED) {
> + __asm __volatile("casp %0, %H0, %1, %H1, [%2]"
> + : "+r" (old)
> + : "r" (neu), "r" (var)
> + : "memory");
> + } else if (mo == __ATOMIC_ACQUIRE) {
> + __asm __volatile("caspa %0, %H0, %1, %H1, [%2]"
> + : "+r" (old)
> + : "r" (neu), "r" (var)
> + : "memory");
> + } else if (mo == __ATOMIC_ACQ_REL) {
> + __asm __volatile("caspal %0, %H0, %1, %H1, [%2]"
> + : "+r" (old)
> + : "r" (neu), "r" (var)
> + : "memory");
> + } else if (mo == __ATOMIC_RELEASE) {
> + __asm __volatile("caspl %0, %H0, %1, %H1, [%2]"
> + : "+r" (old)
> + : "r" (neu), "r" (var)
> + : "memory");
> + } else {
> + abort();
> + }
> + return old;
> +}
> +#endif
> +
> +static inline bool __atomic_compare_exchange_16(register __int128 *var,
> + __int128 *exp, register __int128 neu,
> + bool weak, int mo_success, int mo_failure)
> +{
> +#ifdef __ARM_FEATURE_QRDMX /* Feature only available in v8.1a and beyond */
> + (void)weak;
> + (void)mo_failure;
> + __int128 old;
> + __int128 expected;
> +
> + expected = *exp;
> + old = casp(var, expected, neu, mo_success);
> + *exp = old; /* Always update, atomically read value */
> + return old == expected;
> +#else
> + (void)weak; /* Always do strong CAS or we can't perform atomic read */
> + /* Ignore memory ordering for failure, memory order for
> + * success must be stronger or equal
> + */
> + (void)mo_failure;
> + int ll_mo;
> + int sc_mo;
> +
> + register __int128 old;
> + register __int128 expected;
> +
> + ll_mo = mo_success == __ATOMIC_ACQUIRE ||
> + mo_success == __ATOMIC_ACQ_REL ?
> + __ATOMIC_ACQUIRE : __ATOMIC_RELAXED;
> +
> + sc_mo = mo_success == __ATOMIC_RELEASE ||
> + mo_success == __ATOMIC_ACQ_REL ?
> + __ATOMIC_RELEASE : __ATOMIC_RELAXED;
> +
> + expected = *exp;
> + __asm __volatile("" ::: "memory");
> + do {
> + /* Atomicity of LLD is not guaranteed */
> + old = lld(var, ll_mo);
> + /* Must write back neu or old to verify atomicity of LLD */
> + } while (odp_unlikely(scd(var, old == expected ? neu : old, sc_mo)));
> + *exp = old; /* Always update, atomically read value */
> + return old == expected;
> +#endif
> +}
> +
> +static inline bool __atomic_compare_exchange_16_frail(register __int128 *var,
> + __int128 *exp, register __int128 neu, bool weak,
> + int mo_success, int mo_failure)
> +{
> +#ifdef __ARM_FEATURE_QRDMX /* Feature only available in v8.1a and beyond */
> + (void)weak;
> + (void)mo_failure;
> + __int128 old;
> + __int128 expected;
> +
> + expected = *exp;
> + old = casp(var, expected, neu, mo_success);
> + *exp = old; /* Always update, atomically read value */
> + return old == expected;
> +#else
> + (void)weak; /* Weak CAS and non-atomic load on failure */
> + (void)mo_failure; /* Ignore memory ordering for failure */
> + int ll_mo;
> + int sc_mo;
> + register __int128 expected;
> + register __int128 old;
> +
> + /* memory order for success must be stronger or equal */
> + ll_mo = mo_success == __ATOMIC_ACQUIRE ||
> + mo_success == __ATOMIC_ACQ_REL ?
> + __ATOMIC_ACQUIRE : __ATOMIC_RELAXED;
> + sc_mo = mo_success == __ATOMIC_RELEASE ||
> + mo_success == __ATOMIC_ACQ_REL ?
> + __ATOMIC_RELEASE : __ATOMIC_RELAXED;
> + expected = *exp;
> + __asm __volatile("" ::: "memory");
> + /* Atomicity of LLD is not guaranteed */
> + old = lld(var, ll_mo);
> + if (odp_likely(old == expected && !scd(var, neu, sc_mo))) {
> + /* Right value and SC succeeded */
> + __asm __volatile("" ::: "memory");
> + return 1;
> + }
> + __asm __volatile("" ::: "memory");
> + /* Wrong value or SC failed */
> + *exp = old; /* Old possibly torn value */
> + return 0; /* Failure, *exp updated */
> +#endif
> +}
> +
> +static inline __int128 __atomic_load_16(__int128 *var, int mo)
> +{
> + __int128 old = *var; /* Possibly torn read */
> +
> + /* Do CAS to ensure atomicity
> + * Either CAS succeeds (writing back the same value)
> + * Or CAS fails and returns the old value (atomic read)
> + */
> + (void)__atomic_compare_exchange_n(var,
> + &old,
> + old,
> + false, /* weak= */
> + mo,
> + mo);
> + return old;
> +}
> +
> +static inline __int128 __atomic_exchange_16(__int128 *var,
> + __int128 neu, int mo)
> +{
> +#ifdef __ARM_FEATURE_QRDMX /* Feature only available in v8.1a and beyond */
> + __int128 old;
> + __int128 expected;
> +
> + do {
> + expected = *var;
> + old = casp(var, expected, neu, mo);
> + } while (old != expected);
> + return old;
> +#else
> + int ll_mo;
> + int sc_mo;
> + register __int128 old;
> +
> + ll_mo = mo == __ATOMIC_ACQUIRE || mo == __ATOMIC_ACQ_REL ?
> + __ATOMIC_ACQUIRE : __ATOMIC_RELAXED;
> + sc_mo = mo == __ATOMIC_RELEASE || mo == __ATOMIC_ACQ_REL ?
> + __ATOMIC_RELEASE : __ATOMIC_RELAXED;
> + do {
> + /* Atomicity of LLD is not guaranteed */
> + old = lld(var, ll_mo);
> + /* Must successfully write back to verify atomicity of LLD */
> + } while (odp_unlikely(scd(var, neu, sc_mo)));
> + return old;
> +#endif
> +}
> +
> +static inline __int128 __atomic_fetch_and_16(__int128 *var,
> + __int128 mask, int mo)
> +{
> +#ifdef __ARM_FEATURE_QRDMX /* Feature only available in v8.1a and beyond */
> + __int128 old;
> + __int128 expected;
> +
> + do {
> + expected = *var;
> + old = casp(var, expected, expected & mask, mo);
> + } while (old != expected);
> + return old;
> +#else
> + int ll_mo;
> + int sc_mo;
> + register __int128 old;
> +
> + ll_mo = mo == __ATOMIC_ACQUIRE || mo == __ATOMIC_ACQ_REL ?
> + __ATOMIC_ACQUIRE : __ATOMIC_RELAXED;
> + sc_mo = mo == __ATOMIC_RELEASE || mo == __ATOMIC_ACQ_REL ?
> + __ATOMIC_RELEASE : __ATOMIC_RELAXED;
> + do {
> + /* Atomicity of LLD is not guaranteed */
> + old = lld(var, ll_mo);
> + /* Must successfully write back to verify atomicity of LLD */
> + } while (odp_unlikely(scd(var, old & mask, sc_mo)));
> + return old;
> +#endif
> +}
> +
> +static inline __int128 __atomic_fetch_or_16(__int128 *var,
> + __int128 mask,
> + int mo)
> +{
> +#ifdef __ARM_FEATURE_QRDMX /* Feature only available in v8.1a and beyond */
> + __int128 old;
> + __int128 expected;
> +
> + do {
> + expected = *var;
> + old = casp(var, expected, expected | mask, mo);
> + } while (old != expected);
> + return old;
> +#else
> + int ll_mo;
> + int sc_mo;
> + register __int128 old;
> +
> + ll_mo = mo == __ATOMIC_ACQUIRE || mo == __ATOMIC_ACQ_REL ?
> + __ATOMIC_ACQUIRE : __ATOMIC_RELAXED;
> + sc_mo = mo == __ATOMIC_RELEASE || mo == __ATOMIC_ACQ_REL ?
> + __ATOMIC_RELEASE : __ATOMIC_RELAXED;
> + do {
> + /* Atomicity of LLD is not guaranteed */
> + old = lld(var, ll_mo);
> + /* Must successfully write back to verify atomicity of LLD */
> + } while (odp_unlikely(scd(var, old | mask, sc_mo)));
> + return old;
> +#endif
> +}
> +
> +#endif
> +#endif
> diff --git a/platform/linux-generic/include/odp_bitset.h b/platform/linux-generic/include/odp_bitset.h
> new file mode 100644
> index 00000000..db004267
> --- /dev/null
> +++ b/platform/linux-generic/include/odp_bitset.h
> @@ -0,0 +1,155 @@
> +/* Copyright (c) 2017, ARM Limited
> + * All rights reserved.
> + *
> + * SPDX-License-Identifier: BSD-3-Clause
> + */
> +
> +#ifndef _ODP_BITSET_H_
> +#define _ODP_BITSET_H_
> +
> +/******************************************************************************
> + * bitset abstract data type
> + *****************************************************************************/
> +/* This could be a struct of scalars to support larger bit sets */
> +
> +#if ATOM_BITSET_SIZE <= 32
> +
> +typedef uint32_t bitset_t;
> +
> +static inline bitset_t bitset_mask(uint32_t bit)
> +{
> + return 1UL << bit;
> +}
> +
> +/* Return first-bit-set with StdC ffs() semantics */
> +static inline uint32_t bitset_ffs(bitset_t b)
> +{
> + return __builtin_ffsl(b);
> +}
> +
> +/* Load-exclusive with memory ordering */
> +static inline bitset_t bitset_ldex(bitset_t *bs, int mo)
> +{
> + return LDXR32(bs, mo);
> +}
> +
> +#elif ATOM_BITSET_SIZE <= 64
> +
> +typedef uint64_t bitset_t;
> +
> +static inline bitset_t bitset_mask(uint32_t bit)
> +{
> + return 1ULL << bit;
> +}
> +
> +/* Return first-bit-set with StdC ffs() semantics */
> +static inline uint32_t bitset_ffs(bitset_t b)
> +{
> + return __builtin_ffsll(b);
> +}
> +
> +/* Load-exclusive with memory ordering */
> +static inline bitset_t bitset_ldex(bitset_t *bs, int mo)
> +{
> + return LDXR64(bs, mo);
> +}
> +
> +#elif ATOM_BITSET_SIZE <= 128
> +
> +#if __SIZEOF_INT128__ == 16
> +typedef unsigned __int128 bitset_t;
> +
> +static inline bitset_t bitset_mask(uint32_t bit)
> +{
> + if (bit < 64)
> + return 1ULL << bit;
> + else
> + return (unsigned __int128)(1ULL << (bit - 64)) << 64;
> +}
> +
> +/* Return first-bit-set with StdC ffs() semantics */
> +static inline uint32_t bitset_ffs(bitset_t b)
> +{
> + if ((uint64_t)b != 0)
> + return __builtin_ffsll((uint64_t)b);
> + else if ((b >> 64) != 0)
> + return __builtin_ffsll((uint64_t)(b >> 64)) + 64;
> + else
> + return 0;
> +}
> +
> +/* Load-exclusive with memory ordering */
> +static inline bitset_t bitset_ldex(bitset_t *bs, int mo)
> +{
> + return LDXR128(bs, mo);
> +}
> +
> +#else
> +#error __int128 not supported by compiler
> +#endif
> +
> +#else
> +#error Unsupported size of bit sets (ATOM_BITSET_SIZE)
> +#endif
> +
> +/* Atomic load with memory ordering */
> +static inline bitset_t atom_bitset_load(bitset_t *bs, int mo)
> +{
> + return __atomic_load_n(bs, mo);
> +}
> +
> +/* Atomic bit set with memory ordering */
> +static inline void atom_bitset_set(bitset_t *bs, uint32_t bit, int mo)
> +{
> + (void)__atomic_fetch_or(bs, bitset_mask(bit), mo);
> +}
> +
> +/* Atomic bit clear with memory ordering */
> +static inline void atom_bitset_clr(bitset_t *bs, uint32_t bit, int mo)
> +{
> + (void)__atomic_fetch_and(bs, ~bitset_mask(bit), mo);
> +}
> +
> +/* Atomic exchange with memory ordering */
> +static inline bitset_t atom_bitset_xchg(bitset_t *bs, bitset_t neu, int mo)
> +{
> + return __atomic_exchange_n(bs, neu, mo);
> +}
> +
> +/* Return a & ~b */
> +static inline bitset_t bitset_andn(bitset_t a, bitset_t b)
> +{
> + return a & ~b;
> +}
> +
> +static inline bool bitset_is_eql(bitset_t a, bitset_t b)
> +{
> + return a == b;
> +}
> +
> +static inline bitset_t bitset_clr(bitset_t bs, uint32_t bit)
> +{
> + return bs & ~bitset_mask(bit);
> +}
> +
> +static inline bitset_t bitset_set(bitset_t bs, uint32_t bit)
> +{
> + return bs | bitset_mask(bit);
> +}
> +
> +static inline bitset_t bitset_null(void)
> +{
> + return 0U;
> +}
> +
> +static inline bool bitset_is_null(bitset_t a)
> +{
> + return a == 0U;
> +}
> +
> +static inline bool bitset_is_set(bitset_t a, uint32_t bit)
> +{
> + return (a & bitset_mask(bit)) != 0;
> +}
> +
> +#endif
> diff --git a/platform/linux-generic/include/odp_config_internal.h b/platform/linux-generic/include/odp_config_internal.h
> index dadd59e7..b1ee48f2 100644
> --- a/platform/linux-generic/include/odp_config_internal.h
> +++ b/platform/linux-generic/include/odp_config_internal.h
> @@ -22,6 +22,13 @@ extern "C" {
> #define ODP_CONFIG_QUEUES 1024
>
> /*
> + * Maximum queue depth. Maximum number of elements that can be stored in a
> + * queue. This value is used only when the size is not explicitly provided
> + * during queue creation.
> + */
> +#define ODP_CONFIG_QUEUE_SIZE 4096
> +
> +/*
> * Maximum number of ordered locks per queue
> */
> #define CONFIG_QUEUE_MAX_ORD_LOCKS 4
> @@ -104,7 +111,7 @@ extern "C" {
> *
> * This the the number of separate SHM areas that can be reserved concurrently
> */
> -#define ODP_CONFIG_SHM_BLOCKS (ODP_CONFIG_POOLS + 48)
> +#define ODP_CONFIG_SHM_BLOCKS (ODP_CONFIG_POOLS + ODP_CONFIG_QUEUES + 48)
>
> /*
> * Size of the virtual address space pre-reserver for ISHM
> @@ -120,7 +127,7 @@ extern "C" {
> *
> * This the the number of separate SHM areas that can be reserved concurrently
> */
> -#define ODPDRV_CONFIG_SHM_BLOCKS 48
> +#define ODPDRV_CONFIG_SHM_BLOCKS ODP_CONFIG_SHM_BLOCKS
>
> /* Maximum event burst size
> *
> @@ -129,6 +136,21 @@ extern "C" {
> */
> #define CONFIG_BURST_SIZE 16
>
> +/* Default weight (in events) for WRR in scalable scheduler
> + *
> + * This controls the per-queue weight for WRR between queues of the same
> + * priority in the scalable scheduler
> + * A higher value improves throughput while a lower value increases fairness
> + * and thus likely decreases latency
> + *
> + * If WRR is undesired, set the value to ~0 which will use the largest possible
> + * weight
> + *
> + * Note: an API for specifying this on a per-queue basis would be useful but is
> + * not yet available
> + */
> +#define CONFIG_WRR_WEIGHT 64
> +
> /*
> * Maximum number of events in a pool
> */
> @@ -139,6 +161,70 @@ extern "C" {
> */
> #define CONFIG_POOL_CACHE_SIZE 256
>
> +/*
> + * Split queue producer/consumer metadata into separate cache lines.
> + * This is beneficial on e.g. Cortex-A57 but not so much on A53.
> + */
> +#define ODP_CONFIG_USE_SPLIT_PRODCONS
> +
> +/*
> + * Split queue read/write metadata into separate cache lines.
> + * This enhances scalability even further on Cortex-A57.
> + */
> +#define ODP_CONFIG_USE_SPLIT_READWRITE
> +
> +/*
> + * Use locks to protect queue (ring buffer) and scheduler state updates
> + * On x86, this decreases overhead but also degrades scalability
> + */
> +#ifndef __ARM_ARCH
> +#define CONFIG_QSCHST_LOCK
> +#endif
> +
> +#ifdef CONFIG_QSCHST_LOCK
> +/* Keep all ring buffer/qschst data together when using locks
> + */
> +#undef ODP_CONFIG_USE_SPLIT_PRODCONS
> +#undef ODP_CONFIG_USE_SPLIT_READWRITE
> +#endif
> +
> +/*
> + * Use spin lock instead of (lock-free) atomic operations in llqueue
> + * This is recommended on architectures which implement CAS instead
> + * of LL/SC as CAS is susceptible to the ABA problem
> + */
> +#ifndef __ARM_ARCH
> +#define ODP_CONFIG_LLQ_LOCK
> +#endif
> +
> +/*
> + * Size of atomic bit set. This limits the max number of threads,
> + * scheduler groups and reorder windows. On ARMv8/64-bit and x86-64, the
> + * (lock-free) max is 128
> + */
> +#define ATOM_BITSET_SIZE 64
> +
> +/*
> + * Use LL/SC atomic primitives instead of __atomic_compare_exchange built-ins
> + */
> +#ifdef __ARM_ARCH
> +#define ODP_CONFIG_USE_LLSC
> +#endif
> +
> +/*
> + * Use DMB;STR instead of STRL on ARM
> + */
> +#ifdef __ARM_ARCH
> +#define ODP_CONFIG_USE_DMB
> +#endif
> +
> +/*
> + * Use ARM event signalling mechanism
> + */
> +#ifdef __ARM_ARCH
> +#define ODP_CONFIG_USE_WFE
> +#endif
> +
> #ifdef __cplusplus
> }
> #endif
> diff --git a/platform/linux-generic/include/odp_internal.h b/platform/linux-generic/include/odp_internal.h
> index 05c8a422..60eb9daa 100644
> --- a/platform/linux-generic/include/odp_internal.h
> +++ b/platform/linux-generic/include/odp_internal.h
> @@ -28,6 +28,8 @@ extern __thread int __odp_errno;
>
> #define MAX_CPU_NUMBER 128
>
> +#define IS_POWER_TWO(x) ((((x) - 1) & (x)) == 0)
> +
> typedef struct {
> uint64_t cpu_hz_max[MAX_CPU_NUMBER];
> uint64_t page_size;
> diff --git a/platform/linux-generic/include/odp_llqueue.h b/platform/linux-generic/include/odp_llqueue.h
> new file mode 100644
> index 00000000..6f20c641
> --- /dev/null
> +++ b/platform/linux-generic/include/odp_llqueue.h
> @@ -0,0 +1,386 @@
> +/* Copyright (c) 2017, ARM Limited.
> + * All rights reserved.
> + *
> + * SPDX-License-Identifier: BSD-3-Clause
> + */
> +
> +#ifndef ODP_LLQUEUE_H_
> +#define ODP_LLQUEUE_H_
> +
> +#include <odp/api/cpu.h>
> +#include <odp/api/hints.h>
> +#include <odp/api/spinlock.h>
> +
> +#include <odp_config_internal.h>
> +#include <odp_debug_internal.h>
> +#include <odp_llsc.h>
> +
> +#include <stdint.h>
> +#include <stdlib.h>
> +
> +/******************************************************************************
> + * Linked list queues
> + *****************************************************************************/
> +
> +#if defined(__ARM_ARCH) && !defined(ODP_CONFIG_USE_LLSC) && \
> + !defined(ODP_CONFIG_LLQ_LOCK)
> +#error ODP_CONFIG_USE_LLSC required on ARM for double-word atomics if not \
> + using spinlock
> +#endif
> +
> +/* The scalar equivalent of a double pointer */
> +#if __SIZEOF_PTRDIFF_T__ == 4
> +typedef uint64_t dintptr_t;
> +#endif
> +#if __SIZEOF_PTRDIFF_T__ == 8
> +typedef __int128 dintptr_t;
> +#endif
> +
> +#define SENTINEL ((void *)~(uintptr_t)0)
> +
> +struct llnode {
> + struct llnode *next;
> +};
> +
> +union llht {
> + struct {
> + struct llnode *head, *tail;
> + } st;
> + dintptr_t ui;
> +};
> +
> +struct llqueue {
> + union llht u;
> +#ifdef ODP_CONFIG_LLQ_LOCK
> + odp_spinlock_t lock;
> +#endif
> +};
> +
> +static inline struct llnode *llq_head(struct llqueue *llq)
> +{
> + return __atomic_load_n(&llq->u.st.head, __ATOMIC_RELAXED);
> +}
> +
> +static inline void llqueue_init(struct llqueue *llq)
> +{
> + llq->u.st.head = NULL;
> + llq->u.st.tail = NULL;
> +#ifdef ODP_CONFIG_LLQ_LOCK
> + odp_spinlock_init(&llq->lock);
> +#endif
> +}
> +
> +#ifndef ODP_CONFIG_LLQ_LOCK
> +
> +static inline void llq_enqueue(struct llqueue *llq, struct llnode *node)
> +{
> + union llht old;
> + union llht neu;
> +
> + ODP_ASSERT(node->next == NULL);
> + node->next = SENTINEL;
> +#ifdef ODP_CONFIG_USE_LLSC
> + /* Prefetch for store for faster LL/SC execution */
> + __builtin_prefetch(&llq->u.ui, 1, 0);
> +retry: /* Failed SC requires new LL */
> + old.ui = lld(&llq->u.ui, __ATOMIC_RELAXED);
> +#else
> + __atomic_load(&llq->u, &old, __ATOMIC_RELAXED);
> +retry: /* Failed CAS returns existing value */
> + (void)0; /* Need statement after label */
> +#endif
> + neu.st.head = old.st.head == NULL ? node : old.st.head;
> + neu.st.tail = node;
> +#ifdef ODP_CONFIG_USE_LLSC
> + if (odp_unlikely(scd(&llq->u.ui, neu.ui, __ATOMIC_RELEASE))) {
> +#else
> + if (odp_unlikely(!__atomic_compare_exchange(&llq->u,
> + &old,
> + &neu,
> + false, /* weak= */
> + __ATOMIC_RELEASE,
> + __ATOMIC_RELAXED))) {
> +#endif
> + odp_cpu_pause();
> + goto retry;
> + }
> + if (old.st.tail != NULL) {
> + /* List was not empty */
> + ODP_ASSERT(old.st.tail->next == SENTINEL);
> + old.st.tail->next = node;
> + }
> +}
> +
> +#else
> +
> +static inline void llq_enqueue(struct llqueue *llq, struct llnode *node)
> +{
> + ODP_ASSERT(node->next == NULL);
> + node->next = SENTINEL;
> +
> + odp_spinlock_lock(&llq->lock);
> + if (llq->u.st.head == NULL) {
> + llq->u.st.head = llq->u.st.tail = node;
> + } else {
> + llq->u.st.tail->next = node;
> + llq->u.st.tail = node;
> + }
> + odp_spinlock_unlock(&llq->lock);
> +}
> +
> +#endif
> +
> +#ifndef ODP_CONFIG_LLQ_LOCK
> +
> +static inline struct llnode *llq_dequeue(struct llqueue *llq)
> +{
> + struct llnode *head;
> + union llht old;
> +
> + /* llq_dequeue() may be used in a busy-waiting fashion
> + * Read head using plain load to avoid disturbing remote LL/SC
> + */
> + head = __atomic_load_n(&llq->u.st.head, __ATOMIC_ACQUIRE);
> + if (head == NULL)
> + return NULL;
> + /* Read head->next before LL to minimize cache miss latency
> + * in LL/SC below
> + */
> + (void)__atomic_load_n(&head->next, __ATOMIC_RELAXED);
> +
> +#ifdef ODP_CONFIG_USE_LLSC
> + /* Prefetch for store for faster LL/SC execution */
> + __builtin_prefetch(&llq->u.ui, 1, 0);
> +retry: /* Failed SC requires new LL */
> + old.ui = lld(&llq->u.ui, __ATOMIC_RELAXED);
> +#else
> + __atomic_load(&llq->u, &old, __ATOMIC_RELAXED);
> +retry: /* Failed CAS returns existing value */
> +#endif
> + if (odp_unlikely(old.st.head == NULL)) {
> + /* Empty list */
> + return NULL;
> + } else if (odp_unlikely(old.st.head == old.st.tail)) {
> + /* Single-element in list */
> + union llht neu;
> +
> + neu.st.head = NULL;
> + neu.st.tail = NULL;
> +#ifdef ODP_CONFIG_USE_LLSC
> + if (odp_unlikely(scd(&llq->u.ui, neu.ui, __ATOMIC_RELAXED))) {
> +#else
> + if (odp_unlikely(!__atomic_compare_exchange(
> + &llq->u,
> + &old,
> + &neu,
> + false, /* weak= */
> + __ATOMIC_RELAXED,
> + __ATOMIC_RELAXED))) {
> +#endif
> + /* Failed */
> + odp_cpu_pause();
> + goto retry;
> + }
> + ODP_ASSERT(old.st.head->next == SENTINEL);
> + } else {
> + /* Multi-element list, dequeue head */
> + struct llnode *next;
> + union llht neu;
> + /* Wait until llq_enqueue() has written true next pointer */
> + while ((next = __atomic_load_n(&old.st.head->next,
> + __ATOMIC_RELAXED)) == SENTINEL)
> + odp_cpu_pause();
> + neu.st.head = next;
> + neu.st.tail = old.st.tail;
> +#ifdef ODP_CONFIG_USE_LLSC
> + if (odp_unlikely(scd(&llq->u.ui, neu.ui, __ATOMIC_RELAXED))) {
> +#else
> + if (odp_unlikely(!__atomic_compare_exchange(
> + &llq->u,
> + &old,
> + &neu,
> + false, /* weak= */
> + __ATOMIC_RELAXED,
> + __ATOMIC_RELAXED))) {
> +#endif
> + /* Failed */
> + odp_cpu_pause();
> + goto retry;
> + }
> + ODP_ASSERT(old.st.head->next != SENTINEL);
> + }
> + old.st.head->next = NULL;
> + return old.st.head;
> +}
> +
> +#else
> +
> +static inline struct llnode *llq_dequeue(struct llqueue *llq)
> +{
> + struct llnode *head;
> + struct llnode *node = NULL;
> +
> + head = __atomic_load_n(&llq->u.st.head, __ATOMIC_RELAXED);
> + if (head == NULL)
> + return NULL;
> +
> + odp_spinlock_lock(&llq->lock);
> + if (llq->u.st.head != NULL) {
> + node = llq->u.st.head;
> + if (llq->u.st.head == llq->u.st.tail) {
> + ODP_ASSERT(node->next == SENTINEL);
> + llq->u.st.head = llq->u.st.tail = NULL;
> + } else {
> + ODP_ASSERT(node->next != SENTINEL);
> + llq->u.st.head = node->next;
> + }
> + node->next = NULL;
> + }
> + odp_spinlock_unlock(&llq->lock);
> + return node;
> +}
> +
> +#endif
> +
> +#ifndef ODP_CONFIG_LLQ_LOCK
> +
> +static inline odp_bool_t llq_dequeue_cond(struct llqueue *llq,
> + struct llnode *exp)
> +{
> + union llht old;
> +
> +#ifdef ODP_CONFIG_USE_LLSC
> + /* Prefetch for store for faster LL/SC execution */
> + __builtin_prefetch(&llq->u.ui, 1, 0);
> +retry: /* Failed SC requires new LL */
> + old.ui = lld(&llq->u.ui, __ATOMIC_RELAXED);
> +#else
> + __atomic_load(&llq->u, &old, __ATOMIC_RELAXED);
> +retry: /* Failed CAS returns existing value */
> +#endif
> + if (odp_unlikely(old.st.head == NULL || old.st.head != exp)) {
> + /* Empty list or wrong head */
> + return false;
> + } else if (odp_unlikely(old.st.head == old.st.tail)) {
> + /* Single-element in list */
> + union llht neu;
> +
> + neu.st.head = NULL;
> + neu.st.tail = NULL;
> +#ifdef ODP_CONFIG_USE_LLSC
> + if (odp_unlikely(scd(&llq->u.ui, neu.ui, __ATOMIC_RELAXED))) {
> +#else
> + if (odp_unlikely(!__atomic_compare_exchange(
> + &llq->u,
> + &old,
> + &neu,
> + false, /* weak= */
> + __ATOMIC_RELAXED,
> + __ATOMIC_RELAXED))) {
> +#endif
> + /* Failed */
> + odp_cpu_pause();
> + goto retry;
> + }
> + ODP_ASSERT(old.st.head->next == SENTINEL);
> + } else {
> + /* Multi-element list, dequeue head */
> + struct llnode *next;
> + union llht neu;
> +
> + /* Wait until llq_enqueue() has written true next pointer */
> + while ((next = __atomic_load_n(&old.st.head->next,
> + __ATOMIC_RELAXED)) == SENTINEL)
> + odp_cpu_pause();
> +
> + neu.st.head = next;
> + neu.st.tail = old.st.tail;
> +#ifdef ODP_CONFIG_USE_LLSC
> + if (odp_unlikely(scd(&llq->u.ui, neu.ui, __ATOMIC_RELAXED))) {
> +#else
> + if (odp_unlikely(!__atomic_compare_exchange(
> + &llq->u,
> + &old,
> + &neu,
> + false, /* weak= */
> + __ATOMIC_RELAXED,
> + __ATOMIC_RELAXED))) {
> +#endif
> + /* Failed */
> + odp_cpu_pause();
> + goto retry;
> + }
> + ODP_ASSERT(old.st.head->next != SENTINEL);
> + }
> + old.st.head->next = NULL;
> + return true;
> +}
> +
> +#else
> +
> +static inline odp_bool_t llq_dequeue_cond(struct llqueue *llq,
> + struct llnode *node)
> +{
> + odp_bool_t success = false;
> +
> + odp_spinlock_lock(&llq->lock);
> + if (odp_likely(llq->u.st.head != NULL && llq->u.st.head == node)) {
> + success = true;
> + if (llq->u.st.head == llq->u.st.tail) {
> + ODP_ASSERT(node->next == SENTINEL);
> + llq->u.st.head = llq->u.st.tail = NULL;
> + } else {
> + ODP_ASSERT(node->next != SENTINEL);
> + llq->u.st.head = node->next;
> + }
> + node->next = NULL;
> + }
> + odp_spinlock_unlock(&llq->lock);
> + return success;
> +}
> +
> +#endif
> +
> +#ifndef ODP_CONFIG_LLQ_LOCK
> +
> +/* If 'node' is a head of llq then move it to tail */
> +static inline odp_bool_t llq_cond_rotate(struct llqueue *llq,
> + struct llnode *node)
> +{
> + /* Difficult to make this into a single atomic operation
> + * Instead use existing primitives.
> + */
> + if (odp_likely(llq_dequeue_cond(llq, node))) {
> + llq_enqueue(llq, node);
> + return true;
> + }
> + return false;
> +}
> +
> +#else
> +
> +/* If 'node' is a head of llq then move it to tail */
> +static inline odp_bool_t llq_cond_rotate(struct llqueue *llq,
> + struct llnode *node)
> +{
> + odp_bool_t success = false;
> +
> + odp_spinlock_lock(&llq->lock);
> + if (odp_likely(llq->u.st.head == node)) {
> + success = true;
> + if (llq->u.st.tail != node) {
> + ODP_ASSERT(node->next != SENTINEL);
> + llq->u.st.head = node->next;
> + llq->u.st.tail->next = node;
> + llq->u.st.tail = node;
> + node->next = SENTINEL;
> + }
> + /* Else 'node' is only element on list => nothing to do */
> + }
> + odp_spinlock_unlock(&llq->lock);
> + return success;
> +}
> +
> +#endif
> +
> +#endif
> diff --git a/platform/linux-generic/include/odp_llsc.h b/platform/linux-generic/include/odp_llsc.h
> new file mode 100644
> index 00000000..a7b6166e
> --- /dev/null
> +++ b/platform/linux-generic/include/odp_llsc.h
> @@ -0,0 +1,325 @@
> +/* Copyright (c) 2017, ARM Limited
> + * All rights reserved.
> + *
> + * SPDX-License-Identifier: BSD-3-Clause
> + */
> +
> +#ifndef ODP_LLSC_H_
> +#define ODP_LLSC_H_
> +
> +#include <odp_config_internal.h>
> +
> +/******************************************************************************
> + * LL/SC primitives
> + *****************************************************************************/
> +
> +#if defined __ARM_ARCH
> +#if __ARM_ARCH == 7 || (__ARM_ARCH == 8 && __ARM_64BIT_STATE == 0)
> +static inline void dmb(void)
> +{
> + __asm volatile("dmb" : : : "memory");
> +}
> +
> +static inline uint32_t ll8(uint8_t *var, int mm)
> +{
> + uint8_t old;
> +
> + __asm volatile("ldrexb %0, [%1]"
> + : "=&r" (old)
> + : "r" (var)
> + : );
> + /* Barrier after an acquiring load */
> + if (mm == __ATOMIC_ACQUIRE)
> + dmb();
> + return old;
> +}
> +
> +static inline uint32_t ll16(uint16_t *var, int mm)
> +{
> + uint16_t old;
> +
> + __asm volatile("ldrexh %0, [%1]"
> + : "=&r" (old)
> + : "r" (var)
> + : );
> + /* Barrier after an acquiring load */
> + if (mm == __ATOMIC_ACQUIRE)
> + dmb();
> + return old;
> +}
> +
> +static inline uint32_t ll(uint32_t *var, int mm)
> +{
> + uint32_t old;
> +
> + __asm volatile("ldrex %0, [%1]"
> + : "=&r" (old)
> + : "r" (var)
> + : );
> + /* Barrier after an acquiring load */
> + if (mm == __ATOMIC_ACQUIRE)
> + dmb();
> + return old;
> +}
> +#define ll32(a, b) ll((a), (b))
> +
> +/* Return 0 on success, 1 on failure */
> +static inline uint32_t sc(uint32_t *var, uint32_t neu, int mm)
> +{
> + uint32_t ret;
> +
> + /* Barrier before a releasing store */
> + if (mm == __ATOMIC_RELEASE)
> + dmb();
> + __asm volatile("strex %0, %1, [%2]"
> + : "=&r" (ret)
> + : "r" (neu), "r" (var)
> + : );
> + return ret;
> +}
> +#define sc32(a, b, c) sc((a), (b), (c))
> +
> +static inline uint64_t lld(uint64_t *var, int mm)
> +{
> + uint64_t old;
> +
> + __asm volatile("ldrexd %0, %H0, [%1]"
> + : "=&r" (old)
> + : "r" (var)
> + : );
> + /* Barrier after an acquiring load */
> + if (mm == __ATOMIC_ACQUIRE)
> + dmb();
> + return old;
> +}
> +#define ll64(a, b) lld((a), (b))
> +
> +/* Return 0 on success, 1 on failure */
> +static inline uint32_t scd(uint64_t *var, uint64_t neu, int mm)
> +{
> + uint32_t ret;
> +
> + /* Barrier before a releasing store */
> + if (mm == __ATOMIC_RELEASE)
> + dmb();
> + __asm volatile("strexd %0, %1, %H1, [%2]"
> + : "=&r" (ret)
> + : "r" (neu), "r" (var)
> + : );
> + return ret;
> +}
> +#define sc64(a, b, c) scd((a), (b), (c))
> +
> +#endif
> +
> +#if __ARM_ARCH == 8 && __ARM_64BIT_STATE == 1
> +static inline uint16_t ll8(uint8_t *var, int mm)
> +{
> + uint16_t old;
> +
> + if (mm == __ATOMIC_ACQUIRE)
> + __asm volatile("ldaxrb %w0, [%1]"
> + : "=&r" (old)
> + : "r" (var)
> + : "memory");
> + else if (mm == __ATOMIC_RELAXED)
> + __asm volatile("ldxrb %w0, [%1]"
> + : "=&r" (old)
> + : "r" (var)
> + : );
> + else
> + ODP_ABORT();
> + return old;
> +}
> +
> +static inline uint16_t ll16(uint16_t *var, int mm)
> +{
> + uint16_t old;
> +
> + if (mm == __ATOMIC_ACQUIRE)
> + __asm volatile("ldaxrh %w0, [%1]"
> + : "=&r" (old)
> + : "r" (var)
> + : "memory");
> + else if (mm == __ATOMIC_RELAXED)
> + __asm volatile("ldxrh %w0, [%1]"
> + : "=&r" (old)
> + : "r" (var)
> + : );
> + else
> + ODP_ABORT();
> + return old;
> +}
> +
> +static inline uint32_t ll32(uint32_t *var, int mm)
> +{
> + uint32_t old;
> +
> + if (mm == __ATOMIC_ACQUIRE)
> + __asm volatile("ldaxr %w0, [%1]"
> + : "=&r" (old)
> + : "r" (var)
> + : "memory");
> + else if (mm == __ATOMIC_RELAXED)
> + __asm volatile("ldxr %w0, [%1]"
> + : "=&r" (old)
> + : "r" (var)
> + : );
> + else
> + ODP_ABORT();
> + return old;
> +}
> +
> +/* Return 0 on success, 1 on failure */
> +static inline uint32_t sc32(uint32_t *var, uint32_t neu, int mm)
> +{
> + uint32_t ret;
> +
> + if (mm == __ATOMIC_RELEASE)
> + __asm volatile("stlxr %w0, %w1, [%2]"
> + : "=&r" (ret)
> + : "r" (neu), "r" (var)
> + : "memory");
> + else if (mm == __ATOMIC_RELAXED)
> + __asm volatile("stxr %w0, %w1, [%2]"
> + : "=&r" (ret)
> + : "r" (neu), "r" (var)
> + : );
> + else
> + ODP_ABORT();
> + return ret;
> +}
> +
> +static inline uint64_t ll(uint64_t *var, int mm)
> +{
> + uint64_t old;
> +
> + if (mm == __ATOMIC_ACQUIRE)
> + __asm volatile("ldaxr %0, [%1]"
> + : "=&r" (old)
> + : "r" (var)
> + : "memory");
> + else if (mm == __ATOMIC_RELAXED)
> + __asm volatile("ldxr %0, [%1]"
> + : "=&r" (old)
> + : "r" (var)
> + : );
> + else
> + ODP_ABORT();
> + return old;
> +}
> +#define ll64(a, b) ll((a), (b))
> +
> +/* Return 0 on success, 1 on failure */
> +static inline uint32_t sc(uint64_t *var, uint64_t neu, int mm)
> +{
> + uint32_t ret;
> +
> + if (mm == __ATOMIC_RELEASE)
> + __asm volatile("stlxr %w0, %1, [%2]"
> + : "=&r" (ret)
> + : "r" (neu), "r" (var)
> + : "memory");
> + else if (mm == __ATOMIC_RELAXED)
> + __asm volatile("stxr %w0, %1, [%2]"
> + : "=&r" (ret)
> + : "r" (neu), "r" (var)
> + : );
> + else
> + ODP_ABORT();
> + return ret;
> +}
> +#define sc64(a, b, c) sc((a), (b), (c))
> +static inline __int128 lld(__int128 *var, int mm)
> +{
> + __int128 old;
> +
> + if (mm == __ATOMIC_ACQUIRE)
> + __asm volatile("ldaxp %0, %H0, [%1]"
> + : "=&r" (old)
> + : "r" (var)
> + : "memory");
> + else if (mm == __ATOMIC_RELAXED)
> + __asm volatile("ldxp %0, %H0, [%1]"
> + : "=&r" (old)
> + : "r" (var)
> + : );
> + else
> + ODP_ABORT();
> + return old;
> +}
> +
> +/* Return 0 on success, 1 on failure */
> +static inline uint32_t scd(__int128 *var, __int128 neu, int mm)
> +{
> + uint32_t ret;
> +
> + if (mm == __ATOMIC_RELEASE)
> + __asm volatile("stlxp %w0, %1, %H1, [%2]"
> + : "=&r" (ret)
> + : "r" (neu), "r" (var)
> + : "memory");
> + else if (mm == __ATOMIC_RELAXED)
> + __asm volatile("stxp %w0, %1, %H1, [%2]"
> + : "=&r" (ret)
> + : "r" (neu), "r" (var)
> + : );
> + else
> + ODP_ABORT();
> + return ret;
> +}
> +#endif
> +#endif
> +
> +static inline void sevl(void)
> +{
> +#if defined __ARM_ARCH
> + __asm volatile("sevl" : : : );
> +#endif
> +}
> +
> +static inline void sev(void)
> +{
> +#if defined __ARM_ARCH
> + __asm volatile("sev" : : : "memory");
> +#endif
> +}
> +
> +static inline int wfe(void)
> +{
> +#if defined __ARM_ARCH
> + __asm volatile("wfe" : : : "memory");
> +#endif
> + return 1;
> +}
> +
> +#ifdef ODP_CONFIG_USE_WFE
> +#define SEVL() sevl()
> +#define WFE() wfe()
> +#define SEV() do { __asm volatile("dsb ish" ::: "memory"); sev(); } while (0)
> +#if defined __ARM_ARCH && __ARM_ARCH == 8 && __ARM_64BIT_STATE == 1
> +#define LDXR128(addr, mo) lld((addr), (mo))
> +#endif
> +#define LDXR64(addr, mo) ll64((addr), (mo))
> +#define LDXR32(addr, mo) ll32((addr), (mo))
> +#define LDXR16(addr, mo) ll16((addr), (mo))
> +#define LDXR8(addr, mo) ll8((addr), (mo))
> +/* When using WFE do not stall the pipeline using other means */
> +#define DOZE() (void)0
> +#else
> +#define SEVL() (void)0
> +#define WFE() 1
> +#define SEV() (void)0
> +#define LDXR128(addr, mo) __atomic_load_n((addr), (mo))
> +#define LDXR64(addr, mo) __atomic_load_n((addr), (mo))
> +#define LDXR32(addr, mo) __atomic_load_n((addr), (mo))
> +#define LDXR16(addr, mo) __atomic_load_n((addr), (mo))
> +#define LDXR8(addr, mo) __atomic_load_n((addr), (mo))
> +#if defined __ARM_ARCH
> +#define DOZE() __asm volatile("isb" ::: "memory")
> +#else
> +#define DOZE() __asm volatile("pause" ::: "memory")
> +#endif
> +#endif
> +
> +#endif
> diff --git a/platform/linux-generic/include/odp_packet_internal.h b/platform/linux-generic/include/odp_packet_internal.h
> index 0a9f1779..c9d14766 100644
> --- a/platform/linux-generic/include/odp_packet_internal.h
> +++ b/platform/linux-generic/include/odp_packet_internal.h
> @@ -235,6 +235,9 @@ odp_buffer_t _odp_packet_to_buffer(odp_packet_t pkt);
> /* Convert a buffer handle to a packet handle */
> odp_packet_t _odp_packet_from_buffer(odp_buffer_t buf);
>
> +/* Convert a packet handle to a buffer hdr handle pointer */
> +odp_buffer_hdr_t *_odp_packet_to_buf_hdr_ptr(odp_packet_t pkt);
> +
> static inline int packet_hdr_has_l2(odp_packet_hdr_t *pkt_hdr)
> {
> return pkt_hdr->p.input_flags.l2;
> diff --git a/platform/linux-generic/include/odp_queue_internal.h b/platform/linux-generic/include/odp_queue_internal.h
> index 560f826e..ebfc0f76 100644
> --- a/platform/linux-generic/include/odp_queue_internal.h
> +++ b/platform/linux-generic/include/odp_queue_internal.h
> @@ -19,16 +19,24 @@ extern "C" {
> #endif
>
> #include <odp/api/queue.h>
> -#include <odp_forward_typedefs_internal.h>
> -#include <odp_schedule_if.h>
> -#include <odp_buffer_internal.h>
> -#include <odp_align_internal.h>
> +#include <odp/api/std_types.h>
> +#include <odp/api/buffer.h>
> #include <odp/api/packet_io.h>
> #include <odp/api/align.h>
> #include <odp/api/hints.h>
> #include <odp/api/ticketlock.h>
> +
> #include <odp_config_internal.h>
>
> +#include <odp_align_internal.h>
> +#include <odp_buffer_internal.h>
> +#include <odp_forward_typedefs_internal.h>
> +#ifdef ODP_SCHEDULE_SCALABLE
> +#include <odp_llsc.h>
> +#include <odp_schedule_ordered_internal.h>
> +#endif
> +#include <odp_schedule_if.h>
> +
> #define QUEUE_MULTI_MAX CONFIG_BURST_SIZE
>
> #define QUEUE_STATUS_FREE 0
> @@ -37,8 +45,6 @@ extern "C" {
> #define QUEUE_STATUS_NOTSCHED 3
> #define QUEUE_STATUS_SCHED 4
>
> -
> -/* forward declaration */
> union queue_entry_u;
>
> typedef int (*enq_func_t)(union queue_entry_u *, odp_buffer_hdr_t *);
> @@ -49,6 +55,37 @@ typedef int (*enq_multi_func_t)(union queue_entry_u *,
> typedef int (*deq_multi_func_t)(union queue_entry_u *,
> odp_buffer_hdr_t **, int);
>
> +#ifdef ODP_SCHEDULE_SCALABLE
> +#define BUFFER_HDR_INVALID ((odp_buffer_hdr_t *)ODP_EVENT_INVALID)
> +
> +struct queue_entry_s {
> + sched_elem_t sched_elem;
> + odp_shm_t shm;
> + odp_shm_t rwin_shm;
> +
> + odp_ticketlock_t lock ODP_ALIGNED_CACHE;
> + int status;
> +
> + enq_func_t enqueue ODP_ALIGNED_CACHE;
> + deq_func_t dequeue;
> + enq_multi_func_t enqueue_multi;
> + deq_multi_func_t dequeue_multi;
> +
> + uint32_t index;
> + odp_queue_t handle;
> + odp_queue_type_t type;
> + odp_queue_param_t param;
> + odp_pktin_queue_t pktin;
> + odp_pktout_queue_t pktout;
> + char name[ODP_QUEUE_NAME_LEN];
> +};
> +
> +int _odp_queue_deq(sched_elem_t *q, odp_event_t *evp, int num);
> +int _odp_queue_deq_sc(sched_elem_t *q, odp_event_t *evp, int num);
> +
> +#else
> +#define BUFFER_HDR_INVALID NULL
> +
> struct queue_entry_s {
> odp_ticketlock_t lock ODP_ALIGNED_CACHE;
>
> @@ -77,6 +114,8 @@ struct queue_entry_s {
> char name[ODP_QUEUE_NAME_LEN];
> };
>
> +#endif
> +
> union queue_entry_u {
> struct queue_entry_s s;
> uint8_t pad[ROUNDUP_CACHE_LINE(sizeof(struct queue_entry_s))];
> @@ -94,6 +133,12 @@ int queue_deq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[], int num);
> void queue_lock(queue_entry_t *queue);
> void queue_unlock(queue_entry_t *queue);
>
> +int queue_pktout_enq(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr);
> +int queue_pktout_enq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[],
> + int num);
> +
> +int queue_tm_reorder(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr);
> +
> static inline uint32_t queue_to_id(odp_queue_t handle)
> {
> return _odp_typeval(handle) - 1;
> @@ -107,6 +152,71 @@ static inline queue_entry_t *queue_to_qentry(odp_queue_t handle)
> return get_qentry(queue_id);
> }
>
> +static inline odp_queue_t queue_get_handle(queue_entry_t *queue)
> +{
> + return queue->s.handle;
> +}
> +
> +static inline odp_pktout_queue_t queue_get_pktout(queue_entry_t *queue)
> +{
> + return queue->s.pktout;
> +}
> +
> +static inline void queue_set_pktout(queue_entry_t *queue,
> + odp_pktio_t pktio,
> + int index)
> +{
> + queue->s.pktout.pktio = pktio;
> + queue->s.pktout.index = index;
> +}
> +
> +static inline odp_pktin_queue_t queue_get_pktin(queue_entry_t *queue)
> +{
> + return queue->s.pktin;
> +}
> +
> +static inline void queue_set_pktin(queue_entry_t *queue,
> + odp_pktio_t pktio,
> + int index)
> +{
> + queue->s.pktin.pktio = pktio;
> + queue->s.pktin.index = index;
> +}
> +
> +static inline void queue_set_enq_func(queue_entry_t *queue, enq_func_t func)
> +{
> + queue->s.enqueue = func;
> +}
> +
> +static inline void queue_set_enq_multi_func(queue_entry_t *queue,
> + enq_multi_func_t func)
> +{
> + queue->s.enqueue_multi = func;
> +}
> +
> +static inline void queue_set_deq_func(queue_entry_t *queue, deq_func_t func)
> +{
> + queue->s.dequeue = func;
> +}
> +
> +static inline void queue_set_deq_multi_func(queue_entry_t *queue,
> + deq_multi_func_t func)
> +{
> + queue->s.dequeue_multi = func;
> +}
> +
> +static inline void queue_set_type(queue_entry_t *queue, odp_queue_type_t type)
> +{
> + queue->s.type = type;
> +}
> +
> +#ifdef ODP_SCHEDULE_SCALABLE
> +static inline reorder_window_t *queue_get_rwin(queue_entry_t *queue)
> +{
> + return queue->s.sched_elem.rwin;
> +}
> +#endif
> +
> #ifdef __cplusplus
> }
> #endif
> diff --git a/platform/linux-generic/include/odp_schedule_if.h b/platform/linux-generic/include/odp_schedule_if.h
> index 530d157f..41998833 100644
> --- a/platform/linux-generic/include/odp_schedule_if.h
> +++ b/platform/linux-generic/include/odp_schedule_if.h
> @@ -4,6 +4,12 @@
> * SPDX-License-Identifier: BSD-3-Clause
> */
>
> +/* Copyright (c) 2017, ARM Limited
> + * All rights reserved.
> + *
> + * SPDX-License-Identifier: BSD-3-Clause
> + */
> +
> #ifndef ODP_SCHEDULE_IF_H_
> #define ODP_SCHEDULE_IF_H_
>
> @@ -11,18 +17,168 @@
> extern "C" {
> #endif
>
> +#include <odp/api/align.h>
> #include <odp/api/queue.h>
> -#include <odp_queue_internal.h>
> #include <odp/api/schedule.h>
> +#include <odp/api/ticketlock.h>
> +
> +#include <odp_forward_typedefs_internal.h>
> +#ifdef ODP_SCHEDULE_SCALABLE
> +#include <odp_config_internal.h>
> +#include <odp_llqueue.h>
> +#include <odp_schedule_ordered_internal.h>
> +
> +#include <limits.h>
> +#endif
> +
> +/* Number of ordered locks per queue */
> +#define SCHEDULE_ORDERED_LOCKS_PER_QUEUE 2
> +
> +#ifdef ODP_SCHEDULE_SCALABLE
> +
> +typedef struct {
> + union {
> + struct {
> + struct llqueue llq;
> + uint32_t prio;
> + };
> + char line[ODP_CACHE_LINE_SIZE];
> + };
> +} sched_queue_t ODP_ALIGNED_CACHE;
> +
> +#define TICKET_INVALID (uint16_t)(~0U)
> +
> +typedef struct {
> + int32_t numevts;
> + uint16_t wrr_budget;
> + uint8_t cur_ticket;
> + uint8_t nxt_ticket;
> +} qschedstate_t ODP_ALIGNED(sizeof(uint64_t));
> +
> +typedef uint32_t ringidx_t;
> +
> +#ifdef ODP_CONFIG_USE_SPLIT_PRODCONS
> +#define SPLIT_PC ODP_ALIGNED_CACHE
> +#else
> +#define SPLIT_PC
> +#endif
> +
> +#ifdef ODP_CONFIG_USE_SPLIT_READWRITE
> +#define SPLIT_RW ODP_ALIGNED_CACHE
> +#else
> +#define SPLIT_RW
> +#endif
> +
> +#define ODP_NO_SCHED_QUEUE (ODP_SCHED_SYNC_ORDERED + 1)
> +
> +typedef struct {
> + /* (ll)node must be first */
> + struct llnode node; /* 8: 0.. 7 */
> + sched_queue_t *schedq; /* 8: 8..15 */
> +#ifdef CONFIG_QSCHST_LOCK
> + odp_ticketlock_t qschlock;
> +#endif
> + qschedstate_t qschst; /* 8: 16..23 */
> + uint16_t pop_deficit; /* 2: 24..25 */
> + uint16_t qschst_type; /* 2: 26..27 */
> +
> + ringidx_t prod_read SPLIT_PC; /* 4: 28..31 */
> +
> + ringidx_t prod_write SPLIT_RW;/* 4: 32..35 */
> + ringidx_t prod_mask; /* 4: 36..39 */
> + odp_buffer_hdr_t **prod_ring; /* 8: 40..47 */
> +
> + ringidx_t cons_write SPLIT_PC;/* 4: 48..51 */
> +
> + ringidx_t cons_read SPLIT_RW; /* 4: 52..55 */
> +
> + reorder_window_t *rwin; /* 8: 56..63 */
> + void *user_ctx; /* 8: 64..71 */
> +#ifdef ODP_CONFIG_USE_SPLIT_PRODCONS
> + odp_buffer_hdr_t **cons_ring;
> + ringidx_t cons_mask;
> + uint16_t cons_type;
> +#else
> +#define cons_mask prod_mask
> +#define cons_ring prod_ring
> +#define cons_type qschst_type
> +#endif
> +} sched_elem_t ODP_ALIGNED_CACHE;
> +
> +/*
> + * Scheduler group related declarations.
> + */
> +typedef bitset_t sched_group_mask_t;
> +
> +/* Number of scheduling groups */
> +#define MAX_SCHED_GROUP (sizeof(sched_group_mask_t) * CHAR_BIT)
> +
> +typedef struct {
> + /* Threads currently associated with the sched group */
> + bitset_t thr_actual[ODP_SCHED_PRIO_NUM] ODP_ALIGNED_CACHE;
> + bitset_t thr_wanted;
> + /* Used to spread queues over schedq's */
> + uint32_t xcount[ODP_SCHED_PRIO_NUM];
> + /* Number of schedq's per prio */
> + uint32_t xfactor;
> +
> + char name[ODP_SCHED_GROUP_NAME_LEN];
> + /* shm handle for the group */
> + odp_shm_t shm;
> +
> + /* ODP_SCHED_PRIO_NUM * xfactor.
> + * Must be the last memember in this struct.
> + */
> + sched_queue_t schedq[1] ODP_ALIGNED_CACHE;
> +} sched_group_t;
> +
> +/*
> + * Per thread state
> + */
> +/* Number of reorder contexts per thread */
> +#define TS_RVEC_SIZE 16
> +typedef struct {
> + /* Atomic queue currently being processed or NULL */
> + sched_elem_t *atomq;
> + /* Current reorder context or NULL */
> + reorder_context_t *rctx;
> + uint8_t pause;
> + uint8_t out_of_order;
> + uint8_t tidx;
> + uint8_t pad;
> + uint32_t dequeued; /* Number of events dequeued from atomic queue */
> + uint16_t pktin_next;/* Next pktin tag to poll */
> + uint16_t pktin_poll_cnts;
> + uint16_t ticket; /* Ticket for atomic queue or TICKET_INVALID */
> + uint16_t num_schedq;
> + uint16_t sg_sem; /* Set when sg_wanted is modified by other thread */
> +#define SCHEDQ_PER_THREAD (MAX_SCHED_GROUP * ODP_SCHED_PRIO_NUM)
> + sched_queue_t *schedq_list[SCHEDQ_PER_THREAD];
> + /* Current sched_group membership */
> + sched_group_mask_t sg_actual[ODP_SCHED_PRIO_NUM];
> + /* Future sched_group membership. */
> + sched_group_mask_t sg_wanted[ODP_SCHED_PRIO_NUM];
> + bitset_t priv_rvec_free;
> + /* Bitset of free entries in rvec[] */
> + bitset_t rvec_free ODP_ALIGNED_CACHE;
> + /* Reordering contexts to allocate from */
> + reorder_context_t rvec[TS_RVEC_SIZE] ODP_ALIGNED_CACHE;
> +} sched_scalable_thread_state_t ODP_ALIGNED_CACHE;
> +
> +void sched_update_enq(sched_elem_t *q, uint32_t actual);
> +void sched_update_enq_sp(sched_elem_t *q, uint32_t actual);
> +sched_queue_t *schedq_from_sched_group(odp_schedule_group_t grp, uint32_t prio);
> +void sched_group_xcount_dec(odp_schedule_group_t grp, uint32_t prio);
> +
> +#endif /* ODP_SCHEDULE_SCALABLE */
>
> typedef void (*schedule_pktio_start_fn_t)(int pktio_index, int num_in_queue,
> int in_queue_idx[]);
> typedef int (*schedule_thr_add_fn_t)(odp_schedule_group_t group, int thr);
> typedef int (*schedule_thr_rem_fn_t)(odp_schedule_group_t group, int thr);
> typedef int (*schedule_num_grps_fn_t)(void);
> -typedef int (*schedule_init_queue_fn_t)(uint32_t queue_index,
> - const odp_schedule_param_t *sched_param
> - );
> +typedef int (*schedule_init_queue_fn_t)(
> + uint32_t queue_index, const odp_schedule_param_t *sched_param);
> typedef void (*schedule_destroy_queue_fn_t)(uint32_t queue_index);
> typedef int (*schedule_sched_queue_fn_t)(uint32_t queue_index);
> typedef int (*schedule_unsched_queue_fn_t)(uint32_t queue_index);
> @@ -64,6 +220,7 @@ extern const schedule_fn_t *sched_fn;
> int sched_cb_pktin_poll(int pktio_index, int num_queue, int index[]);
> void sched_cb_pktio_stop_finalize(int pktio_index);
> int sched_cb_num_pktio(void);
> +#ifndef ODP_SCHEDULE_SCALABLE
> int sched_cb_num_queues(void);
> int sched_cb_queue_prio(uint32_t queue_index);
> int sched_cb_queue_grp(uint32_t queue_index);
> @@ -73,6 +230,7 @@ odp_queue_t sched_cb_queue_handle(uint32_t queue_index);
> void sched_cb_queue_destroy_finalize(uint32_t queue_index);
> int sched_cb_queue_deq_multi(uint32_t queue_index, odp_event_t ev[], int num);
> int sched_cb_queue_empty(uint32_t queue_index);
> +#endif
>
> /* API functions */
> typedef struct {
> diff --git a/platform/linux-generic/include/odp_schedule_ordered_internal.h b/platform/linux-generic/include/odp_schedule_ordered_internal.h
> new file mode 100644
> index 00000000..a16deca8
> --- /dev/null
> +++ b/platform/linux-generic/include/odp_schedule_ordered_internal.h
> @@ -0,0 +1,150 @@
> +/* Copyright (c) 2017, ARM Limited
> + * All rights reserved.
> + *
> + * SPDX-License-Identifier: BSD-3-Clause
> + */
> +
> +#ifndef ODP_SCHEDULE_ORDERED_INTERNAL_H_
> +#define ODP_SCHEDULE_ORDERED_INTERNAL_H_
> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
> +
> +#ifdef ODP_SCHEDULE_SCALABLE
> +
> +#include <odp/api/shared_memory.h>
> +
> +#include <odp_internal.h>
> +
> +#include <odp_align_internal.h>
> +#include <odp_bitset.h>
> +#include <odp_llsc.h>
> +
> +/* High level functioning of reordering
> + * Datastructures -
> + * Reorder Window - Every ordered queue is associated with a reorder window.
> + * Reorder window stores reorder contexts from threads that
> + * have completed processing out-of-order.
> + * Reorder Context - Reorder context consists of events that a thread
> + * wants to enqueue while processing a batch of events
> + * from an ordered queue.
> + *
> + * Algorithm -
> + * 1) Thread identifies the ordered queue.
> + * 2) It 'reserves a slot in the reorder window and dequeues the
> + * events' atomically. Atomicity is achieved by using a ticket-lock
> + * like design where the reorder window slot is the ticket.
> + * 3a) Upon order-release/next schedule call, the thread
> + * checks if it's slot (ticket) equals the head of the reorder window.
> + * If yes, enqueues the events to the destination queue till
> + * i) the reorder window is empty or
> + * ii) there is a gap in the reorder window
> + * If no, the reorder context is stored in the reorder window at
> + * the reserved slot.
> + * 3b) Upon the first enqueue, the thread checks if it's slot (ticket)
> + * equals the head of the reorder window.
> + * If yes, enqueues the events immediately to the destination queue
> + * If no, these (and subsequent) events are stored in the reorder context
> + * (in the application given order)
> + */
> +
> +/* Head and change indicator variables are used to synchronise between
> + * concurrent insert operations in the reorder window. A thread performing
> + * an in-order insertion must be notified about the newly inserted
> + * reorder contexts so that it doesn’t halt the retire process too early.
> + * A thread performing an out-of-order insertion must correspondingly
> + * notify the thread doing in-order insertion of the new waiting reorder
> + * context, which may need to be handled by that thread.
> + *
> + * Also, an out-of-order insertion may become an in-order insertion if the
> + * thread doing an in-order insertion completes before this thread completes.
> + * We need a point of synchronisation where this knowledge and potential state
> + * change can be transferred between threads.
> + */
> +typedef struct hc {
> + /* First missing context */
> + uint32_t head;
> + /* Change indicator */
> + uint32_t chgi;
> +} hc_t ODP_ALIGNED(sizeof(uint64_t));
> +
> +/* Number of reorder contects in the reorder window.
> + * Should be at least one per CPU.
> + */
> +#define RWIN_SIZE 32
> +ODP_STATIC_ASSERT(IS_POWER_TWO(RWIN_SIZE), "RWIN_SIZE is not a power of 2");
> +
> +#define NUM_OLOCKS 2
> +
> +typedef struct reorder_context reorder_context_t;
> +
> +typedef struct reorder_window {
> + /* head and change indicator */
> + hc_t hc;
> + uint32_t winmask;
> + uint32_t tail;
> + uint32_t turn;
> + uint32_t olock[NUM_OLOCKS];
> + uint16_t lock_count;
> + /* Reorder contexts in this window */
> + reorder_context_t *ring[RWIN_SIZE];
> +} reorder_window_t;
> +
> +/* Number of events that can be stored in a reorder context.
> + * This size is chosen so that there is no space left unused at the end
> + * of the last cache line (for 64b architectures and 64b handles).
> + */
> +#define RC_EVT_SIZE 18
> +
> +typedef struct reorder_context {
> + /* Reorder window to which this context belongs */
> + reorder_window_t *rwin;
> + /* Pointer to TS->rvec_free */
> + bitset_t *rvec_free;
> + /* Our slot number in the reorder window */
> + uint32_t sn;
> + uint8_t olock_flags;
> + /* Our index in thread_state rvec array */
> + uint8_t idx;
> + /* Use to link reorder contexts together */
> + uint8_t next_idx;
> + /* Current reorder context to save events in */
> + uint8_t cur_idx;
> + /* Number of events stored in this reorder context */
> + uint8_t numevts;
> + /* Events stored in this context */
> + odp_buffer_hdr_t *events[RC_EVT_SIZE];
> + queue_entry_t *destq[RC_EVT_SIZE];
> +} reorder_context_t ODP_ALIGNED_CACHE;
> +
> +reorder_window_t *rwin_alloc(int rwin_id, odp_shm_t *shm, unsigned lock_count);
> +bool rwin_reserve(reorder_window_t *rwin, uint32_t *sn);
> +void rwin_insert(reorder_window_t *rwin,
> + reorder_context_t *rctx,
> + uint32_t sn,
> + void (*callback)(reorder_context_t *));
> +void rctx_init(reorder_context_t *rctx, uint16_t idx,
> + reorder_window_t *rwin, uint32_t sn);
> +void rctx_free(const reorder_context_t *rctx);
> +void olock_unlock(const reorder_context_t *rctx, reorder_window_t *rwin,
> + uint32_t lock_index);
> +void olock_release(const reorder_context_t *rctx);
> +void rctx_retire(reorder_context_t *first);
> +void rctx_release(reorder_context_t *rctx);
> +
> +#else
> +
> +#define SUSTAIN_ORDER 1
> +
> +int schedule_ordered_queue_enq(uint32_t queue_index, void *p_buf_hdr,
> + int sustain, int *ret);
> +int schedule_ordered_queue_enq_multi(uint32_t queue_index, void *p_buf_hdr[],
> + int num, int sustain, int *ret);
> +#endif
> +
> +#ifdef __cplusplus
> +}
> +#endif
> +
> +#endif
> diff --git a/platform/linux-generic/m4/odp_schedule.m4 b/platform/linux-generic/m4/odp_schedule.m4
> index 91c19f21..d862b8b2 100644
> --- a/platform/linux-generic/m4/odp_schedule.m4
> +++ b/platform/linux-generic/m4/odp_schedule.m4
> @@ -1,13 +1,44 @@
> -AC_ARG_ENABLE([schedule-sp],
> - [ --enable-schedule-sp enable strict priority scheduler],
> - [if test x$enableval = xyes; then
> - schedule_sp_enabled=yes
> - ODP_CFLAGS="$ODP_CFLAGS -DODP_SCHEDULE_SP"
> - fi])
> +# Checks for --enable-schedule-sp and defines ODP_SCHEDULE_SP and adds
> +# -DODP_SCHEDULE_SP to CFLAGS.
> +AC_ARG_ENABLE(
> + [schedule_sp],
> + [AC_HELP_STRING([--enable-schedule-sp],
> + [enable strict priority scheduler])],
> + [if test "x$enableval" = xyes; then
> + schedule_sp=true
> + ODP_CFLAGS="$ODP_CFLAGS -DODP_SCHEDULE_SP"
> + else
> + schedule_sp=false
> + fi],
> + [schedule_sp=false])
> +AM_CONDITIONAL([ODP_SCHEDULE_SP], [test x$schedule_sp = xtrue])
>
> -AC_ARG_ENABLE([schedule-iquery],
> - [ --enable-schedule-iquery enable interests query (sparse bitmap) scheduler],
> - [if test x$enableval = xyes; then
> - schedule_iquery_enabled=yes
> - ODP_CFLAGS="$ODP_CFLAGS -DODP_SCHEDULE_IQUERY"
> - fi])
> +# Checks for --enable-schedule-iquery and defines ODP_SCHEDULE_IQUERY and adds
> +# -DODP_SCHEDULE_IQUERY to CFLAGS.
> +AC_ARG_ENABLE(
> + [schedule_iquery],
> + [AC_HELP_STRING([--enable-schedule-iquery],
> + [enable interests query (sparse bitmap) scheduler])],
> + [if test "x$enableval" = xyes; then
> + schedule_iquery=true
> + ODP_CFLAGS="$ODP_CFLAGS -DODP_SCHEDULE_IQUERY"
> + else
> + schedule_iquery=false
> + fi],
> + [schedule_iquery=false])
> +AM_CONDITIONAL([ODP_SCHEDULE_IQUERY], [test x$schedule_iquery = xtrue])
> +
> +# Checks for --enable-schedule-scalable and defines ODP_SCHEDULE_SCALABLE and
> +# adds -DODP_SCHEDULE_SCALABLE to CFLAGS.
> +AC_ARG_ENABLE(
> + [schedule_scalable],
> + [AC_HELP_STRING([--enable-schedule-scalable],
> + [enable scalable scheduler])],
> + [if test "x$enableval" = xyes; then
> + schedule_scalable=true
> + ODP_CFLAGS="$ODP_CFLAGS -DODP_SCHEDULE_SCALABLE"
> + else
> + schedule_scalable=false
> + fi],
> + [schedule_scalable=false])
> +AM_CONDITIONAL([ODP_SCHEDULE_SCALABLE], [test x$schedule_scalable = xtrue])
> diff --git a/platform/linux-generic/odp_classification.c b/platform/linux-generic/odp_classification.c
> index 5d96b00b..8fb5c32f 100644
> --- a/platform/linux-generic/odp_classification.c
> +++ b/platform/linux-generic/odp_classification.c
> @@ -282,7 +282,7 @@ odp_queue_t odp_cos_queue(odp_cos_t cos_id)
> if (!cos->s.queue)
> return ODP_QUEUE_INVALID;
>
> - return cos->s.queue->s.handle;
> + return queue_get_handle(cos->s.queue);
> }
>
> int odp_cos_drop_set(odp_cos_t cos_id, odp_cls_drop_t drop_policy)
> @@ -849,7 +849,7 @@ int cls_classify_packet(pktio_entry_t *entry, const uint8_t *base,
>
> *pool = cos->s.pool;
> pkt_hdr->p.input_flags.dst_queue = 1;
> - pkt_hdr->dst_queue = cos->s.queue->s.handle;
> + pkt_hdr->dst_queue = queue_get_handle(cos->s.queue);
>
> return 0;
> }
> diff --git a/platform/linux-generic/odp_packet.c b/platform/linux-generic/odp_packet.c
> index b8aac6bf..8ae35867 100644
> --- a/platform/linux-generic/odp_packet.c
> +++ b/platform/linux-generic/odp_packet.c
> @@ -651,6 +651,11 @@ odp_buffer_t _odp_packet_to_buffer(odp_packet_t pkt)
> return buffer_handle(packet_hdr(pkt));
> }
>
> +odp_buffer_hdr_t *_odp_packet_to_buf_hdr_ptr(odp_packet_t pkt)
> +{
> + return (odp_buffer_hdr_t *)(_odp_packet_to_buffer(pkt));
> +}
> +
> odp_packet_t odp_packet_from_event(odp_event_t ev)
> {
> if (odp_unlikely(ev == ODP_EVENT_INVALID))
> diff --git a/platform/linux-generic/odp_packet_io.c b/platform/linux-generic/odp_packet_io.c
> index 5e783d83..a267ee97 100644
> --- a/platform/linux-generic/odp_packet_io.c
> +++ b/platform/linux-generic/odp_packet_io.c
> @@ -5,23 +5,25 @@
> */
> #include <odp_posix_extensions.h>
>
> -#include <odp/api/packet_io.h>
> -#include <odp_packet_io_internal.h>
> -#include <odp_packet_io_queue.h>
> #include <odp/api/packet.h>
> -#include <odp_packet_internal.h>
> -#include <odp_internal.h>
> +#include <odp/api/packet_io.h>
> #include <odp/api/spinlock.h>
> #include <odp/api/ticketlock.h>
> #include <odp/api/shared_memory.h>
> -#include <odp_packet_socket.h>
> +#include <odp/api/time.h>
> +
> +#include <odp_internal.h>
> #include <odp_config_internal.h>
> -#include <odp_queue_internal.h>
> -#include <odp_schedule_if.h>
> -#include <odp_classification_internal.h>
> #include <odp_debug_internal.h>
> +
> +#include <odp_classification_internal.h>
> +#include <odp_queue_internal.h>
> #include <odp_packet_io_ipc_internal.h>
> -#include <odp/api/time.h>
> +#include <odp_packet_io_internal.h>
> +#include <odp_packet_io_queue.h>
> +#include <odp_packet_internal.h>
> +#include <odp_packet_socket.h>
> +#include <odp_schedule_if.h>
>
> #include <string.h>
> #include <inttypes.h>
> @@ -470,7 +472,6 @@ int odp_pktio_start(odp_pktio_t hdl)
> return -1;
> }
> }
> -
> sched_fn->pktio_start(pktio_to_id(hdl), num, index);
> }
>
> @@ -552,7 +553,6 @@ static inline int pktin_recv_buf(odp_pktin_queue_t queue,
> odp_packet_t packets[num];
> odp_packet_hdr_t *pkt_hdr;
> odp_buffer_hdr_t *buf_hdr;
> - odp_buffer_t buf;
> int i;
> int pkts;
> int num_rx = 0;
> @@ -562,9 +562,11 @@ static inline int pktin_recv_buf(odp_pktin_queue_t queue,
> for (i = 0; i < pkts; i++) {
> pkt = packets[i];
> pkt_hdr = odp_packet_hdr(pkt);
> - buf = _odp_packet_to_buffer(pkt);
> - buf_hdr = buf_hdl_to_hdr(buf);
> -
> +#ifdef ODP_SCHEDULE_SCALABLE
> + buf_hdr = _odp_packet_to_buf_hdr_ptr(pkt);
> +#else
> + buf_hdr = buf_hdl_to_hdr(_odp_packet_to_buffer(pkt));
> +#endif
> if (pkt_hdr->p.input_flags.dst_queue) {
> queue_entry_t *dst_queue;
> int ret;
> @@ -582,11 +584,17 @@ static inline int pktin_recv_buf(odp_pktin_queue_t queue,
>
> int pktout_enqueue(queue_entry_t *qentry, odp_buffer_hdr_t *buf_hdr)
> {
> - odp_packet_t pkt = _odp_packet_from_buffer(buf_hdr->handle.handle);
> + odp_packet_t pkt;
> +
> +#ifdef ODP_SCHEDULE_SCALABLE
> + pkt = _odp_packet_from_buffer((odp_buffer_t)buf_hdr);
> +#else
> + pkt = _odp_packet_from_buffer(buf_hdr->handle.handle);
> +#endif
> int len = 1;
> int nbr;
>
> - nbr = odp_pktout_send(qentry->s.pktout, &pkt, len);
> + nbr = odp_pktout_send(queue_get_pktout(qentry), &pkt, len);
> return (nbr == len ? 0 : -1);
> }
>
> @@ -604,9 +612,13 @@ int pktout_enq_multi(queue_entry_t *qentry, odp_buffer_hdr_t *buf_hdr[],
> int i;
>
> for (i = 0; i < num; ++i)
> +#ifdef ODP_SCHEDULE_SCALABLE
> + pkt_tbl[i] = _odp_packet_from_buffer((odp_buffer_t)buf_hdr[i]);
> +#else
> pkt_tbl[i] = _odp_packet_from_buffer(buf_hdr[i]->handle.handle);
> +#endif
>
> - nbr = odp_pktout_send(qentry->s.pktout, pkt_tbl, num);
> + nbr = odp_pktout_send(queue_get_pktout(qentry), pkt_tbl, num);
> return nbr;
> }
>
> @@ -632,13 +644,14 @@ odp_buffer_hdr_t *pktin_dequeue(queue_entry_t *qentry)
> int pkts;
>
> buf_hdr = queue_deq(qentry);
> - if (buf_hdr != NULL)
> + if (buf_hdr != BUFFER_HDR_INVALID)
> return buf_hdr;
>
> - pkts = pktin_recv_buf(qentry->s.pktin, hdr_tbl, QUEUE_MULTI_MAX);
> + pkts = pktin_recv_buf(queue_get_pktin(qentry),
> + hdr_tbl, QUEUE_MULTI_MAX);
>
> if (pkts <= 0)
> - return NULL;
> + return BUFFER_HDR_INVALID;
>
> if (pkts > 1)
> queue_enq_multi(qentry, &hdr_tbl[1], pkts - 1);
> @@ -669,7 +682,8 @@ int pktin_deq_multi(queue_entry_t *qentry, odp_buffer_hdr_t *buf_hdr[], int num)
> if (nbr == num)
> return nbr;
>
> - pkts = pktin_recv_buf(qentry->s.pktin, hdr_tbl, QUEUE_MULTI_MAX);
> + pkts = pktin_recv_buf(queue_get_pktin(qentry),
> + hdr_tbl, QUEUE_MULTI_MAX);
> if (pkts <= 0)
> return nbr;
>
> @@ -684,7 +698,6 @@ int pktin_deq_multi(queue_entry_t *qentry, odp_buffer_hdr_t *buf_hdr[], int num)
> queue_enq_multi(qentry, hdr_tbl, j);
> return nbr;
> }
> -
> int sched_cb_pktin_poll(int pktio_index, int num_queue, int index[])
> {
> odp_buffer_hdr_t *hdr_tbl[QUEUE_MULTI_MAX];
> @@ -1266,13 +1279,14 @@ int odp_pktin_queue_config(odp_pktio_t pktio,
> queue_entry_t *qentry;
>
> qentry = queue_to_qentry(queue);
> - qentry->s.pktin.index = i;
> - qentry->s.pktin.pktio = pktio;
> -
> - qentry->s.enqueue = pktin_enqueue;
> - qentry->s.dequeue = pktin_dequeue;
> - qentry->s.enqueue_multi = pktin_enq_multi;
> - qentry->s.dequeue_multi = pktin_deq_multi;
> + queue_set_pktin(qentry, pktio, i);
> +
> + queue_set_enq_func(qentry, pktin_enqueue);
> + queue_set_deq_func(qentry, pktin_dequeue);
> + queue_set_enq_multi_func(qentry,
> + pktin_enq_multi);
> + queue_set_deq_multi_func(qentry,
> + pktin_deq_multi);
> }
>
> entry->s.in_queue[i].queue = queue;
> @@ -1390,14 +1404,12 @@ int odp_pktout_queue_config(odp_pktio_t pktio,
> }
>
> qentry = queue_to_qentry(queue);
> - qentry->s.pktout.index = i;
> - qentry->s.pktout.pktio = pktio;
> -
> - /* Override default enqueue / dequeue functions */
> - qentry->s.enqueue = pktout_enqueue;
> - qentry->s.dequeue = pktout_dequeue;
> - qentry->s.enqueue_multi = pktout_enq_multi;
> - qentry->s.dequeue_multi = pktout_deq_multi;
> + queue_set_pktout(qentry, pktio, i);
> +
> + queue_set_enq_func(qentry, pktout_enqueue);
> + queue_set_deq_func(qentry, pktout_dequeue);
> + queue_set_enq_multi_func(qentry, pktout_enq_multi);
> + queue_set_deq_multi_func(qentry, pktout_deq_multi);
>
> entry->s.out_queue[i].queue = queue;
> }
> diff --git a/platform/linux-generic/odp_queue.c b/platform/linux-generic/odp_queue.c
> index fcf4bf5b..42b58c44 100644
> --- a/platform/linux-generic/odp_queue.c
> +++ b/platform/linux-generic/odp_queue.c
> @@ -291,11 +291,11 @@ void sched_cb_queue_destroy_finalize(uint32_t queue_index)
> int odp_queue_destroy(odp_queue_t handle)
> {
> queue_entry_t *queue;
> - queue = queue_to_qentry(handle);
>
> if (handle == ODP_QUEUE_INVALID)
> return -1;
>
> + queue = queue_to_qentry(handle);
> LOCK(&queue->s.lock);
> if (queue->s.status == QUEUE_STATUS_FREE) {
> UNLOCK(&queue->s.lock);
> diff --git a/platform/linux-generic/odp_queue_scalable.c b/platform/linux-generic/odp_queue_scalable.c
> new file mode 100644
> index 00000000..c08dd06a
> --- /dev/null
> +++ b/platform/linux-generic/odp_queue_scalable.c
> @@ -0,0 +1,944 @@
> +/* Copyright (c) 2017, ARM Limited
> + * All rights reserved.
> + *
> + * SPDX-License-Identifier: BSD-3-Clause
> + */
> +
> +#include <odp/api/hints.h>
> +#include <odp/api/plat/ticketlock_inlines.h>
> +#include <odp/api/queue.h>
> +#include <odp/api/schedule.h>
> +#include <odp/api/shared_memory.h>
> +#include <odp/api/sync.h>
> +#include <odp/api/traffic_mngr.h>
> +
> +#include <odp_internal.h>
> +#include <odp_config_internal.h>
> +#include <odp_debug_internal.h>
> +
> +#include <odp_buffer_inlines.h>
> +#include <odp_packet_io_internal.h>
> +#include <odp_packet_io_queue.h>
> +#include <odp_pool_internal.h>
> +#include <odp_queue_internal.h>
> +#include <odp_schedule_if.h>
> +
> +#include <string.h>
> +#include <inttypes.h>
> +
> +#define NUM_INTERNAL_QUEUES 64
> +
> +#define MIN(a, b) \
> + ({ \
> + __typeof__(a) tmp_a = (a); \
> + __typeof__(b) tmp_b = (b); \
> + tmp_a < tmp_b ? tmp_a : tmp_b; \
> + })
> +
> +#define LOCK(a) _odp_ticketlock_lock(a)
> +#define UNLOCK(a) _odp_ticketlock_unlock(a)
> +#define LOCK_INIT(a) odp_ticketlock_init(a)
> +
> +extern __thread sched_scalable_thread_state_t *sched_ts;
> +
> +typedef struct queue_table_t {
> + queue_entry_t queue[ODP_CONFIG_QUEUES];
> +} queue_table_t;
> +
> +static queue_table_t *queue_tbl;
> +
> +static inline odp_queue_t queue_from_id(uint32_t queue_id)
> +{
> + return _odp_cast_scalar(odp_queue_t, queue_id + 1);
> +}
> +
> +queue_entry_t *get_qentry(uint32_t queue_id)
> +{
> + return &queue_tbl->queue[queue_id];
> +}
> +
> +static int _odp_queue_disable_enq(sched_elem_t *q)
> +{
> + ringidx_t old_read, old_write, new_write;
> + uint32_t size;
> +
> +#ifndef ODP_CONFIG_USE_LLSC
> + old_write = q->prod_write;
> +#endif
> + size = q->prod_mask + 1;
> + do {
> + /* Need __atomic_load to avoid compiler reordering */
> + old_read = __atomic_load_n(&q->prod_read, __ATOMIC_ACQUIRE);
> +#ifdef ODP_CONFIG_USE_LLSC
> + old_write = ll32(&q->prod_write, __ATOMIC_RELAXED);
> +#endif
> + if (old_write != old_read) {
> + /* Queue is not empty, cannot claim all elements
> + * Cannot disable enqueue.
> + */
> + return -1;
> + }
> + /* Claim all elements in ring */
> + new_write = old_write + size;
> +#ifdef ODP_CONFIG_USE_LLSC
> + } while (odp_unlikely(sc32(&q->prod_write,
> + new_write,
> + __ATOMIC_RELAXED)));
> +#else
> + } while (!__atomic_compare_exchange_n(&q->prod_write,
> + &old_write, /* Updated on failure */
> + new_write,
> + true,
> + __ATOMIC_RELAXED,
> + __ATOMIC_RELAXED));
> +#endif
> + /* All remaining elements claimed, noone else can enqueue */
> + return 0;
> +}
> +
> +static uint32_t roundup_pow2(uint32_t v)
> +{
> + uint32_t i;
> +
> + /* Check if already power of 2 */
> + if ((v & (v - 1)) == 0)
> + return v;
> +
> + for (i = 0; (i < (sizeof(v) * CHAR_BIT)) && (v != 0); i++)
> + v = v >> 1;
> +
> + return 1<<i;
> +}
> +
> +static int queue_init(int queue_idx, queue_entry_t *queue, const char *name,
> + const odp_queue_param_t *param)
> +{
> + ringidx_t ring_idx;
> + sched_elem_t *sched_elem;
> + odp_shm_t shm;
> + uint32_t ring_size;
> + odp_buffer_hdr_t **ring;
> +
> + sched_elem = &queue->s.sched_elem;
> + ring_size = param->ring_size > 0 ?
> + roundup_pow2(param->ring_size) :
> + ODP_CONFIG_QUEUE_SIZE;
> + strncpy(queue->s.name, name ? name : "", ODP_QUEUE_NAME_LEN - 1);
> + queue->s.name[ODP_QUEUE_NAME_LEN - 1] = 0;
> + memcpy(&queue->s.param, param, sizeof(odp_queue_param_t));
> +
> + shm = odp_shm_reserve(name,
> + ring_size * sizeof(odp_buffer_hdr_t *),
> + ODP_CACHE_LINE_SIZE,
> + ODP_SHM_PROC);
> + if (ODP_SHM_INVALID == shm)
> + return -1;
> +
> + ring = (odp_buffer_hdr_t **)odp_shm_addr(shm);
> +
> + for (ring_idx = 0; ring_idx < ring_size; ring_idx++)
> + ring[ring_idx] = NULL;
> +
> + queue->s.shm = shm;
> + queue->s.type = queue->s.param.type;
> + queue->s.enqueue = queue_enq;
> + queue->s.dequeue = queue_deq;
> + queue->s.enqueue_multi = queue_enq_multi;
> + queue->s.dequeue_multi = queue_deq_multi;
> + queue->s.pktin = PKTIN_INVALID;
> +
> + sched_elem->node.next = NULL;
> +#ifdef CONFIG_QSCHST_LOCK
> + LOCK_INIT(&sched_elem->qschlock);
> +#endif
> + sched_elem->qschst.numevts = 0;
> + sched_elem->qschst.wrr_budget = CONFIG_WRR_WEIGHT;
> + sched_elem->qschst.cur_ticket = 0;
> + sched_elem->qschst.nxt_ticket = 0;
> + sched_elem->pop_deficit = 0;
> + if (queue->s.type == ODP_QUEUE_TYPE_SCHED)
> + sched_elem->qschst_type = queue->s.param.sched.sync;
> + else
> + sched_elem->qschst_type = ODP_NO_SCHED_QUEUE;
> + /* 2nd cache line - enqueue */
> + sched_elem->prod_read = 0;
> + sched_elem->prod_write = 0;
> + sched_elem->prod_ring = ring;
> + sched_elem->prod_mask = ring_size - 1;
> + /* 3rd cache line - dequeue */
> + sched_elem->cons_read = 0;
> + sched_elem->cons_write = 0;
> + sched_elem->rwin = NULL;
> + sched_elem->schedq = NULL;
> + sched_elem->user_ctx = queue->s.param.context;
> +#ifdef ODP_CONFIG_USE_SPLIT_PRODCONS
> + sched_elem->cons_ring = ring;
> + sched_elem->cons_mask = ring_size - 1;
> + sched_elem->cons_type = sched_elem->qschst_type;
> +#endif
> +
> + /* Queue initialized successfully, add it to the sched group */
> + if (queue->s.type == ODP_QUEUE_TYPE_SCHED) {
> + if (queue->s.param.sched.sync == ODP_SCHED_SYNC_ORDERED) {
> + sched_elem->rwin =
> + rwin_alloc(queue_idx, &queue->s.rwin_shm,
> + queue->s.param.sched.lock_count);
> + if (sched_elem->rwin == NULL) {
> + ODP_ERR("Reorder window not created\n");
> + goto rwin_create_failed;
> + }
> + }
> + sched_elem->schedq =
> + schedq_from_sched_group(param->sched.group,
> + param->sched.prio);
> + }
> +
> + return 0;
> +
> +rwin_create_failed:
> + odp_shm_free(queue->s.shm);
> + return -1;
> +}
> +
> +int odp_queue_init_global(void)
> +{
> + uint32_t i;
> + odp_shm_t shm;
> +
> + ODP_DBG("Queue init ... ");
> +
> + shm = odp_shm_reserve("odp_queues",
> + sizeof(queue_table_t),
> + sizeof(queue_entry_t), 0);
> +
> + queue_tbl = odp_shm_addr(shm);
> +
> + if (queue_tbl == NULL)
> + return -1;
> +
> + memset(queue_tbl, 0, sizeof(queue_table_t));
> +
> + for (i = 0; i < ODP_CONFIG_QUEUES; i++) {
> + /* init locks */
> + queue_entry_t *queue;
> +
> + queue = get_qentry(i);
> + LOCK_INIT(&queue->s.lock);
> + queue->s.index = i;
> + queue->s.handle = queue_from_id(i);
> + }
> +
> + ODP_DBG("done\n");
> + ODP_DBG("Queue init global\n");
> + ODP_DBG(" struct queue_entry_s size %zu\n",
> + sizeof(struct queue_entry_s));
> + ODP_DBG(" queue_entry_t size %zu\n",
> + sizeof(queue_entry_t));
> + ODP_DBG("\n");
> +
> + return 0;
> +}
> +
> +int odp_queue_term_global(void)
> +{
> + int ret = 0;
> + int rc = 0;
> + queue_entry_t *queue;
> + int i;
> +
> + for (i = 0; i < ODP_CONFIG_QUEUES; i++) {
> + queue = &queue_tbl->queue[i];
> + if (__atomic_load_n(&queue->s.status,
> + __ATOMIC_RELAXED) != QUEUE_STATUS_FREE) {
> + ODP_ERR("Not destroyed queue: %s\n", queue->s.name);
> + rc = -1;
> + }
> + }
> +
> + ret = odp_shm_free(odp_shm_lookup("odp_queues"));
> + if (ret < 0) {
> + ODP_ERR("shm free failed for odp_queues");
> + rc = -1;
> + }
> +
> + return rc;
> +}
> +
> +int odp_queue_capability(odp_queue_capability_t *capa)
> +{
> + memset(capa, 0, sizeof(odp_queue_capability_t));
> +
> + /* Reserve some queues for internal use */
> + capa->max_queues = ODP_CONFIG_QUEUES - NUM_INTERNAL_QUEUES;
> + capa->max_ordered_locks = SCHEDULE_ORDERED_LOCKS_PER_QUEUE;
> + capa->max_sched_groups = sched_fn->num_grps();
> + capa->sched_prios = odp_schedule_num_prio();
> +
> + return 0;
> +}
> +
> +odp_queue_type_t odp_queue_type(odp_queue_t handle)
> +{
> + return queue_to_qentry(handle)->s.type;
> +}
> +
> +odp_schedule_sync_t odp_queue_sched_type(odp_queue_t handle)
> +{
> + return queue_to_qentry(handle)->s.param.sched.sync;
> +}
> +
> +odp_schedule_prio_t odp_queue_sched_prio(odp_queue_t handle)
> +{
> + return queue_to_qentry(handle)->s.param.sched.prio;
> +}
> +
> +odp_schedule_group_t odp_queue_sched_group(odp_queue_t handle)
> +{
> + return queue_to_qentry(handle)->s.param.sched.group;
> +}
> +
> +int odp_queue_lock_count(odp_queue_t handle)
> +{
> + queue_entry_t *queue = queue_to_qentry(handle);
> +
> + return queue->s.param.sched.sync == ODP_SCHED_SYNC_ORDERED ?
> + (int)queue->s.param.sched.lock_count : -1;
> +}
> +
> +odp_queue_t odp_queue_create(const char *name, const odp_queue_param_t *param)
> +{
> + int queue_idx;
> + odp_queue_t handle = ODP_QUEUE_INVALID;
> + queue_entry_t *queue;
> + odp_queue_param_t default_param;
> +
> + if (param == NULL) {
> + odp_queue_param_init(&default_param);
> + param = &default_param;
> + }
> +
> + for (queue_idx = 0; queue_idx < ODP_CONFIG_QUEUES; queue_idx++) {
> + queue = &queue_tbl->queue[queue_idx];
> +
> + if (queue->s.status != QUEUE_STATUS_FREE)
> + continue;
> +
> + LOCK(&queue->s.lock);
> + if (queue->s.status == QUEUE_STATUS_FREE) {
> + if (queue_init(queue_idx, queue, name, param)) {
> + UNLOCK(&queue->s.lock);
> + return handle;
> + }
> + queue->s.status = QUEUE_STATUS_READY;
> + handle = queue->s.handle;
> + UNLOCK(&queue->s.lock);
> + break;
> + }
> + UNLOCK(&queue->s.lock);
> + }
> + return handle;
> +}
> +
> +int odp_queue_destroy(odp_queue_t handle)
> +{
> + queue_entry_t *queue;
> + sched_elem_t *q;
> +
> + if (handle == ODP_QUEUE_INVALID)
> + return -1;
> +
> + queue = queue_to_qentry(handle);
> + LOCK(&queue->s.lock);
> + if (queue->s.status != QUEUE_STATUS_READY) {
> + UNLOCK(&queue->s.lock);
> + return -1;
> + }
> + q = &queue->s.sched_elem;
> +
> +#ifdef CONFIG_QSCHST_LOCK
> + LOCK(&q->qschlock);
> +#endif
> + if (_odp_queue_disable_enq(q)) {
> + /* Producer side not empty */
> +#ifdef CONFIG_QSCHST_LOCK
> + UNLOCK(&q->qschlock);
> +#endif
> + UNLOCK(&queue->s.lock);
> + return -1;
> + }
> + /* Enqueue is now disabled */
> + if (q->cons_read != q->cons_write) {
> + /* Consumer side is not empty
> + * Roll back previous change, enable enqueue again.
> + */
> + uint32_t size;
> +
> + size = q->prod_mask + 1;
> + __atomic_fetch_sub(&q->prod_write, size, __ATOMIC_RELAXED);
> +#ifdef CONFIG_QSCHST_LOCK
> + UNLOCK(&q->qschlock);
> +#endif
> + UNLOCK(&queue->s.lock);
> + return -1;
> + }
> +#ifdef CONFIG_QSCHST_LOCK
> + UNLOCK(&q->qschlock);
> +#endif
> + /* Producer and consumer sides empty, enqueue disabled
> + * Now wait until schedq state is empty and no outstanding tickets
> + */
> + while (__atomic_load_n(&q->qschst.numevts, __ATOMIC_RELAXED) != 0 ||
> + __atomic_load_n(&q->qschst.cur_ticket, __ATOMIC_RELAXED) !=
> + __atomic_load_n(&q->qschst.nxt_ticket, __ATOMIC_RELAXED)) {
> + SEVL();
> + while (WFE() && LDXR32((uint32_t *)&q->qschst.numevts,
> + __ATOMIC_RELAXED) != 0)
> + DOZE();
> + }
> +
> + /* Adjust the spread factor for the queues in the schedule group */
> + if (queue->s.type == ODP_QUEUE_TYPE_SCHED)
> + sched_group_xcount_dec(queue->s.param.sched.group,
> + queue->s.param.sched.prio);
> +
> + if (odp_shm_free(queue->s.shm) < 0) {
> + UNLOCK(&queue->s.lock);
> + return -1;
> + }
> + if (queue->s.param.sched.sync == ODP_SCHED_SYNC_ORDERED) {
> + if (odp_shm_free(queue->s.rwin_shm) < 0) {
> + ODP_ERR("Failed to free reorder window shm\n");
> + UNLOCK(&queue->s.lock);
> + return -1;
> + }
> + }
> + queue->s.status = QUEUE_STATUS_FREE;
> + UNLOCK(&queue->s.lock);
> + return 0;
> +}
> +
> +int odp_queue_context_set(odp_queue_t handle, void *context,
> + uint32_t len ODP_UNUSED)
> +{
> + odp_mb_full();
> + queue_to_qentry(handle)->s.param.context = context;
> + odp_mb_full();
> + return 0;
> +}
> +
> +void *odp_queue_context(odp_queue_t handle)
> +{
> + return queue_to_qentry(handle)->s.param.context;
> +}
> +
> +odp_queue_t odp_queue_lookup(const char *name)
> +{
> + uint32_t i;
> +
> + for (i = 0; i < ODP_CONFIG_QUEUES; i++) {
> + queue_entry_t *queue = &queue_tbl->queue[i];
> +
> + if (queue->s.status == QUEUE_STATUS_FREE ||
> + queue->s.status == QUEUE_STATUS_DESTROYED)
> + continue;
> +
> + LOCK(&queue->s.lock);
> + if (strcmp(name, queue->s.name) == 0) {
> + /* found it */
> + UNLOCK(&queue->s.lock);
> + return queue->s.handle;
> + }
> + UNLOCK(&queue->s.lock);
> + }
> +
> + return ODP_QUEUE_INVALID;
> +}
> +
> +static inline int _odp_queue_enq(sched_elem_t *q,
> + odp_buffer_hdr_t *buf_hdr[],
> + int num)
> +{
> + ringidx_t old_read;
> + ringidx_t old_write;
> + ringidx_t new_write;
> + int actual;
> + uint32_t mask;
> + odp_buffer_hdr_t **ring;
> +
> + mask = q->prod_mask;
> + ring = q->prod_ring;
> +
> + /* Load producer ring state (read & write index) */
> +#ifndef ODP_CONFIG_USE_LLSC
> + old_write = __atomic_load_n(&q->prod_write, __ATOMIC_RELAXED);
> +#endif
> + do {
> + /* Consumer does store-release prod_read, we need
> + * load-acquire.
> + */
> + old_read = __atomic_load_n(&q->prod_read, __ATOMIC_ACQUIRE);
> +#ifdef ODP_CONFIG_USE_LLSC
> + old_write = ll32(&q->prod_write, __ATOMIC_RELAXED);
> +#endif
> +
> + actual = MIN(num, (int)((mask + 1) - (old_write - old_read)));
> + if (odp_unlikely(actual <= 0))
> + return 0;
> +
> + new_write = old_write + actual;
> +#ifdef ODP_CONFIG_USE_LLSC
> + } while (odp_unlikely(sc32(&q->prod_write,
> + new_write,
> + __ATOMIC_RELAXED)));
> +#else
> + } while (!__atomic_compare_exchange_n(&q->prod_write,
> + &old_write, /* Updated on failure */
> + new_write,
> + true,
> + __ATOMIC_RELAXED,
> + __ATOMIC_RELAXED));
> +#endif
> +
> +#ifdef ODP_CONFIG_USE_SPLIT_PRODCONS
> + __builtin_prefetch(&q->cons_write, 0, 0);
> +#endif
> + /* Store our event(s) in the ring */
> + do {
> + ring[old_write & mask] = *buf_hdr++;
> + } while (++old_write != new_write);
> + old_write -= actual;
> +
> +#ifdef ODP_CONFIG_USE_SPLIT_PRODCONS
> + __builtin_prefetch(&q->node, 1, 0);
> +#endif
> + /* Wait for our turn to signal consumers */
> + if (odp_unlikely(__atomic_load_n(&q->cons_write,
> + __ATOMIC_RELAXED) != old_write)) {
> + SEVL();
> + while (WFE() && LDXR32(&q->cons_write,
> + __ATOMIC_RELAXED) != old_write)
> + DOZE();
> + }
> +
> + /* Signal consumers that events are available (release events)
> + * Enable other producers to continue
> + */
> +#ifdef ODP_CONFIG_USE_DMB
> + /* Wait for writes (to ring slots) to complete */
> + __asm volatile("dmb ishst" ::: "memory");
> + q->cons_write = new_write;
> +#else
> + __atomic_store_n(&q->cons_write, new_write, __ATOMIC_RELEASE);
> +#endif
> +
> + return actual;
> +}
> +
> +static inline int _odp_queue_enq_sp(sched_elem_t *q,
> + odp_buffer_hdr_t *buf_hdr[],
> + int num)
> +{
> + ringidx_t old_read;
> + ringidx_t old_write;
> + ringidx_t new_write;
> + int actual;
> + uint32_t mask;
> + odp_buffer_hdr_t **ring;
> +
> + mask = q->prod_mask;
> + ring = q->prod_ring;
> +
> + /* Load producer ring state (read & write index) */
> + old_write = q->prod_write;
> + /* Consumer does store-release prod_read, we need load-acquire */
> + old_read = __atomic_load_n(&q->prod_read, __ATOMIC_ACQUIRE);
> + actual = MIN(num, (int)((mask + 1) - (old_write - old_read)));
> + if (odp_unlikely(actual <= 0))
> + return 0;
> +
> + new_write = old_write + actual;
> + q->prod_write = new_write;
> +
> + /* Store our event(s) in the ring */
> + do {
> + ring[old_write & mask] = *buf_hdr++;
> + } while (++old_write != new_write);
> + old_write -= actual;
> +
> +#ifdef ODP_CONFIG_USE_SPLIT_PRODCONS
> + __builtin_prefetch(&q->node, 1, 0);
> +#endif
> +
> + /* Signal consumers that events are available (release events)
> + * Enable other producers to continue
> + */
> +#ifdef CONFIG_QSCHST_LOCK
> + q->cons_write = new_write;
> +#else
> +#ifdef ODP_CONFIG_USE_DMB
> + __asm volatile("dmb ishst" ::: "memory");
> + q->cons_write = new_write;
> +#else
> + __atomic_store_n(&q->cons_write, new_write, __ATOMIC_RELEASE);
> +#endif
> +#endif
> +
> + return actual;
> +}
> +
> +int queue_enq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[], int num)
> +{
> + int actual;
> + int i;
> + sched_scalable_thread_state_t *ts;
> + reorder_context_t *first;
> + reorder_context_t *cur;
> + bitset_t next_idx;
> +
> +#ifdef CONFIG_QSCHST_LOCK
> + LOCK(&queue->s.sched_elem.qschlock);
> +#endif
> + ts = sched_ts;
> + if (ts && odp_unlikely(ts->out_of_order)) {
> + first = ts->rctx;
> + ODP_ASSERT(ts->rctx != NULL);
> + cur = &first[(int)first->cur_idx - (int)first->idx];
> + for (i = 0; i < num; i++) {
> + if (odp_unlikely(cur->numevts == RC_EVT_SIZE)) {
> + /* No more space in current reorder context
> + * Try to allocate another.
> + */
> + if (odp_unlikely(
> + bitset_is_null(ts->priv_rvec_free))) {
> + ts->priv_rvec_free =
> + atom_bitset_xchg(
> + &ts->rvec_free,
> + 0,
> + __ATOMIC_RELAXED);
> + if (odp_unlikely(bitset_is_null(
> + ts->priv_rvec_free)))
> + /* Out of reorder contexts.
> + * Return the number of events
> + * stored so far.
> + */
> +#ifdef CONFIG_QSCHST_LOCK
> + UNLOCK(&queue->s.sched_elem.
> + qschlock);
> +#endif
> + return i;
> + }
> + next_idx = bitset_ffs(ts->priv_rvec_free) - 1;
> + ts->priv_rvec_free =
> + bitset_clr(ts->priv_rvec_free,
> + next_idx);
> + /* Link current to next (for eventual
> + * retiring)
> + */
> + cur->next_idx = next_idx;
> + /* Link first to next (for next call to
> + * queue_enq_multi())
> + */
> + first->cur_idx = next_idx;
> + /* Update current to next */
> + cur = &ts->rvec[next_idx];
> + rctx_init(cur, next_idx, NULL, 0);
> + /* The last rctx (so far) */
> + cur->next_idx = first->idx;
> + }
> + cur->events[cur->numevts] = buf_hdr[i];
> + cur->destq[cur->numevts] = queue;
> + cur->numevts++;
> + }
> + /* All events stored. */
> +#ifdef CONFIG_QSCHST_LOCK
> + UNLOCK(&queue->s.sched_elem.qschlock);
> +#endif
> + return num;
> + }
> +
> +#ifdef CONFIG_QSCHST_LOCK
> + actual = _odp_queue_enq_sp(&queue->s.sched_elem, buf_hdr, num);
> +#else
> + actual = _odp_queue_enq(&queue->s.sched_elem, buf_hdr, num);
> +#endif
> +
> + if (odp_likely(queue->s.sched_elem.schedq != NULL && actual != 0)) {
> + /* Perform scheduler related updates. */
> +#ifdef CONFIG_QSCHST_LOCK
> + sched_update_enq_sp(&queue->s.sched_elem, actual);
> +#else
> + sched_update_enq(&queue->s.sched_elem, actual);
> +#endif
> + }
> +
> +#ifdef CONFIG_QSCHST_LOCK
> + UNLOCK(&queue->s.sched_elem.qschlock);
> +#endif
> + return actual;
> +}
> +
> +int queue_enq(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr)
> +{
> + return odp_likely(
> + queue_enq_multi(queue, &buf_hdr, 1) == 1) ? 0 : -1;
> +}
> +
> +int odp_queue_enq_multi(odp_queue_t handle, const odp_event_t ev[], int num)
> +{
> + odp_buffer_hdr_t *buf_hdr[QUEUE_MULTI_MAX];
> + queue_entry_t *queue;
> + int i;
> +
> + if (num > QUEUE_MULTI_MAX)
> + num = QUEUE_MULTI_MAX;
> +
> + queue = queue_to_qentry(handle);
> +
> + for (i = 0; i < num; i++)
> + buf_hdr[i] = (odp_buffer_hdr_t *)ev[i];
> +
> + return queue->s.enqueue_multi(queue, buf_hdr, num);
> +}
> +
> +int odp_queue_enq(odp_queue_t handle, odp_event_t ev)
> +{
> + queue_entry_t *queue;
> +
> + queue = queue_to_qentry(handle);
> + return queue->s.enqueue(queue, (odp_buffer_hdr_t *)ev);
> +}
> +
> +/* Single-consumer dequeue. */
> +int _odp_queue_deq_sc(sched_elem_t *q, odp_event_t *evp, int num)
> +{
> + int actual;
> + int i;
> + ringidx_t old_read;
> + ringidx_t old_write;
> + ringidx_t new_read;
> + uint32_t mask;
> + odp_event_t *ring;
> +
> + /* Load consumer ring state (read & write index). */
> + old_read = q->cons_read;
> + /* Producer does store-release cons_write, we need load-acquire */
> + old_write = __atomic_load_n(&q->cons_write, __ATOMIC_ACQUIRE);
> + actual = MIN(num, (int)(old_write - old_read));
> +
> + if (odp_unlikely(actual <= 0))
> + return 0;
> +
> +#ifdef ODP_CONFIG_USE_SPLIT_PRODCONS
> + __builtin_prefetch(&q->node, 1, 0);
> +#endif
> + new_read = old_read + actual;
> + q->cons_read = new_read;
> +
> + mask = q->cons_mask;
> + ring = (odp_event_t *)q->cons_ring;
> + do {
> + *evp++ = ring[old_read & mask];
> + } while (++old_read != new_read);
> +
> + /* Signal producers that empty slots are available
> + * (release ring slots). Enable other consumers to continue.
> + */
> +#ifdef CONFIG_QSCHST_LOCK
> + q->prod_read = new_read;
> +#else
> +#ifdef ODP_CONFIG_USE_DMB
> + /* Wait for loads (from ring slots) to complete. */
> + __asm volatile("dmb ishld" ::: "memory");
> + q->prod_read = new_read;
> +#else
> + __atomic_store_n(&q->prod_read, new_read, __ATOMIC_RELEASE);
> +#endif
> +#endif
> + /* Prefetch events after we have release ring buffer slots */
> + for (i = 0; i < actual; i++) {
> + odp_event_t evt = *--evp;
> +
> + __builtin_prefetch(
> + odp_buffer_addr(odp_buffer_from_event(evt)), 0, 0);
> + }
> + return actual;
> +}
> +
> +inline int _odp_queue_deq(sched_elem_t *q, odp_event_t *evp, int num)
> +{
> + int actual;
> + ringidx_t old_read;
> + ringidx_t old_write;
> + ringidx_t new_read;
> + uint32_t mask;
> + odp_buffer_hdr_t **ring;
> + odp_buffer_hdr_t **p_buf_hdr;
> +
> + mask = q->cons_mask;
> + ring = q->cons_ring;
> +
> + /* Load consumer ring state (read & write index) */
> +#ifndef ODP_CONFIG_USE_LLSC
> + old_read = __atomic_load_n(&q->cons_read, __ATOMIC_RELAXED);
> +#endif
> + do {
> + /* Need __atomic_load to avoid compiler reordering
> + * Producer does store-release cons_write, we need
> + * load-acquire.
> + */
> + old_write = __atomic_load_n(&q->cons_write, __ATOMIC_ACQUIRE);
> +#ifdef ODP_CONFIG_USE_LLSC
> + old_read = ll32(&q->cons_read, __ATOMIC_RELAXED);
> +#endif
> + /* Prefetch ring buffer array */
> + __builtin_prefetch(&q->cons_ring[old_read & mask], 0, 0);
> +
> + actual = MIN(num, (int)(old_write - old_read));
> + if (odp_unlikely(actual <= 0))
> + return 0;
> +
> + /* Attempt to free ring slot(s) */
> + new_read = old_read + actual;
> +#ifdef ODP_CONFIG_USE_LLSC
> + } while (odp_unlikely(sc32(&q->cons_read, new_read, __ATOMIC_RELAXED)));
> +#else
> + } while (!__atomic_compare_exchange_n(&q->cons_read,
> + &old_read, /* Updated on failure */
> + new_read,
> + true,
> + __ATOMIC_RELAXED,
> + __ATOMIC_RELAXED));
> +#endif
> +#ifdef ODP_CONFIG_USE_SPLIT_PRODCONS
> + __builtin_prefetch(&q->prod_read, 0, 0);
> +#endif
> + p_buf_hdr = (odp_buffer_hdr_t **)evp;
> + do {
> + odp_buffer_hdr_t *p_buf;
> +
> + p_buf = ring[old_read & mask];
> + __builtin_prefetch(
> + odp_buffer_addr(
> + odp_buffer_from_event((odp_event_t)p_buf)),
> + 0, 0);
> + *p_buf_hdr++ = p_buf;
> + } while (++old_read != new_read);
> + old_read -= actual;
> +
> +#ifdef ODP_CONFIG_USE_SPLIT_PRODCONS
> + __builtin_prefetch(&q->node, 1, 0);
> +#endif
> + /* Wait for our turn to signal producers */
> + if (odp_unlikely(__atomic_load_n(&q->prod_read, __ATOMIC_RELAXED) !=
> + old_read)) {
> + SEVL();
> + while (WFE() && LDXR32(&q->prod_read,
> + __ATOMIC_RELAXED) != old_read)
> + DOZE();
> + }
> +
> + /* Signal producers that empty slots are available
> + * (release ring slots)
> + * Enable other consumers to continue
> + */
> +#ifdef ODP_CONFIG_USE_DMB
> + /* Wait for loads (from ring slots) to complete */
> + __asm volatile("dmb ishld" ::: "memory");
> + q->prod_read = new_read;
> +#else
> + __atomic_store_n(&q->prod_read, new_read, __ATOMIC_RELEASE);
> +#endif
> +
> + return actual;
> +}
> +
> +int queue_deq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[], int num)
> +{
> + sched_elem_t *q;
> +
> + q = &queue->s.sched_elem;
> + return _odp_queue_deq(q, (odp_event_t *)buf_hdr, num);
> +}
> +
> +odp_buffer_hdr_t *queue_deq(queue_entry_t *queue)
> +{
> + sched_elem_t *q;
> + odp_buffer_hdr_t *buf_hdr;
> +
> + q = &queue->s.sched_elem;
> + if (_odp_queue_deq(q, (odp_event_t *)&buf_hdr, 1) == 1)
> + return buf_hdr;
> + else
> + return BUFFER_HDR_INVALID;
> +}
> +
> +int odp_queue_deq_multi(odp_queue_t handle, odp_event_t events[], int num)
> +{
> + queue_entry_t *queue;
> +
> + queue = queue_to_qentry(handle);
> + return queue->s.dequeue_multi(queue, (odp_buffer_hdr_t **)events, num);
> +}
> +
> +odp_event_t odp_queue_deq(odp_queue_t handle)
> +{
> + queue_entry_t *queue;
> +
> + queue = queue_to_qentry(handle);
> + return (odp_event_t)queue->s.dequeue(queue);
> +}
> +
> +void odp_queue_param_init(odp_queue_param_t *params)
> +{
> + memset(params, 0, sizeof(odp_queue_param_t));
> + params->type = ODP_QUEUE_TYPE_PLAIN;
> + params->enq_mode = ODP_QUEUE_OP_MT;
> + params->deq_mode = ODP_QUEUE_OP_MT;
> + params->sched.prio = ODP_SCHED_PRIO_DEFAULT;
> + params->sched.sync = ODP_SCHED_SYNC_PARALLEL;
> + params->sched.group = ODP_SCHED_GROUP_ALL;
> +}
> +
> +int odp_queue_info(odp_queue_t handle, odp_queue_info_t *info)
> +{
> + uint32_t queue_id;
> + queue_entry_t *queue;
> + int status;
> +
> + if (odp_unlikely(info == NULL)) {
> + ODP_ERR("Unable to store info, NULL ptr given\n");
> + return -1;
> + }
> +
> + queue_id = queue_to_id(handle);
> +
> + if (odp_unlikely(queue_id >= ODP_CONFIG_QUEUES)) {
> + ODP_ERR("Invalid queue handle:%" PRIu64 "\n",
> + odp_queue_to_u64(handle));
> + return -1;
> + }
> +
> + queue = get_qentry(queue_id);
> +
> + LOCK(&queue->s.lock);
> + status = queue->s.status;
> +
> + if (odp_unlikely(status == QUEUE_STATUS_FREE ||
> + status == QUEUE_STATUS_DESTROYED)) {
> + UNLOCK(&queue->s.lock);
> + ODP_ERR("Invalid queue status:%d\n", status);
> + return -1;
> + }
> +
> + info->name = queue->s.name;
> + info->param = queue->s.param;
> +
> + UNLOCK(&queue->s.lock);
> +
> + return 0;
> +}
> +
> +uint64_t odp_queue_to_u64(odp_queue_t hdl)
> +{
> + return _odp_pri(hdl);
> +}
> diff --git a/platform/linux-generic/odp_schedule_if.c b/platform/linux-generic/odp_schedule_if.c
> index a9ede98d..bf92333e 100644
> --- a/platform/linux-generic/odp_schedule_if.c
> +++ b/platform/linux-generic/odp_schedule_if.c
> @@ -6,24 +6,38 @@
>
> #include <odp_schedule_if.h>
>
> -extern const schedule_fn_t schedule_sp_fn;
> -extern const schedule_api_t schedule_sp_api;
> +#if defined(ODP_SCHEDULE_SCALABLE)
>
> -extern const schedule_fn_t schedule_default_fn;
> -extern const schedule_api_t schedule_default_api;
> +extern const schedule_fn_t schedule_scalable_fn;
> +extern const schedule_api_t schedule_scalable_api;
>
> -extern const schedule_fn_t schedule_iquery_fn;
> -extern const schedule_api_t schedule_iquery_api;
> +const schedule_fn_t *sched_fn = &schedule_scalable_fn;
> +const schedule_api_t *sched_api = &schedule_scalable_api;
>
> -#ifdef ODP_SCHEDULE_SP
> -const schedule_fn_t *sched_fn = &schedule_sp_fn;
> -const schedule_api_t *sched_api = &schedule_sp_api;
> #elif defined(ODP_SCHEDULE_IQUERY)
> -const schedule_fn_t *sched_fn = &schedule_iquery_fn;
> +
> +extern const schedule_fn_t schedule_iquery_fn;
> +extern const schedule_api_t schedule_iquery_api;
> +
> +const schedule_fn_t *sched_fn = &schedule_iquery_fn;
> const schedule_api_t *sched_api = &schedule_iquery_api;
> -#else
> +
> +#elif defined(ODP_SCHEDULE_SP)
> +
> +extern const schedule_fn_t schedule_sp_fn;
> +extern const schedule_api_t schedule_sp_api;
> +
> +const schedule_fn_t *sched_fn = &schedule_sp_fn;
> +const schedule_api_t *sched_api = &schedule_sp_api;
> +
> +#else /* default scheduler */
> +
> +extern const schedule_fn_t schedule_default_fn;
> +extern const schedule_api_t schedule_default_api;
> +
> const schedule_fn_t *sched_fn = &schedule_default_fn;
> const schedule_api_t *sched_api = &schedule_default_api;
> +
> #endif
>
> uint64_t odp_schedule_wait_time(uint64_t ns)
> diff --git a/platform/linux-generic/odp_schedule_scalable.c b/platform/linux-generic/odp_schedule_scalable.c
> new file mode 100644
> index 00000000..1e503f5e
> --- /dev/null
> +++ b/platform/linux-generic/odp_schedule_scalable.c
> @@ -0,0 +1,1959 @@
> +/* Copyright (c) 2017, ARM Limited
> + * All rights reserved.
> + *
> + * SPDX-License-Identifier: BSD-3-Clause
> + */
> +
> +#include <odp/api/align.h>
> +#include <odp/api/atomic.h>
> +#include <odp/api/cpu.h>
> +#include <odp/api/hints.h>
> +#include <odp/api/schedule.h>
> +#include <odp/api/shared_memory.h>
> +#include <odp/api/sync.h>
> +#include <odp/api/thread.h>
> +#include <odp/api/thrmask.h>
> +#include <odp/api/time.h>
> +
> +#include <odp_internal.h>
> +#include <odp_config_internal.h>
> +#include <odp_debug_internal.h>
> +
> +#include <odp_align_internal.h>
> +#include <odp_buffer_inlines.h>
> +#include <odp_llqueue.h>
> +#include <odp_queue_internal.h>
> +#include <odp_schedule_if.h>
> +#include <odp_llsc.h>
> +#include <odp_bitset.h>
> +#include <odp_atomic16.h>
> +#include <odp_packet_io_internal.h>
> +
> +#include <limits.h>
> +#include <stdbool.h>
> +#include <string.h>
> +
> +#include <odp/api/plat/ticketlock_inlines.h>
> +#define LOCK(a) _odp_ticketlock_lock((a))
> +#define UNLOCK(a) _odp_ticketlock_unlock((a))
> +
> +#define TAG_EMPTY 0U
> +#define TAG_USED (1U << 15)
> +#define TAG_BUSY (1U << 31)
> +#define PKTIO_QUEUE_2_TAG(p, q) ((p) << 16 | (q) | TAG_USED)
> +#define TAG_2_PKTIO(t) (((t) >> 16) & 0x7FFF)
> +#define TAG_2_QUEUE(t) ((t) & 0x7FFF)
> +#define TAG_IS_READY(t) (((t) & (TAG_USED | TAG_BUSY)) == TAG_USED)
> +#define PKTIN_MAX (ODP_CONFIG_PKTIO_ENTRIES * PKTIO_MAX_QUEUES)
> +#define MAXTHREADS ATOM_BITSET_SIZE
> +
> +static uint32_t pktin_num;
> +static uint32_t pktin_hi;
> +static uint16_t pktin_count[ODP_CONFIG_PKTIO_ENTRIES];
> +static uint32_t pktin_tags[PKTIN_MAX] ODP_ALIGNED_CACHE;
> +
> +#define __atomic_fetch_max(var, v, mo) do { \
> + /* Evalulate 'v' once */ \
> + __typeof__(v) tmp_v = (v); \
> + __typeof__(*var) old_var = \
> + __atomic_load_n((var), __ATOMIC_RELAXED); \
> + while (tmp_v > old_var) { \
> + /* Attempt to store 'v' in '*var' */ \
> + if (__atomic_compare_exchange_n((var), &old_var, \
> + tmp_v, true, (mo), \
> + (mo))) \
> + break; \
> + /* Else failure, try again (with updated value of \
> + * old_var). \
> + */ \
> + } \
> + /* v <= old_var, nothing to do */ \
> + } while (0)
> +
> +ODP_STATIC_ASSERT(ODP_SCHED_PRIO_LOWEST == (ODP_SCHED_PRIO_NUM - 1),
> + "lowest_prio_does_not_match_with_num_prios");
> +
> +ODP_STATIC_ASSERT((ODP_SCHED_PRIO_NORMAL > 0) &&
> + (ODP_SCHED_PRIO_NORMAL < (ODP_SCHED_PRIO_NUM - 1)),
> + "normal_prio_is_not_between_highest_and_lowest");
> +
> +ODP_STATIC_ASSERT(IS_POWER_TWO(ODP_CONFIG_QUEUES),
> + "Number_of_queues_is_not_power_of_two");
> +
> +/*
> + * Scheduler group related variables.
> + */
> +/* Currently used scheduler groups */
> +static sched_group_mask_t sg_free;
> +static sched_group_t *sg_vec[MAX_SCHED_GROUP];
> +/* Group lock for MT-safe APIs */
> +odp_spinlock_t sched_grp_lock;
> +
> +#define SCHED_GROUP_JOIN 0
> +#define SCHED_GROUP_LEAVE 1
> +
> +/*
> + * Per thread state
> + */
> +static sched_scalable_thread_state_t thread_state[MAXTHREADS];
> +__thread sched_scalable_thread_state_t *sched_ts;
> +
> +/*
> + * Forward declarations.
> + */
> +static int thread_state_init(int tidx)
> +{
> + sched_scalable_thread_state_t *ts;
> + uint32_t i;
> +
> + ODP_ASSERT(tidx < MAXTHREADS);
> + ts = &thread_state[tidx];
> + ts->atomq = NULL;
> + ts->rctx = NULL;
> + ts->pause = false;
> + ts->out_of_order = false;
> + ts->tidx = tidx;
> + ts->dequeued = 0;
> + ts->pktin_next = 0;
> + ts->pktin_poll_cnts = 0;
> + ts->ticket = TICKET_INVALID;
> + ts->priv_rvec_free = 0;
> + ts->rvec_free = (1ULL << TS_RVEC_SIZE) - 1;
> + ts->num_schedq = 0;
> + ts->sg_sem = 1; /* Start with sched group semaphore changed */
> + memset(ts->sg_actual, 0, sizeof(ts->sg_actual));
> + for (i = 0; i < TS_RVEC_SIZE; i++) {
> + ts->rvec[i].rvec_free = &ts->rvec_free;
> + ts->rvec[i].idx = i;
> + }
> + sched_ts = ts;
> +
> + return 0;
> +}
> +
> +static void insert_schedq_in_list(sched_scalable_thread_state_t *ts,
> + sched_queue_t *schedq)
> +{
> + /* Find slot for schedq */
> + for (uint32_t i = 0; i < ts->num_schedq; i++) {
> + /* Lower value is higher priority and closer to start of list */
> + if (schedq->prio <= ts->schedq_list[i]->prio) {
> + /* This is the slot! */
> + sched_queue_t *tmp;
> +
> + tmp = ts->schedq_list[i];
> + ts->schedq_list[i] = schedq;
> + schedq = tmp;
> + /* Continue the insertion procedure with the
> + * new schedq.
> + */
> + }
> + }
> + if (ts->num_schedq == SCHEDQ_PER_THREAD)
> + ODP_ABORT("Too many schedqs\n");
> + ts->schedq_list[ts->num_schedq++] = schedq;
> +}
> +
> +static void remove_schedq_from_list(sched_scalable_thread_state_t *ts,
> + sched_queue_t *schedq)
> +{
> + /* Find schedq */
> + for (uint32_t i = 0; i < ts->num_schedq; i++)
> + if (ts->schedq_list[i] == schedq) {
> + /* Move remaining schedqs */
> + for (uint32_t j = i + 1; j < ts->num_schedq; j++)
> + ts->schedq_list[j - 1] = ts->schedq_list[j];
> + ts->num_schedq--;
> + return;
> + }
> + ODP_ABORT("Cannot find schedq\n");
> +}
> +
> +/*******************************************************************************
> + * Scheduler queues
> + ******************************************************************************/
> +#ifndef odp_container_of
> +#define odp_container_of(pointer, type, member) \
> + ((type *)(void *)(((char *)pointer) - offsetof(type, member)))
> +#endif
> +
> +static inline void schedq_init(sched_queue_t *schedq, uint32_t prio)
> +{
> + llqueue_init(&schedq->llq);
> + schedq->prio = prio;
> +}
> +
> +static inline sched_elem_t *schedq_peek(sched_queue_t *schedq)
> +{
> + struct llnode *ptr;
> +
> + ptr = llq_head(&schedq->llq);
> + return odp_container_of(ptr, sched_elem_t, node);
> +}
> +
> +static inline odp_bool_t schedq_cond_pop(sched_queue_t *schedq,
> + sched_elem_t *elem)
> +{
> + return llq_dequeue_cond(&schedq->llq, &elem->node);
> +}
> +
> +static inline void schedq_push(sched_queue_t *schedq, sched_elem_t *elem)
> +{
> + llq_enqueue(&schedq->llq, &elem->node);
> +}
> +
> +static inline odp_bool_t schedq_cond_rotate(sched_queue_t *schedq,
> + sched_elem_t *elem)
> +{
> + return llq_cond_rotate(&schedq->llq, &elem->node);
> +}
> +
> +static inline bool schedq_elem_on_queue(sched_elem_t *elem)
> +{
> + return elem->node.next != NULL;
> +}
> +
> +/*******************************************************************************
> + * Shared metadata btwn scheduler and queue
> + ******************************************************************************/
> +
> +void sched_update_enq(sched_elem_t *q, uint32_t actual)
> +{
> + union {
> + qschedstate_t ss;
> + uint64_t ui;
> + } oss, nss;
> + uint32_t ticket;
> +
> +#ifndef ODP_CONFIG_USE_LLSC
> + oss.ss = q->qschst;
> +#endif
> + /* Update event counter, optionally taking a ticket. */
> + do {
> + ticket = TICKET_INVALID;
> +#ifdef ODP_CONFIG_USE_LLSC
> + oss.ui = ll64((uint64_t *)&q->qschst, __ATOMIC_RELAXED);
> +#endif
> + nss = oss;
> + nss.ss.numevts += actual;
> + if (odp_unlikely(oss.ss.numevts <= 0 && nss.ss.numevts > 0))
> + /* E -> NE transition */
> + if (q->qschst_type != ODP_SCHED_SYNC_ATOMIC ||
> + oss.ss.cur_ticket == oss.ss.nxt_ticket)
> + /* Parallel or ordered queues: always take
> + * ticket.
> + * Atomic queue: only take ticket if one is
> + * immediately available.
> + * Otherwise ticket already taken => queue
> + * processed by some thread.
> + */
> + ticket = nss.ss.nxt_ticket++;
> + /* Else queue already was non-empty. */
> + /* Attempt to update numevts counter and optionally take ticket. */
> +#ifdef ODP_CONFIG_USE_LLSC
> + } while (odp_unlikely(
> + sc64((uint64_t *)&q->qschst, nss.ui, __ATOMIC_RELAXED)));
> +#else
> + } while (!__atomic_compare_exchange(
> + &q->qschst, &oss.ss, &nss.ss,
> + true, __ATOMIC_RELAXED, __ATOMIC_RELAXED));
> +#endif
> +
> + if (odp_unlikely(ticket != TICKET_INVALID)) {
> + /* Wait for our turn to update schedq. */
> + if (odp_unlikely(
> + __atomic_load_n(&q->qschst.cur_ticket,
> + __ATOMIC_ACQUIRE) != ticket)) {
> + SEVL();
> + while (WFE() &&
> + LDXR8(&q->qschst.cur_ticket,
> + __ATOMIC_ACQUIRE) != ticket)
> + DOZE();
> + }
> + /* Enqueue at end of scheduler queue */
> + /* We are here because of empty-to-non-empty transition
> + * This means queue must be pushed to schedq if possible
> + * but we can't do that if it already is on the schedq
> + */
> + if (odp_likely(!schedq_elem_on_queue(q) &&
> + q->pop_deficit == 0)) {
> + /* Queue not already on schedq and no pop deficit means
> + * we can push queue to schedq */
> + schedq_push(q->schedq, q);
> + } else {
> + /* Missed push => cancels one missed pop */
> + q->pop_deficit--;
> + }
> +#ifdef ODP_CONFIG_USE_DMB
> + __atomic_thread_fence(__ATOMIC_RELEASE);
> + __atomic_store_n(&q->qschst.cur_ticket, ticket + 1,
> + __ATOMIC_RELAXED);
> +#else
> + __atomic_store_n(&q->qschst.cur_ticket, ticket + 1,
> + __ATOMIC_RELEASE);
> +#endif
> + }
> + /* Else queue was not empty or atomic queue already busy. */
> +}
> +
> +void sched_update_enq_sp(sched_elem_t *q, uint32_t actual)
> +{
> + qschedstate_t oss, nss;
> + uint32_t ticket;
> +
> + oss = q->qschst;
> + /* Update event counter, optionally taking a ticket. */
> + ticket = TICKET_INVALID;
> + nss = oss;
> + nss.numevts += actual;
> + if (odp_unlikely(oss.numevts <= 0 && nss.numevts > 0)) {
> + /* E -> NE transition */
> + if (q->qschst_type != ODP_SCHED_SYNC_ATOMIC ||
> + oss.cur_ticket == oss.nxt_ticket) {
> + /* Parallel or ordered queues: always take
> + * ticket.
> + * Atomic queue: only take ticket if one is
> + * immediately available. Otherwise ticket already
> + * taken => queue owned/processed by some thread
> + */
> + ticket = nss.nxt_ticket++;
> + }
> + }
> + /* Else queue already was non-empty. */
> + /* Attempt to update numevts counter and optionally take ticket. */
> + q->qschst = nss;
> +
> + if (odp_unlikely(ticket != TICKET_INVALID)) {
> + /* Enqueue at end of scheduler queue */
> + /* We are here because of empty-to-non-empty transition
> + * This means queue must be pushed to schedq if possible
> + * but we can't do that if it already is on the schedq
> + */
> + if (odp_likely(!schedq_elem_on_queue(q) &&
> + q->pop_deficit == 0)) {
> + /* Queue not already on schedq and no pop deficit means
> + * we can push queue to schedq */
> + schedq_push(q->schedq, q);
> + } else {
> + /* Missed push => cancels one missed pop */
> + q->pop_deficit--;
> + }
> + q->qschst.cur_ticket = ticket + 1;
> + }
> + /* Else queue was not empty or atomic queue already busy. */
> +}
> +
> +/* The scheduler is the only entity that performs the dequeue from a queue. */
> +static void sched_update_deq(sched_elem_t *q,
> + uint32_t actual, bool atomic) __attribute__((always_inline));
> +static inline void sched_update_deq(sched_elem_t *q,
> + uint32_t actual, bool atomic)
> +{
> + union {
> + qschedstate_t ss;
> + uint64_t ui;
> + } oss, nss;
> + uint32_t ticket;
> +
> + if (atomic) {
> + qschedstate_t oss, nss;
> + bool pushed = false;
> +
> + /* We own this atomic queue, only we can dequeue from it and
> + * thus decrease numevts. Other threads may enqueue and thus
> + * increase numevts.
> + * This means that numevts can't unexpectedly become 0 and
> + * invalidate a push operation already performed
> + */
> + oss = q->qschst;
> + do {
> + ODP_ASSERT(oss.cur_ticket == sched_ts->ticket);
> + nss = oss;
> + nss.numevts -= actual;
> + if (nss.numevts > 0 && !pushed) {
> + schedq_push(q->schedq, q);
> + pushed = true;
> + }
> + /* Attempt to release ticket expecting our view of
> + * numevts to be correct
> + * Unfortunately nxt_ticket will also be included in
> + * the CAS operation
> + */
> + nss.cur_ticket = sched_ts->ticket + 1;
> + } while (odp_unlikely(!__atomic_compare_exchange(
> + &q->qschst,
> + &oss, &nss,
> + true,
> + __ATOMIC_RELEASE,
> + __ATOMIC_RELAXED)));
> + return;
> + }
> +
> +#ifndef ODP_CONFIG_USE_LLSC
> + oss.ss = q->qschst;
> +#endif
> + do {
> + ticket = TICKET_INVALID;
> +#ifdef ODP_CONFIG_USE_LLSC
> + oss.ui = ll64((uint64_t *)&q->qschst, __ATOMIC_RELAXED);
> +#endif
> + nss = oss;
> + nss.ss.numevts -= actual;
> + nss.ss.wrr_budget -= actual;
> + if ((oss.ss.numevts > 0 && nss.ss.numevts <= 0) ||
> + oss.ss.wrr_budget <= actual) {
> + /* If we have emptied parallel/ordered queue or
> + * exchausted its WRR budget, we need a ticket
> + * for a later pop.
> + */
> + ticket = nss.ss.nxt_ticket++;
> + /* Reset wrr_budget as we might also push the
> + * queue to the schedq.
> + */
> + nss.ss.wrr_budget = CONFIG_WRR_WEIGHT;
> + }
> + /* Attempt to update numevts and optionally take ticket. */
> +#ifdef ODP_CONFIG_USE_LLSC
> + } while (odp_unlikely(
> + sc64((uint64_t *)&q->qschst, nss.ui, __ATOMIC_RELAXED)));
> +#else
> + } while (!__atomic_compare_exchange(
> + &q->qschst, &oss.ss, &nss.ss,
> + true, __ATOMIC_RELAXED, __ATOMIC_RELAXED));
> +#endif
> +
> + if (odp_unlikely(ticket != TICKET_INVALID)) {
> + ODP_ASSERT(q->qschst_type != ODP_SCHED_SYNC_ATOMIC);
> + /* Wait for our turn to update schedq. */
> + if (odp_unlikely(
> + __atomic_load_n(&q->qschst.cur_ticket,
> + __ATOMIC_ACQUIRE) != ticket)) {
> + SEVL();
> + while (WFE() &&
> + LDXR8(&q->qschst.cur_ticket,
> + __ATOMIC_ACQUIRE) != ticket)
> + DOZE();
> + }
> + /* We are here because of non-empty-to-empty transition or
> + * WRR budget exhausted
> + * This means the queue must be popped from the schedq, now or
> + * later
> + * If there was no NE->E transition but instead the WRR budget
> + * was exhausted, the queue needs to be moved (popped and
> + * pushed) to the tail of the schedq
> + */
> + if (oss.ss.numevts > 0 && nss.ss.numevts <= 0) {
> + /* NE->E transition, need to pop */
> + if (!schedq_elem_on_queue(q) ||
> + !schedq_cond_pop(q->schedq, q)) {
> + /* Queue not at head, failed to dequeue
> + * Missed a pop.
> + */
> + q->pop_deficit++;
> + }
> + } else {
> + /* WRR budget exhausted
> + * Need to move queue to tail of schedq if possible
> + */
> + if (odp_likely(schedq_elem_on_queue(q))) {
> + /* Queue is on schedq, try to move it to
> + * the tail
> + */
> + (void)schedq_cond_rotate(q->schedq, q);
> + }
> + /* Else queue not on schedq or not at head of schedq
> + * No pop => no push
> + */
> + }
> +#ifdef ODP_CONFIG_USE_DMB
> + __atomic_thread_fence(__ATOMIC_RELEASE);
> + __atomic_store_n(&q->qschst.cur_ticket, ticket + 1,
> + __ATOMIC_RELAXED);
> +#else
> + __atomic_store_n(&q->qschst.cur_ticket, ticket + 1,
> + __ATOMIC_RELEASE);
> +#endif
> + }
> +}
> +
> +static void sched_update_deq_sc(sched_elem_t *q,
> + uint32_t actual, bool atomic) __attribute__((always_inline));
> +static inline void sched_update_deq_sc(sched_elem_t *q,
> + uint32_t actual, bool atomic)
> +{
> + qschedstate_t oss, nss;
> + uint32_t ticket;
> +
> + if (atomic) {
> + ODP_ASSERT(q->qschst.cur_ticket == sched_ts->ticket);
> + ODP_ASSERT(q->qschst.cur_ticket != q->qschst.nxt_ticket);
> + q->qschst.numevts -= actual;
> + q->qschst.cur_ticket = sched_ts->ticket + 1;
> + if (q->qschst.numevts > 0)
> + schedq_push(q->schedq, q);
> + return;
> + }
> +
> + oss = q->qschst;
> + ticket = TICKET_INVALID;
> + nss = oss;
> + nss.numevts -= actual;
> + nss.wrr_budget -= actual;
> + if ((oss.numevts > 0 && nss.numevts <= 0) || oss.wrr_budget <= actual) {
> + /* If we emptied the queue or
> + * if we have served the maximum number of events
> + * then we need a ticket for a later pop.
> + */
> + ticket = nss.nxt_ticket++;
> + /* Also reset wrr_budget as we might also push the
> + * queue to the schedq.
> + */
> + nss.wrr_budget = CONFIG_WRR_WEIGHT;
> + }
> + q->qschst = nss;
> +
> + if (ticket != TICKET_INVALID) {
> + if (oss.numevts > 0 && nss.numevts <= 0) {
> + /* NE->E transition, need to pop */
> + if (!schedq_elem_on_queue(q) ||
> + !schedq_cond_pop(q->schedq, q)) {
> + /* Queue not at head, failed to dequeue.
> + * Missed a pop.
> + */
> + q->pop_deficit++;
> + }
> + } else {
> + /* WRR budget exhausted
> + * Need to move queue to tail of schedq if possible
> + */
> + if (odp_likely(schedq_elem_on_queue(q))) {
> + /* Queue is on schedq, try to move it to
> + * the tail
> + */
> + (void)schedq_cond_rotate(q->schedq, q);
> + }
> + /* Else queue not on schedq or not at head of schedq
> + * No pop => no push
> + */
> + }
> + q->qschst.cur_ticket = ticket + 1;
> + }
> +}
> +
> +static inline void sched_update_popd_sc(sched_elem_t *elem)
> +{
> + if (elem->pop_deficit != 0 &&
> + schedq_elem_on_queue(elem) &&
> + schedq_cond_pop(elem->schedq, elem))
> + elem->pop_deficit--;
> +}
> +
> +static inline void sched_update_popd(sched_elem_t *elem)
> +{
> + uint32_t ticket = __atomic_fetch_add(&elem->qschst.nxt_ticket,
> + 1,
> + __ATOMIC_RELAXED);
> + if (odp_unlikely(__atomic_load_n(&elem->qschst.cur_ticket,
> + __ATOMIC_ACQUIRE) != ticket)) {
> + SEVL();
> + while (WFE() && LDXR8(&elem->qschst.cur_ticket,
> + __ATOMIC_ACQUIRE) != ticket)
> + DOZE();
> + }
> + sched_update_popd_sc(elem);
> +#ifdef ODP_CONFIG_USE_DMB
> + __atomic_thread_fence(__ATOMIC_RELEASE);
> + __atomic_store_n(&elem->qschst.cur_ticket,
> + ticket + 1,
> + __ATOMIC_RELAXED);
> +#else
> + __atomic_store_n(&elem->qschst.cur_ticket,
> + ticket + 1,
> + __ATOMIC_RELEASE);
> +#endif
> +}
> +
> +sched_queue_t *schedq_from_sched_group(odp_schedule_group_t grp, uint32_t prio)
> +{
> + uint32_t sgi;
> + sched_group_t *sg;
> + uint32_t x;
> +
> + ODP_ASSERT(grp >= 0 && grp <= (odp_schedule_group_t)MAX_SCHED_GROUP);
> + ODP_ASSERT((sg_free & (1ULL << grp)) == 0);
> + ODP_ASSERT(prio < ODP_SCHED_PRIO_NUM);
> +
> + sgi = grp;
> + sg = sg_vec[sgi];
> +
> + /* Use xcount to spread queues over the xfactor schedq's
> + * per priority.
> + */
> + x = __atomic_fetch_add(&sg->xcount[prio], 1, __ATOMIC_RELAXED);
> + if (x == 0) {
> + /* First ODP queue for this priority
> + * Notify all threads in sg->thr_wanted that they
> + * should join.
> + */
> + sched_group_mask_t thrds = sg->thr_wanted;
> +
> + while (!bitset_is_null(thrds)) {
> + uint32_t thr;
> +
> + thr = bitset_ffs(thrds) - 1;
> + thrds = bitset_clr(thrds, thr);
> + /* Notify the thread about membership in this
> + * group/priority.
> + */
> + atom_bitset_set(&thread_state[thr].sg_wanted[prio],
> + sgi, __ATOMIC_RELEASE);
> + __atomic_store_n(&thread_state[thr].sg_sem, 1,
> + __ATOMIC_RELEASE);
> + }
> + }
> + return &sg->schedq[prio * sg->xfactor + x % sg->xfactor];
> +}
> +
> +void sched_group_xcount_dec(odp_schedule_group_t grp, uint32_t prio)
> +{
> + uint32_t sgi;
> + sched_group_t *sg;
> + uint32_t x;
> +
> + ODP_ASSERT(grp >= 0 && grp <= (odp_schedule_group_t)MAX_SCHED_GROUP);
> + ODP_ASSERT((sg_free & (1ULL << grp)) == 0);
> + ODP_ASSERT(prio < ODP_SCHED_PRIO_NUM);
> +
> + sgi = grp;
> + sg = sg_vec[sgi];
> + x = __atomic_sub_fetch(&sg->xcount[prio], 1, __ATOMIC_RELAXED);
> +
> + if (x == 0) {
> + /* Last ODP queue for this priority
> + * Notify all threads in sg->thr_wanted that they
> + * should leave.
> + */
> + sched_group_mask_t thrds = sg->thr_wanted;
> +
> + while (!bitset_is_null(thrds)) {
> + uint32_t thr;
> +
> + thr = bitset_ffs(thrds) - 1;
> + thrds = bitset_clr(thrds, thr);
> + /* Notify the thread about membership in this
> + * group/priority.
> + */
> + atom_bitset_clr(&thread_state[thr].sg_wanted[prio],
> + sgi, __ATOMIC_RELEASE);
> + __atomic_store_n(&thread_state[thr].sg_sem, 1,
> + __ATOMIC_RELEASE);
> + }
> + }
> +}
> +
> +static void update_sg_membership(sched_scalable_thread_state_t *ts)
> +{
> + uint32_t p;
> + sched_group_mask_t sg_wanted;
> + sched_group_mask_t added;
> + sched_group_mask_t removed;
> + uint32_t sgi;
> + sched_group_t *sg;
> + uint32_t x;
> +
> + for (p = 0; p < ODP_SCHED_PRIO_NUM; p++) {
> + sg_wanted = __atomic_load_n(&ts->sg_wanted[p],
> + __ATOMIC_ACQUIRE);
> + if (!bitset_is_eql(ts->sg_actual[p], sg_wanted)) {
> + /* Our sched_group membership has changed */
> + added = bitset_andn(sg_wanted, ts->sg_actual[p]);
> + while (!bitset_is_null(added)) {
> + sgi = bitset_ffs(added) - 1;
> + sg = sg_vec[sgi];
> + for (x = 0; x < sg->xfactor; x++) {
> + /* Include our thread index to shift
> + * (rotate) the order of schedq's
> + */
> + insert_schedq_in_list(ts,
> + &sg->schedq[p * sg->xfactor +
> + (x + ts->tidx) % sg->xfactor]);
> + }
> + atom_bitset_set(&sg->thr_actual[p], ts->tidx,
> + __ATOMIC_RELAXED);
> + added = bitset_clr(added, sgi);
> + }
> + removed = bitset_andn(ts->sg_actual[p], sg_wanted);
> + while (!bitset_is_null(removed)) {
> + sgi = bitset_ffs(removed) - 1;
> + sg = sg_vec[sgi];
> + for (x = 0; x < sg->xfactor; x++) {
> + remove_schedq_from_list(ts,
> + &sg->schedq[p * sg->xfactor + x]);
> + }
> + atom_bitset_clr(&sg->thr_actual[p], ts->tidx,
> + __ATOMIC_RELAXED);
> + removed = bitset_clr(removed, sgi);
> + }
> + ts->sg_actual[p] = sg_wanted;
> + }
> + }
> +}
> +
> +/*******************************************************************************
> + * Scheduler
> + ******************************************************************************/
> +
> +static inline void _schedule_release_atomic(sched_scalable_thread_state_t *ts)
> +{
> +#ifdef CONFIG_QSCHST_LOCK
> + sched_update_deq_sc(ts->atomq, ts->dequeued, true);
> + ODP_ASSERT(ts->atomq->qschst.cur_ticket != ts->ticket);
> + ODP_ASSERT(ts->atomq->qschst.cur_ticket ==
> + ts->atomq->qschst.nxt_ticket);
> +#else
> + sched_update_deq(ts->atomq, ts->dequeued, true);
> +#endif
> + ts->atomq = NULL;
> + ts->ticket = TICKET_INVALID;
> +}
> +
> +static inline void _schedule_release_ordered(sched_scalable_thread_state_t *ts)
> +{
> + ts->out_of_order = false;
> + rctx_release(ts->rctx);
> + ts->rctx = NULL;
> +}
> +
> +static void pktin_poll(sched_scalable_thread_state_t *ts)
> +{
> + uint32_t i, tag, hi, npolls = 0;
> + int pktio_index, queue_index;
> +
> + hi = __atomic_load_n(&pktin_hi, __ATOMIC_RELAXED);
> + if (hi == 0)
> + return;
> +
> + for (i = ts->pktin_next; npolls != hi; i = (i + 1) % hi, npolls++) {
> + tag = __atomic_load_n(&pktin_tags[i], __ATOMIC_RELAXED);
> + if (!TAG_IS_READY(tag))
> + continue;
> + if (!__atomic_compare_exchange_n(&pktin_tags[i], &tag,
> + tag | TAG_BUSY,
> + true,
> + __ATOMIC_ACQUIRE,
> + __ATOMIC_RELAXED))
> + continue;
> + /* Tag grabbed */
> + pktio_index = TAG_2_PKTIO(tag);
> + queue_index = TAG_2_QUEUE(tag);
> + if (odp_unlikely(sched_cb_pktin_poll(pktio_index,
> + 1, &queue_index))) {
> + /* Pktio stopped or closed
> + * Remove tag from pktin_tags
> + */
> + __atomic_store_n(&pktin_tags[i],
> + TAG_EMPTY, __ATOMIC_RELAXED);
> + __atomic_fetch_sub(&pktin_num,
> + 1, __ATOMIC_RELEASE);
> + /* Call stop_finalize when all queues
> + * of the pktio have been removed
> + */
> + if (__atomic_sub_fetch(&pktin_count[pktio_index], 1,
> + __ATOMIC_RELAXED) == 0)
> + sched_cb_pktio_stop_finalize(pktio_index);
> + } else {
> + /* We don't know whether any packets were found and enqueued
> + * Write back original tag value to release pktin queue
> + */
> + __atomic_store_n(&pktin_tags[i], tag, __ATOMIC_RELAXED);
> + /* Do not iterate through all pktin queues every time */
> + if ((ts->pktin_poll_cnts & 0xf) != 0)
> + break;
> + }
> + }
> + ODP_ASSERT(i < hi);
> + ts->pktin_poll_cnts++;
> + ts->pktin_next = i;
> +}
> +
> +static int _schedule(odp_queue_t *from, odp_event_t ev[], int num_evts)
> +{
> + sched_scalable_thread_state_t *ts;
> + sched_elem_t *atomq;
> + int num;
> + uint32_t i;
> +
> + ts = sched_ts;
> + atomq = ts->atomq;
> +
> + /* Once an atomic queue has been scheduled to a thread, it will stay
> + * on that thread until empty or 'rotated' by WRR
> + */
> + if (atomq != NULL) {
> + ODP_ASSERT(ts->ticket != TICKET_INVALID);
> +#ifdef CONFIG_QSCHST_LOCK
> + LOCK(&atomq->qschlock);
> +#endif
> +dequeue_atomic:
> + ODP_ASSERT(ts->ticket == atomq->qschst.cur_ticket);
> + ODP_ASSERT(ts->ticket != atomq->qschst.nxt_ticket);
> + /* Atomic queues can be dequeued without lock since this thread
> + * has the only reference to the atomic queue being processed.
> + */
> + if (ts->dequeued < atomq->qschst.wrr_budget) {
> + num = _odp_queue_deq_sc(atomq, ev, num_evts);
> + if (odp_likely(num != 0)) {
> +#ifdef CONFIG_QSCHST_LOCK
> + UNLOCK(&atomq->qschlock);
> +#endif
> + ts->dequeued += num;
> + /* Allow this thread to continue to 'own' this
> + * atomic queue until all events have been
> + * processed and the thread re-invokes the
> + * scheduler.
> + */
> + if (from)
> + *from = queue_get_handle(
> + (queue_entry_t *)atomq);
> + return num;
> + }
> + }
> + /* Atomic queue was empty or interrupted by WRR, release it. */
> + _schedule_release_atomic(ts);
> +#ifdef CONFIG_QSCHST_LOCK
> + UNLOCK(&atomq->qschlock);
> +#endif
> + }
> +
> + /* Release any previous reorder context. */
> + if (ts->rctx != NULL)
> + _schedule_release_ordered(ts);
> +
> + /* Check for and perform any scheduler group updates. */
> + if (odp_unlikely(__atomic_load_n(&ts->sg_sem, __ATOMIC_RELAXED) != 0)) {
> + (void)__atomic_load_n(&ts->sg_sem, __ATOMIC_ACQUIRE);
> + ts->sg_sem = 0;
> + update_sg_membership(ts);
> + }
> +
> + /* Scan our schedq list from beginning to end */
> + for (i = 0; i < ts->num_schedq; i++) {
> + sched_queue_t *schedq = ts->schedq_list[i];
> + sched_elem_t *elem;
> +restart_same:
> + elem = schedq_peek(schedq);
> + if (odp_unlikely(elem == NULL)) {
> + /* Schedq empty, look at next one. */
> + continue;
> + }
> +
> + if (elem->cons_type == ODP_SCHED_SYNC_ATOMIC) {
> + /* Dequeue element only if it is still at head
> + * of schedq.
> + */
> + if (odp_unlikely(!schedq_cond_pop(schedq, elem))) {
> + /* Queue not at head of schedq anymore, some
> + * other thread popped it.
> + */
> + goto restart_same;
> + }
> + ts->atomq = atomq = elem;
> + ts->dequeued = 0;
> +#ifdef CONFIG_QSCHST_LOCK
> + LOCK(&atomq->qschlock);
> + ts->ticket = atomq->qschst.nxt_ticket++;
> + ODP_ASSERT(atomq->qschst.cur_ticket == ts->ticket);
> +#else
> + /* Dequeued atomic queue from the schedq, only we
> + * can process it and any qschst updates are our
> + * responsibility.
> + */
> + /* The ticket taken below will signal producers */
> + ts->ticket = __atomic_fetch_add(
> + &atomq->qschst.nxt_ticket, 1, __ATOMIC_RELAXED);
> + while (__atomic_load_n(
> + &atomq->qschst.cur_ticket,
> + __ATOMIC_ACQUIRE) != ts->ticket) {
> + /* No need to use WFE, spinning here seems
> + * very infrequent.
> + */
> + odp_cpu_pause();
> + }
> +#endif
> + goto dequeue_atomic;
> + } else if (elem->cons_type == ODP_SCHED_SYNC_PARALLEL) {
> +#ifdef CONFIG_QSCHST_LOCK
> + LOCK(&elem->qschlock);
> + num = _odp_queue_deq_sc(elem, ev, num_evts);
> + if (odp_likely(num != 0)) {
> + sched_update_deq_sc(elem, num, false);
> + UNLOCK(&elem->qschlock);
> + if (from)
> + *from =
> + queue_get_handle((queue_entry_t *)elem);
> + return num;
> + }
> + UNLOCK(&elem->qschlock);
> +#else
> + num = _odp_queue_deq(elem, ev, num_evts);
> + if (odp_likely(num != 0)) {
> + sched_update_deq(elem, num, false);
> + if (from)
> + *from =
> + queue_get_handle((queue_entry_t *)elem);
> + return num;
> + }
> +#endif
> + } else if (elem->cons_type == ODP_SCHED_SYNC_ORDERED) {
> + reorder_window_t *rwin;
> + reorder_context_t *rctx;
> + uint32_t sn;
> + uint32_t idx;
> + int ret;
> +
> + /* The ordered queue has a reorder window so requires
> + * order restoration. We must use a reorder context to
> + * collect all outgoing events. Ensure there is at least
> + * one available reorder context.
> + */
> + if (odp_unlikely(bitset_is_null(ts->priv_rvec_free))) {
> + ts->priv_rvec_free = atom_bitset_xchg(
> + &ts->rvec_free, 0,
> + __ATOMIC_RELAXED);
> + if (odp_unlikely(bitset_is_null(
> + ts->priv_rvec_free))) {
> + /* No free reorder contexts for
> + * this thread. Look at next schedq,
> + * hope we find non-ordered queue.
> + */
> + continue;
> + }
> + }
> + /* rwin_reserve and odp_queue_deq must be atomic or
> + * there will be a potential race condition.
> + * Allocate a slot in the reorder window.
> + */
> + rwin = queue_get_rwin((queue_entry_t *)elem);
> + ODP_ASSERT(rwin != NULL);
> + if (odp_unlikely(!rwin_reserve(rwin, &sn))) {
> + /* Reorder window full */
> + /* Look at next schedq, find other queue */
> + continue;
> + }
> + /* Wait for our turn to dequeue */
> + if (odp_unlikely(__atomic_load_n(&rwin->turn,
> + __ATOMIC_ACQUIRE) != sn)) {
> + SEVL();
> + while (WFE() &&
> + LDXR32(&rwin->turn, __ATOMIC_ACQUIRE)
> + != sn)
> + DOZE();
> + }
> +#ifdef CONFIG_QSCHST_LOCK
> + LOCK(&elem->qschlock);
> +#endif
> + ret = _odp_queue_deq_sc(elem, ev, num_evts);
> +#ifdef ODP_CONFIG_USE_DMB
> + /* Wait for prod_read write in _odp_queue_dequeue_sc()
> + * to complete.
> + */
> + __asm volatile("dmb ishst" ::: "memory");
> + /* before we signal the next consumer */
> + rwin->turn = sn + 1;
> +#else
> + __atomic_store_n(&rwin->turn, sn + 1, __ATOMIC_RELEASE);
> +#endif
> + /* Find and initialise an unused reorder context. */
> + idx = bitset_ffs(ts->priv_rvec_free) - 1;
> + ts->priv_rvec_free =
> + bitset_clr(ts->priv_rvec_free, idx);
> + rctx = &ts->rvec[idx];
> + /* Need to initialise reorder context or we can't
> + * release it later.
> + */
> + rctx_init(rctx, idx, rwin, sn);
> +
> + /* Was dequeue successful? */
> + if (odp_likely(ret != 0)) {
> + /* Perform scheduler related updates */
> +#ifdef CONFIG_QSCHST_LOCK
> + sched_update_deq_sc(elem, ret,
> + /*atomic=*/false);
> + UNLOCK(&elem->qschlock);
> +#else
> + sched_update_deq(elem, ret, /*atomic=*/false);
> +#endif
> +
> + /* Are we in-order or out-of-order? */
> + ts->out_of_order = sn != rwin->hc.head;
> +
> + ts->rctx = rctx;
> + if (from)
> + *from = queue_get_handle(
> + (queue_entry_t *)elem);
> + return ret;
> + }
> +#ifdef CONFIG_QSCHST_LOCK
> + UNLOCK(&elem->qschlock);
> +#endif
> + /* Since a slot was reserved in the reorder window,
> + * the reorder context needs to be released and
> + * inserted into the reorder window.
> + */
> + rctx_release(rctx);
> + ODP_ASSERT(ts->rctx == NULL);
> + }
> + /* Dequeue from parallel/ordered queue failed
> + * Check if we have a queue at the head of the schedq that needs
> + * to be popped
> + */
> + if (odp_unlikely(__atomic_load_n(&elem->pop_deficit,
> + __ATOMIC_RELAXED) != 0)) {
> +#ifdef CONFIG_QSCHST_LOCK
> + LOCK(&elem->qschlock);
> + sched_update_popd_sc(elem);
> + UNLOCK(&elem->qschlock);
> +#else
> + sched_update_popd(elem);
> +#endif
> + }
> + }
> +
> + pktin_poll(ts);
> + return 0;
> +}
> +
> +/******************************************************************************/
> +
> +static void schedule_order_lock(unsigned lock_index)
> +{
> + struct reorder_context *rctx = sched_ts->rctx;
> +
> + if (odp_unlikely(rctx == NULL ||
> + rctx->rwin == NULL ||
> + lock_index >= rctx->rwin->lock_count)) {
> + ODP_ERR("Invalid call to odp_schedule_order_lock\n");
> + return;
> + }
> + if (odp_unlikely(__atomic_load_n(&rctx->rwin->olock[lock_index],
> + __ATOMIC_ACQUIRE) != rctx->sn)) {
> + SEVL();
> + while (WFE() &&
> + LDXR32(&rctx->rwin->olock[lock_index],
> + __ATOMIC_ACQUIRE) != rctx->sn)
> + DOZE();
> + }
> +}
> +
> +static void schedule_order_unlock(unsigned lock_index)
> +{
> + struct reorder_context *rctx;
> +
> + rctx = sched_ts->rctx;
> + if (odp_unlikely(rctx == NULL ||
> + rctx->rwin == NULL ||
> + lock_index >= rctx->rwin->lock_count ||
> + rctx->rwin->olock[lock_index] != rctx->sn)) {
> + ODP_ERR("Invalid call to odp_schedule_order_unlock\n");
> + return;
> + }
> +#ifdef ODP_CONFIG_USE_DMB
> + __atomic_thread_fence(__ATOMIC_RELEASE);
> + __atomic_store_n(&rctx->rwin->olock[lock_index],
> + rctx->sn + 1,
> + __ATOMIC_RELAXED);
> +#else
> + __atomic_store_n(&rctx->rwin->olock[lock_index],
> + rctx->sn + 1,
> + __ATOMIC_RELEASE);
> +#endif
> + rctx->olock_flags |= 1U << lock_index;
> +}
> +
> +static void schedule_release_atomic(void)
> +{
> + sched_scalable_thread_state_t *ts;
> +
> + ts = sched_ts;
> + if (odp_likely(ts->atomq != NULL)) {
> +#ifdef CONFIG_QSCHST_LOCK
> + sched_elem_t *atomq;
> +
> + atomq = ts->atomq;
> + LOCK(&atomq->qschlock);
> +#endif
> + _schedule_release_atomic(ts);
> +#ifdef CONFIG_QSCHST_LOCK
> + UNLOCK(&atomq->qschlock);
> +#endif
> + }
> +}
> +
> +static void schedule_release_ordered(void)
> +{
> + sched_scalable_thread_state_t *ts;
> +
> + ts = sched_ts;
> + if (ts->rctx != NULL)
> + _schedule_release_ordered(ts);
> +}
> +
> +static int schedule_multi(odp_queue_t *from, uint64_t wait, odp_event_t ev[],
> + int num)
> +{
> + sched_scalable_thread_state_t *ts;
> + int n;
> + odp_time_t start;
> + odp_time_t delta;
> + odp_time_t deadline;
> +
> + ts = sched_ts;
> + if (odp_unlikely(ts->pause)) {
> + if (ts->atomq != NULL) {
> +#ifdef CONFIG_QSCHST_LOCK
> + sched_elem_t *atomq;
> +
> + atomq = ts->atomq;
> + LOCK(&atomq->qschlock);
> +#endif
> + _schedule_release_atomic(ts);
> +#ifdef CONFIG_QSCHST_LOCK
> + UNLOCK(&atomq->qschlock);
> +#endif
> + } else if (ts->rctx != NULL) {
> + _schedule_release_ordered(ts);
> + }
> + return 0;
> + }
> +
> + if (wait == ODP_SCHED_NO_WAIT)
> + return _schedule(from, ev, num);
> +
> + if (wait == ODP_SCHED_WAIT) {
> + for (;;) {
> + n = _schedule(from, ev, num);
> + if (odp_likely(n > 0))
> + return n;
> + }
> + }
> +
> + start = odp_time_local();
> +
> + n = _schedule(from, ev, num);
> + if (odp_likely(n > 0))
> + return n;
> +
> + delta = odp_time_local_from_ns(wait);
> + deadline = odp_time_sum(start, delta);
> +
> + while (odp_time_cmp(deadline, odp_time_local()) > 0) {
> + n = _schedule(from, ev, num);
> + if (odp_likely(n > 0))
> + return n;
> + }
> +
> + return 0;
> +}
> +
> +static odp_event_t schedule(odp_queue_t *from, uint64_t wait)
> +{
> + odp_event_t ev = ODP_EVENT_INVALID;
> + const int num = 1;
> + sched_scalable_thread_state_t *ts;
> + int n;
> + odp_time_t start;
> + odp_time_t delta;
> + odp_time_t deadline;
> +
> + ts = sched_ts;
> + if (odp_unlikely(ts->pause)) {
> + if (ts->atomq != NULL) {
> +#ifdef CONFIG_QSCHST_LOCK
> + sched_elem_t *atomq;
> +
> + atomq = ts->atomq;
> + LOCK(&atomq->qschlock);
> +#endif
> + _schedule_release_atomic(ts);
> +#ifdef CONFIG_QSCHST_LOCK
> + UNLOCK(&atomq->qschlock);
> +#endif
> + } else if (ts->rctx != NULL) {
> + _schedule_release_ordered(ts);
> + }
> + return ev;
> + }
> +
> + if (wait == ODP_SCHED_NO_WAIT) {
> + (void)_schedule(from, &ev, num);
> + return ev;
> + }
> +
> + if (wait == ODP_SCHED_WAIT) {
> + for (;;) {
> + n = _schedule(from, &ev, num);
> + if (odp_likely(n > 0))
> + return ev;
> + }
> + }
> +
> + start = odp_time_local();
> +
> + n = _schedule(from, &ev, num);
> + if (odp_likely(n > 0))
> + return ev;
> +
> + delta = odp_time_local_from_ns(wait);
> + deadline = odp_time_sum(start, delta);
> +
> + while (odp_time_cmp(deadline, odp_time_local()) > 0) {
> + n = _schedule(from, &ev, num);
> + if (odp_likely(n > 0))
> + return ev;
> + }
> +
> + return ev;
> +}
> +
> +static void schedule_pause(void)
> +{
> + sched_ts->pause = true;
> +}
> +
> +static void schedule_resume(void)
> +{
> + sched_ts->pause = false;
> +}
> +
> +static uint64_t schedule_wait_time(uint64_t ns)
> +{
> + return ns;
> +}
> +
> +static int schedule_num_prio(void)
> +{
> + return ODP_SCHED_PRIO_NUM;
> +}
> +
> +static int schedule_group_update(sched_group_t *sg,
> + uint32_t sgi,
> + const odp_thrmask_t *mask,
> + int join_leave)
> +{
> + int thr;
> + uint32_t p;
> +
> + /* Internal function, do not validate inputs */
> +
> + /* Notify relevant threads about the change */
> + thr = odp_thrmask_first(mask);
> + while (0 <= thr) {
> + /* Add thread to scheduler group's wanted thread mask */
> + if (join_leave == SCHED_GROUP_JOIN)
> + atom_bitset_set(&sg->thr_wanted, thr, __ATOMIC_RELAXED);
> + else
> + atom_bitset_clr(&sg->thr_wanted, thr, __ATOMIC_RELAXED);
> + for (p = 0; p < ODP_SCHED_PRIO_NUM; p++) {
> + if (sg->xcount[p] != 0) {
> + /* This priority level has ODP queues
> + * Notify the thread about membership in
> + * this group/priority
> + */
> + if (join_leave == SCHED_GROUP_JOIN)
> + atom_bitset_set(
> + &thread_state[thr].sg_wanted[p],
> + sgi,
> + __ATOMIC_RELEASE);
> + else
> + atom_bitset_clr(
> + &thread_state[thr].sg_wanted[p],
> + sgi,
> + __ATOMIC_RELEASE);
> + __atomic_store_n(&thread_state[thr].sg_sem,
> + 1,
> + __ATOMIC_RELEASE);
> + }
> + }
> + thr = odp_thrmask_next(mask, thr);
> + }
> +
> + return 0;
> +}
> +
> +static int _schedule_group_thrmask(sched_group_t *sg, odp_thrmask_t *mask)
> +{
> + bitset_t bs;
> + uint32_t bit;
> +
> + /* Internal function, do not validate inputs */
> +
> + odp_thrmask_zero(mask);
> + bs = sg->thr_wanted;
> + while (!bitset_is_null(bs)) {
> + bit = bitset_ffs(bs) - 1;
> + bs = bitset_clr(bs, bit);
> + odp_thrmask_set(mask, bit);
> + }
> +
> + return 0;
> +}
> +
> +static odp_schedule_group_t schedule_group_create(const char *name,
> + const odp_thrmask_t *mask)
> +{
> + odp_shm_t shm;
> + uint32_t sgi;
> + sched_group_mask_t free;
> + uint32_t xfactor;
> + sched_group_t *sg;
> + uint32_t p;
> + uint32_t x;
> +
> + /* Validate inputs */
> + if (mask == NULL)
> + ODP_ABORT("mask is NULL\n");
> +
> + odp_spinlock_lock(&sched_grp_lock);
> +
> + /* Allocate a scheduler group */
> + free = __atomic_load_n(&sg_free, __ATOMIC_RELAXED);
> + do {
> + /* All sched_groups in use */
> + if (bitset_is_null(free))
> + goto no_free_sched_group;
> +
> + sgi = bitset_ffs(free) - 1;
> + /* All sched_groups in use */
> + if (sgi >= MAX_SCHED_GROUP)
> + goto no_free_sched_group;
> + } while (!__atomic_compare_exchange_n(&sg_free,
> + &free,
> + bitset_clr(free, sgi),
> + true,
> + __ATOMIC_ACQUIRE,
> + __ATOMIC_ACQUIRE));
> +
> + /* Compute xfactor (spread factor) from the number of threads
> + * present in the thread mask. Preferable this would be an
> + * explicit parameter.
> + */
> + xfactor = odp_thrmask_count(mask);
> + if (xfactor < 1)
> + xfactor = 1;
> +
> + shm = odp_shm_reserve(name ? name : "",
> + (sizeof(sched_group_t) +
> + (ODP_SCHED_PRIO_NUM * xfactor - 1) *
> + sizeof(sched_queue_t)),
> + ODP_CACHE_LINE_SIZE,
> + ODP_SHM_PROC);
> + if (ODP_SHM_INVALID == shm)
> + goto shm_reserve_failed;
> +
> + sg = (sched_group_t *) odp_shm_addr(shm);
> + if (sg == NULL)
> + goto shm_addr_failed;
> +
> + strncpy(sg->name, name ? name : "", ODP_SCHED_GROUP_NAME_LEN - 1);
> + sg->shm = shm;
> + sg_vec[sgi] = sg;
> + memset(sg->thr_actual, 0, sizeof(sg->thr_actual));
> + sg->thr_wanted = bitset_null();
> + sg->xfactor = xfactor;
> + for (p = 0; p < ODP_SCHED_PRIO_NUM; p++) {
> + sg->xcount[p] = 0;
> + for (x = 0; x < xfactor; x++)
> + schedq_init(&sg->schedq[p * xfactor + x], p);
> + }
> + if (odp_thrmask_count(mask) != 0)
> + schedule_group_update(sg, sgi, mask, SCHED_GROUP_JOIN);
> +
> + odp_spinlock_unlock(&sched_grp_lock);
> +
> + return (odp_schedule_group_t) (sgi);
> +
> +shm_addr_failed:
> + odp_shm_free(shm);
> +
> +shm_reserve_failed:
> + /* Free the allocated group index */
> + atom_bitset_set(&sg_free, sgi, __ATOMIC_RELAXED);
> +
> +no_free_sched_group:
> + odp_spinlock_unlock(&sched_grp_lock);
> +
> + return ODP_SCHED_GROUP_INVALID;
> +}
> +
> +static int schedule_group_destroy(odp_schedule_group_t group)
> +{
> + uint32_t sgi;
> + sched_group_t *sg;
> + uint32_t p;
> + int ret = 0;
> +
> + /* Validate inputs */
> + if (group < 0 && group >= (odp_schedule_group_t)MAX_SCHED_GROUP) {
> + ret = -1;
> + goto invalid_group;
> + }
> +
> + if (sched_ts &&
> + odp_unlikely(__atomic_load_n(&sched_ts->sg_sem,
> + __ATOMIC_RELAXED) != 0)) {
> + (void)__atomic_load_n(&sched_ts->sg_sem,
> + __ATOMIC_ACQUIRE);
> + sched_ts->sg_sem = 0;
> + update_sg_membership(sched_ts);
> + }
> + odp_spinlock_lock(&sched_grp_lock);
> +
> + sgi = (uint32_t)group;
> + if (bitset_is_set(sg_free, sgi)) {
> + ret = -1;
> + goto group_not_found;
> + }
> +
> + sg = sg_vec[sgi];
> + /* First ensure all threads have processed group_join/group_leave
> + * requests.
> + */
> + for (p = 0; p < ODP_SCHED_PRIO_NUM; p++) {
> + if (sg->xcount[p] != 0) {
> + bitset_t wanted = atom_bitset_load(
> + &sg->thr_wanted, __ATOMIC_RELAXED);
> +
> + SEVL();
> + while (WFE() &&
> + !bitset_is_eql(wanted,
> + bitset_ldex(&sg->thr_actual[p],
> + __ATOMIC_RELAXED)))
> + DOZE();
> + }
> + /* Else ignore because no ODP queues on this prio */
> + }
> +
> + /* Check if all threads/queues have left the group */
> + for (p = 0; p < ODP_SCHED_PRIO_NUM; p++) {
> + if (!bitset_is_null(sg->thr_actual[p])) {
> + ODP_ERR("Group has threads\n");
> + ret = -1;
> + goto thrd_q_present_in_group;
> + }
> + if (sg->xcount[p] != 0) {
> + ODP_ERR("Group has queues\n");
> + ret = -1;
> + goto thrd_q_present_in_group;
> + }
> + }
> +
> + odp_shm_free(sg->shm);
> + sg_vec[sgi] = NULL;
> + atom_bitset_set(&sg_free, sgi, __ATOMIC_RELEASE);
> +
> + odp_spinlock_unlock(&sched_grp_lock);
> +
> + return ret;
> +
> +thrd_q_present_in_group:
> +
> +group_not_found:
> + odp_spinlock_unlock(&sched_grp_lock);
> +
> +invalid_group:
> +
> + return ret;
> +}
> +
> +static odp_schedule_group_t schedule_group_lookup(const char *name)
> +{
> + uint32_t sgi;
> + odp_schedule_group_t group;
> +
> + /* Validate inputs */
> + if (name == NULL)
> + ODP_ABORT("name or mask is NULL\n");
> +
> + group = ODP_SCHED_GROUP_INVALID;
> +
> + odp_spinlock_lock(&sched_grp_lock);
> +
> + /* Scan through the schedule group array */
> + for (sgi = 0; sgi < MAX_SCHED_GROUP; sgi++) {
> + if ((sg_vec[sgi] != NULL) &&
> + (strncmp(name, sg_vec[sgi]->name,
> + ODP_SCHED_GROUP_NAME_LEN) == 0)) {
> + group = (odp_schedule_group_t)sgi;
> + break;
> + }
> + }
> +
> + odp_spinlock_unlock(&sched_grp_lock);
> +
> + return group;
> +}
> +
> +static int schedule_group_join(odp_schedule_group_t group,
> + const odp_thrmask_t *mask)
> +{
> + uint32_t sgi;
> + sched_group_t *sg;
> + int ret;
> +
> + /* Validate inputs */
> + if (group < 0 && group >= ((odp_schedule_group_t)MAX_SCHED_GROUP))
> + return -1;
> +
> + if (mask == NULL)
> + ODP_ABORT("name or mask is NULL\n");
> +
> + odp_spinlock_lock(&sched_grp_lock);
> +
> + sgi = (uint32_t)group;
> + if (bitset_is_set(sg_free, sgi)) {
> + odp_spinlock_unlock(&sched_grp_lock);
> + return -1;
> + }
> +
> + sg = sg_vec[sgi];
> + ret = schedule_group_update(sg, sgi, mask, SCHED_GROUP_JOIN);
> +
> + odp_spinlock_unlock(&sched_grp_lock);
> +
> + return ret;
> +}
> +
> +static int schedule_group_leave(odp_schedule_group_t group,
> + const odp_thrmask_t *mask)
> +{
> + uint32_t sgi;
> + sched_group_t *sg;
> + int ret = 0;
> +
> + /* Validate inputs */
> + if (group < 0 && group >= (odp_schedule_group_t)MAX_SCHED_GROUP) {
> + ret = -1;
> + goto invalid_group;
> + }
> +
> + if (mask == NULL)
> + ODP_ABORT("name or mask is NULL\n");
> +
> + odp_spinlock_lock(&sched_grp_lock);
> +
> + sgi = (uint32_t)group;
> + if (bitset_is_set(sg_free, sgi)) {
> + ret = -1;
> + goto group_not_found;
> + }
> +
> + sg = sg_vec[sgi];
> +
> + ret = schedule_group_update(sg, sgi, mask, SCHED_GROUP_LEAVE);
> +
> + odp_spinlock_unlock(&sched_grp_lock);
> +
> + return ret;
> +
> +group_not_found:
> + odp_spinlock_unlock(&sched_grp_lock);
> +
> +invalid_group:
> + return ret;
> +}
> +
> +static int schedule_group_thrmask(odp_schedule_group_t group,
> + odp_thrmask_t *mask)
> +{
> + uint32_t sgi;
> + sched_group_t *sg;
> + int ret = 0;
> +
> + /* Validate inputs */
> + if (group < 0 && group >= ((odp_schedule_group_t)MAX_SCHED_GROUP)) {
> + ret = -1;
> + goto invalid_group;
> + }
> +
> + if (mask == NULL)
> + ODP_ABORT("name or mask is NULL\n");
> +
> + odp_spinlock_lock(&sched_grp_lock);
> +
> + sgi = (uint32_t)group;
> + if (bitset_is_set(sg_free, sgi)) {
> + ret = -1;
> + goto group_not_found;
> + }
> +
> + sg = sg_vec[sgi];
> + ret = _schedule_group_thrmask(sg, mask);
> +
> + odp_spinlock_unlock(&sched_grp_lock);
> +
> + return ret;
> +
> +group_not_found:
> + odp_spinlock_unlock(&sched_grp_lock);
> +
> +invalid_group:
> + return ret;
> +}
> +
> +static int schedule_group_info(odp_schedule_group_t group,
> + odp_schedule_group_info_t *info)
> +{
> + uint32_t sgi;
> + sched_group_t *sg;
> + int ret = 0;
> +
> + /* Validate inputs */
> + if (group < 0 && group >= ((odp_schedule_group_t)MAX_SCHED_GROUP)) {
> + ret = -1;
> + goto invalid_group;
> + }
> +
> + if (info == NULL)
> + ODP_ABORT("name or mask is NULL\n");
> +
> + odp_spinlock_lock(&sched_grp_lock);
> +
> + sgi = (uint32_t)group;
> + if (bitset_is_set(sg_free, sgi)) {
> + ret = -1;
> + goto group_not_found;
> + }
> +
> + sg = sg_vec[sgi];
> +
> + ret = _schedule_group_thrmask(sg, &info->thrmask);
> +
> + info->name = sg->name;
> +
> + odp_spinlock_unlock(&sched_grp_lock);
> +
> + return ret;
> +
> +group_not_found:
> + odp_spinlock_unlock(&sched_grp_lock);
> +
> +invalid_group:
> + return ret;
> +}
> +
> +static int schedule_init_global(void)
> +{
> + odp_thrmask_t mask;
> + odp_schedule_group_t tmp_all;
> + odp_schedule_group_t tmp_wrkr;
> + odp_schedule_group_t tmp_ctrl;
> + uint32_t bits;
> +
> + odp_spinlock_init(&sched_grp_lock);
> +
> + bits = MAX_SCHED_GROUP;
> + if (MAX_SCHED_GROUP == sizeof(sg_free) * CHAR_BIT)
> + sg_free = ~0ULL;
> + else
> + sg_free = (1ULL << bits) - 1;
> +
> + for (uint32_t i = 0; i < MAX_SCHED_GROUP; i++)
> + sg_vec[i] = NULL;
> + for (uint32_t i = 0; i < MAXTHREADS; i++) {
> + thread_state[i].sg_sem = 0;
> + for (uint32_t j = 0; j < ODP_SCHED_PRIO_NUM; j++)
> + thread_state[i].sg_wanted[j] = bitset_null();
> + }
> +
> + /* Create sched groups for default GROUP_ALL, GROUP_WORKER and
> + * GROUP_CONTROL groups.
> + */
> + odp_thrmask_zero(&mask);
> + tmp_all = odp_schedule_group_create("__group_all", &mask);
> + if (tmp_all != ODP_SCHED_GROUP_ALL) {
> + ODP_ERR("Could not create ODP_SCHED_GROUP_ALL()\n");
> + goto failed_create_group_all;
> + }
> +
> + tmp_wrkr = odp_schedule_group_create("__group_worker", &mask);
> + if (tmp_wrkr != ODP_SCHED_GROUP_WORKER) {
> + ODP_ERR("Could not create ODP_SCHED_GROUP_WORKER()\n");
> + goto failed_create_group_worker;
> + }
> +
> + tmp_ctrl = odp_schedule_group_create("__group_control", &mask);
> + if (tmp_ctrl != ODP_SCHED_GROUP_CONTROL) {
> + ODP_ERR("Could not create ODP_SCHED_GROUP_CONTROL()\n");
> + goto failed_create_group_control;
> + }
> +
> + return 0;
> +
> +failed_create_group_control:
> + if (tmp_ctrl != ODP_SCHED_GROUP_INVALID)
> + odp_schedule_group_destroy(ODP_SCHED_GROUP_CONTROL);
> +
> +failed_create_group_worker:
> + if (tmp_wrkr != ODP_SCHED_GROUP_INVALID)
> + odp_schedule_group_destroy(ODP_SCHED_GROUP_WORKER);
> +
> +failed_create_group_all:
> + if (tmp_all != ODP_SCHED_GROUP_INVALID)
> + odp_schedule_group_destroy(ODP_SCHED_GROUP_ALL);
> +
> + return -1;
> +}
> +
> +static int schedule_term_global(void)
> +{
> + /* Destroy sched groups for default GROUP_ALL, GROUP_WORKER and
> + * GROUP_CONTROL groups.
> + */
> + if (odp_schedule_group_destroy(ODP_SCHED_GROUP_ALL) != 0)
> + ODP_ERR("Failed to destroy ODP_SCHED_GROUP_ALL\n");
> + if (odp_schedule_group_destroy(ODP_SCHED_GROUP_WORKER) != 0)
> + ODP_ERR("Failed to destroy ODP_SCHED_GROUP_WORKER\n");
> + if (odp_schedule_group_destroy(ODP_SCHED_GROUP_CONTROL) != 0)
> + ODP_ERR("Failed to destroy ODP_SCHED_GROUP_CONTROL\n");
> +
> + return 0;
> +}
> +
> +static int schedule_init_local(void)
> +{
> + int thr_id;
> + odp_thread_type_t thr_type;
> + odp_thrmask_t mask;
> +
> + thr_id = odp_thread_id();
> + if (thread_state_init(thr_id))
> + goto failed_to_init_ts;
> +
> + /* Add this thread to default schedule groups */
> + thr_type = odp_thread_type();
> + odp_thrmask_zero(&mask);
> + odp_thrmask_set(&mask, thr_id);
> +
> + if (odp_schedule_group_join(ODP_SCHED_GROUP_ALL, &mask) != 0) {
> + ODP_ERR("Failed to join ODP_SCHED_GROUP_ALL\n");
> + goto failed_to_join_grp_all;
> + }
> + if (thr_type == ODP_THREAD_CONTROL) {
> + if (odp_schedule_group_join(ODP_SCHED_GROUP_CONTROL,
> + &mask) != 0) {
> + ODP_ERR("Failed to join ODP_SCHED_GROUP_CONTROL\n");
> + goto failed_to_join_grp_ctrl;
> + }
> + } else {
> + if (odp_schedule_group_join(ODP_SCHED_GROUP_WORKER,
> + &mask) != 0) {
> + ODP_ERR("Failed to join ODP_SCHED_GROUP_WORKER\n");
> + goto failed_to_join_grp_wrkr;
> + }
> + }
> +
> + return 0;
> +
> +failed_to_join_grp_wrkr:
> +
> +failed_to_join_grp_ctrl:
> + odp_schedule_group_leave(ODP_SCHED_GROUP_ALL, &mask);
> +
> +failed_to_join_grp_all:
> +failed_to_init_ts:
> +
> + return -1;
> +}
> +
> +static int schedule_term_local(void)
> +{
> + int thr_id;
> + odp_thread_type_t thr_type;
> + odp_thrmask_t mask;
> + int rc = 0;
> +
> + /* Remove this thread from default schedule groups */
> + thr_id = odp_thread_id();
> + thr_type = odp_thread_type();
> + odp_thrmask_zero(&mask);
> + odp_thrmask_set(&mask, thr_id);
> +
> + if (odp_schedule_group_leave(ODP_SCHED_GROUP_ALL, &mask) != 0)
> + ODP_ERR("Failed to leave ODP_SCHED_GROUP_ALL\n");
> + if (thr_type == ODP_THREAD_CONTROL) {
> + if (odp_schedule_group_leave(ODP_SCHED_GROUP_CONTROL,
> + &mask) != 0)
> + ODP_ERR("Failed to leave ODP_SCHED_GROUP_CONTROL\n");
> + } else {
> + if (odp_schedule_group_leave(ODP_SCHED_GROUP_WORKER,
> + &mask) != 0)
> + ODP_ERR("Failed to leave ODP_SCHED_GROUP_WORKER\n");
> + }
> +
> + update_sg_membership(sched_ts);
> +
> + /* Check if the thread is still part of any groups */
> + if (sched_ts->num_schedq != 0) {
> + ODP_ERR("Thread %d still part of scheduler group(s)\n",
> + sched_ts->tidx);
> + rc = -1;
> + }
> +
> + return rc;
> +}
> +
> +int queue_tm_reorder(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr)
> +{
> + (void)queue;
> + (void)buf_hdr;
> + return 0;
> +}
> +
> +static void pktio_start(int pktio_index, int num_in_queue, int in_queue_idx[])
> +{
> + int i;
> + uint32_t old, tag, j;
> +
> + for (i = 0; i < num_in_queue; i++) {
> + /* Try to reserve a slot */
> + if (__atomic_fetch_add(&pktin_num,
> + 1, __ATOMIC_RELAXED) >= PKTIN_MAX) {
> + __atomic_fetch_sub(&pktin_num, 1, __ATOMIC_RELAXED);
> + ODP_ABORT("Too many pktio in queues for scheduler\n");
> + }
> + /* A slot has been reserved, now we need to find an empty one */
> + for (j = 0; ; j = (j + 1) % PKTIN_MAX) {
> + if (__atomic_load_n(&pktin_tags[j],
> + __ATOMIC_RELAXED) != TAG_EMPTY)
> + /* Slot used, continue with next */
> + continue;
> + /* Empty slot found */
> + old = TAG_EMPTY;
> + tag = PKTIO_QUEUE_2_TAG(pktio_index, in_queue_idx[i]);
> + if (__atomic_compare_exchange_n(&pktin_tags[j],
> + &old,
> + tag,
> + true,
> + __ATOMIC_RELEASE,
> + __ATOMIC_RELAXED)) {
> + /* Success grabbing slot,update high
> + * watermark
> + */
> + __atomic_fetch_max(&pktin_hi,
> + j + 1, __ATOMIC_RELAXED);
> + /* One more tag (queue) for this pktio
> + * instance
> + */
> + __atomic_fetch_add(&pktin_count[pktio_index],
> + 1, __ATOMIC_RELAXED);
> + /* Continue with next RX queue */
> + break;
> + }
> + /* Failed to grab slot */
> + }
> + }
> +}
> +
> +static int num_grps(void)
> +{
> + return MAX_SCHED_GROUP;
> +}
> +
> +/*
> + * Stubs for internal scheduler abstraction layer due to absence of NULL
> + * checking before calling the function pointer.
> + */
> +
> +static int thr_add(odp_schedule_group_t group, int thr)
> +{
> + /* This function is a schedule_init_local duplicate. */
> + (void)group;
> + (void)thr;
> + return 0;
> +}
> +static int thr_rem(odp_schedule_group_t group, int thr)
> +{
> + /* This function is a schedule_term_local duplicate. */
> + (void)group;
> + (void)thr;
> + return 0;
> +}
> +static int init_queue(uint32_t queue_index,
> + const odp_schedule_param_t *sched_param)
> +{
> + /* Not used in scalable scheduler. */
> + (void)queue_index;
> + (void)sched_param;
> + return 0;
> +}
> +static void destroy_queue(uint32_t queue_index)
> +{
> + /* Not used in scalable scheduler. */
> + (void)queue_index;
> +}
> +static int sched_queue(uint32_t queue_index)
> +{
> + /* Not used in scalable scheduler. */
> + (void)queue_index;
> + return 0;
> +}
> +static int ord_enq_multi(uint32_t queue_index, void *p_buf_hdr[],
> + int num, int *ret)
> +{
> + (void)queue_index;
> + (void)p_buf_hdr;
> + (void)num;
> + (void)ret;
> + return 0;
> +}
> +
> +static void schedule_prefetch(int num)
> +{
> + (void)num;
> +}
> +
> +static void order_lock(void)
> +{
> +}
> +
> +static void order_unlock(void)
> +{
> +}
> +
> +const schedule_fn_t schedule_scalable_fn = {
> + .pktio_start = pktio_start,
> + .thr_add = thr_add,
> + .thr_rem = thr_rem,
> + .num_grps = num_grps,
> + .init_queue = init_queue,
> + .destroy_queue = destroy_queue,
> + .sched_queue = sched_queue,
> + .ord_enq_multi = ord_enq_multi,
> + .init_global = schedule_init_global,
> + .term_global = schedule_term_global,
> + .init_local = schedule_init_local,
> + .term_local = schedule_term_local,
> + .order_lock = order_lock,
> + .order_unlock = order_unlock,
> +};
> +
> +const schedule_api_t schedule_scalable_api = {
> + .schedule_wait_time = schedule_wait_time,
> + .schedule = schedule,
> + .schedule_multi = schedule_multi,
> + .schedule_pause = schedule_pause,
> + .schedule_resume = schedule_resume,
> + .schedule_release_atomic = schedule_release_atomic,
> + .schedule_release_ordered = schedule_release_ordered,
> + .schedule_prefetch = schedule_prefetch,
> + .schedule_num_prio = schedule_num_prio,
> + .schedule_group_create = schedule_group_create,
> + .schedule_group_destroy = schedule_group_destroy,
> + .schedule_group_lookup = schedule_group_lookup,
> + .schedule_group_join = schedule_group_join,
> + .schedule_group_leave = schedule_group_leave,
> + .schedule_group_thrmask = schedule_group_thrmask,
> + .schedule_group_info = schedule_group_info,
> + .schedule_order_lock = schedule_order_lock,
> + .schedule_order_unlock = schedule_order_unlock,
> +};
> diff --git a/platform/linux-generic/odp_schedule_scalable_ordered.c b/platform/linux-generic/odp_schedule_scalable_ordered.c
> new file mode 100644
> index 00000000..ff4431b5
> --- /dev/null
> +++ b/platform/linux-generic/odp_schedule_scalable_ordered.c
> @@ -0,0 +1,298 @@
> +/* Copyright (c) 2017, ARM Limited
> + * All rights reserved.
> + *
> + * SPDX-License-Identifier: BSD-3-Clause
> + */
> +
> +#include <odp/api/shared_memory.h>
> +#include <odp_queue_internal.h>
> +#include <odp_schedule_if.h>
> +#include <odp_schedule_ordered_internal.h>
> +#include <odp_llsc.h>
> +#include <odp_bitset.h>
> +
> +#include <string.h>
> +
> +extern __thread sched_scalable_thread_state_t *sched_ts;
> +
> +#define RWIN_NAME_SIZE 32
> +
> +reorder_window_t *rwin_alloc(int rwin_id, odp_shm_t *shm, unsigned lock_count)
> +{
> + char rwin_name[RWIN_NAME_SIZE];
> + reorder_window_t *rwin;
> + uint32_t i;
> +
> + strncpy(rwin_name, "rwin", RWIN_NAME_SIZE);
> + i = strlen(rwin_name);
> + snprintf(rwin_name + i,
> + (RWIN_NAME_SIZE - i), "%d", rwin_id);
> +
> + *shm = odp_shm_reserve(rwin_name,
> + sizeof(reorder_window_t),
> + ODP_CACHE_LINE_SIZE,
> + ODP_SHM_PROC);
> + if (ODP_SHM_INVALID == *shm)
> + goto shm_reserve_failed;
> +
> + rwin = (reorder_window_t *) odp_shm_addr(*shm);
> + if (rwin == NULL)
> + goto shm_addr_failed;
> +
> + rwin->hc.head = 0;
> + rwin->hc.chgi = 0;
> + rwin->winmask = RWIN_SIZE - 1;
> + rwin->tail = 0;
> + rwin->turn = 0;
> + rwin->lock_count = (uint16_t)lock_count;
> + memset(rwin->olock, 0, sizeof(rwin->olock));
> + for (i = 0; i < RWIN_SIZE; i++)
> + rwin->ring[i] = NULL;
> +
> + return rwin;
> +
> +shm_addr_failed:
> + odp_shm_free(*shm);
> +
> +shm_reserve_failed:
> + return NULL;
> +}
> +
> +bool rwin_reserve(reorder_window_t *rwin, uint32_t *sn)
> +{
> + uint32_t head;
> + uint32_t oldt;
> + uint32_t newt;
> + uint32_t winmask;
> +
> + /* Read head and tail separately */
> +#ifndef ODP_CONFIG_USE_LLSC
> + oldt = rwin->tail;
> +#endif
> + __builtin_prefetch(&rwin->tail, 1, 0);
> + winmask = rwin->winmask;
> + do {
> + /* Need __atomic_load to avoid compiler reordering */
> + head = __atomic_load_n(&rwin->hc.head, __ATOMIC_RELAXED);
> +#ifdef ODP_CONFIG_USE_LLSC
> + oldt = ll32(&rwin->tail, __ATOMIC_RELAXED);
> +#endif
> + if (odp_unlikely(oldt - head >= winmask))
> + return false;
> +
> + newt = oldt + 1;
> +#ifdef ODP_CONFIG_USE_LLSC
> + } while (odp_unlikely(sc32(&rwin->tail, newt, __ATOMIC_RELAXED)));
> +#else
> + } while (!__atomic_compare_exchange(&rwin->tail,
> + &oldt,
> + &newt,
> + true,
> + __ATOMIC_RELAXED,
> + __ATOMIC_RELAXED));
> +#endif
> + *sn = oldt;
> +
> + return true;
> +}
> +
> +void rwin_insert(reorder_window_t *rwin,
> + reorder_context_t *rctx,
> + uint32_t sn,
> + void (*callback)(reorder_context_t *))
> +{
> + /* Initialise to silence scan-build */
> + hc_t old = {0, 0};
> + hc_t new;
> + uint32_t winmask;
> +
> + __atomic_load(&rwin->hc, &old, __ATOMIC_ACQUIRE);
> + winmask = rwin->winmask;
> + if (old.head != sn) {
> + /* We are out-of-order. Store context in reorder window,
> + * releasing its content.
> + */
> + ODP_ASSERT(rwin->ring[sn & winmask] == NULL);
> +#ifdef ODP_CONFIG_USE_DMB
> + __atomic_thread_fence(__ATOMIC_RELEASE);
> + __atomic_store_n(&rwin->ring[sn & winmask], rctx,
> + __ATOMIC_RELAXED);
> +#else
> + __atomic_store_n(&rwin->ring[sn & winmask], rctx,
> + __ATOMIC_RELEASE);
> +#endif
> + rctx = NULL;
> + do {
> + hc_t new;
> +
> + new.head = old.head;
> + new.chgi = old.chgi + 1; /* Changed value */
> + /* Update head & chgi, fail if any has changed */
> + if (__atomic_compare_exchange(&rwin->hc,
> + &old, /* Updated on failure */
> + &new,
> + true,
> + __ATOMIC_RELEASE, /* Release our ring update */
> + __ATOMIC_ACQUIRE))
> + /* CAS succeeded => head same (we are not
> + * in-order), chgi updated.
> + */
> + return;
> + /* CAS failed => head and/or chgi changed.
> + * We might not be out-of-order anymore.
> + */
> + } while (old.head != sn);
> + }
> +
> + /* old.head == sn => we are now in-order! */
> + ODP_ASSERT(old.head == sn);
> + /* We are in-order so our responsibility to retire contexts */
> + new.head = old.head;
> + new.chgi = old.chgi + 1;
> +
> + /* Retire our in-order context (if we still have it) */
> + if (rctx != NULL) {
> + callback(rctx);
> + new.head++;
> + }
> +
> + /* Retire in-order contexts in the ring
> + * The first context might actually be ours (if we were originally
> + * out-of-order)
> + */
> + do {
> + for (;;) {
> + rctx = __atomic_load_n(&rwin->ring[new.head & winmask],
> + __ATOMIC_ACQUIRE);
> + if (rctx == NULL)
> + break;
> + /* We are the only thread that are in-order
> + * (until head updated) so don't have to use
> + * atomic load-and-clear (exchange)
> + */
> + rwin->ring[new.head & winmask] = NULL;
> + callback(rctx);
> + new.head++;
> + }
> + /* Update head&chgi, fail if chgi has changed (head cannot change) */
> + } while (!__atomic_compare_exchange(&rwin->hc,
> + &old, /* Updated on failure */
> + &new,
> + false, /* weak */
> + __ATOMIC_RELEASE, /* Release our ring updates */
> + __ATOMIC_ACQUIRE));
> +}
> +
> +void rctx_init(reorder_context_t *rctx, uint16_t idx,
> + reorder_window_t *rwin, uint32_t sn)
> +{
> + /* rctx->rvec_free and rctx->idx already initialised in
> + * thread_state_init function.
> + */
> + ODP_ASSERT(rctx->idx == idx);
> + rctx->rwin = rwin;
> + rctx->sn = sn;
> + rctx->olock_flags = 0;
> + /* First => no next reorder context */
> + rctx->next_idx = idx;
> + /* Where to store next event */
> + rctx->cur_idx = idx;
> + rctx->numevts = 0;
> +}
> +
> +inline void rctx_free(const reorder_context_t *rctx)
> +{
> + const reorder_context_t *const base = &rctx[-(int)rctx->idx];
> + const uint32_t first = rctx->idx;
> + uint32_t next_idx;
> +
> + next_idx = rctx->next_idx;
> +
> + ODP_ASSERT(rctx->rwin != NULL);
> + /* Set free bit */
> + if (rctx->rvec_free == &sched_ts->rvec_free)
> + /* Since it is our own reorder context, we can instead
> + * perform a non-atomic and relaxed update on our private
> + * rvec_free.
> + */
> + sched_ts->priv_rvec_free =
> + bitset_set(sched_ts->priv_rvec_free, rctx->idx);
> + else
> + atom_bitset_set(rctx->rvec_free, rctx->idx, __ATOMIC_RELEASE);
> +
> + /* Can't dereference rctx after the corresponding free bit is set */
> + while (next_idx != first) {
> + rctx = &base[next_idx];
> + next_idx = rctx->next_idx;
> + /* Set free bit */
> + if (rctx->rvec_free == &sched_ts->rvec_free)
> + sched_ts->priv_rvec_free =
> + bitset_set(sched_ts->priv_rvec_free, rctx->idx);
> + else
> + atom_bitset_set(rctx->rvec_free, rctx->idx,
> + __ATOMIC_RELEASE);
> + }
> +}
> +
> +inline void olock_unlock(const reorder_context_t *rctx, reorder_window_t *rwin,
> + uint32_t lock_index)
> +{
> + if ((rctx->olock_flags & (1U << lock_index)) == 0) {
> + /* Use relaxed ordering, we are not releasing any updates */
> + rwin->olock[lock_index] = rctx->sn + 1;
> + }
> +}
> +
> +void olock_release(const reorder_context_t *rctx)
> +{
> + reorder_window_t *rwin;
> +
> + rwin = rctx->rwin;
> +
> + switch (rwin->lock_count) {
> + case 2:
> + olock_unlock(rctx, rwin, 1);
> + case 1:
> + olock_unlock(rctx, rwin, 0);
> + }
> + ODP_STATIC_ASSERT(NUM_OLOCKS == 2, "Number of ordered locks != 2");
> +}
> +
> +void rctx_retire(reorder_context_t *first)
> +{
> + reorder_context_t *rctx;
> + queue_entry_t *q;
> + uint32_t i;
> + uint32_t j;
> + uint32_t num;
> + int rc;
> +
> + rctx = first;
> + do {
> + /* Process all events in this reorder context */
> + for (i = 0; i < rctx->numevts;) {
> + q = rctx->destq[i];
> + /* Find index of next different destq */
> + j = i + 1;
> + while (j < rctx->numevts && rctx->destq[j] == q)
> + j++;
> + num = j - i;
> + rc = q->s.enqueue_multi(q, &rctx->events[i], num);
> + if (odp_unlikely(rc != (int)num))
> + ODP_ERR("Failed to enqueue deferred events\n");
> + i += num;
> + }
> + /* Update rctx pointer to point to 'next_idx' element */
> + rctx += (int)rctx->next_idx - (int)rctx->idx;
> + } while (rctx != first);
> + olock_release(first);
> + rctx_free(first);
> +}
> +
> +void rctx_release(reorder_context_t *rctx)
> +{
> + /* Insert reorder context into reorder window, potentially calling the
> + * rctx_retire function for all pending reorder_contexts.
> + */
> + rwin_insert(rctx->rwin, rctx, rctx->sn, rctx_retire);
> +}
> diff --git a/platform/linux-generic/odp_traffic_mngr.c b/platform/linux-generic/odp_traffic_mngr.c
> index 4e9358b9..3244dfe3 100644
> --- a/platform/linux-generic/odp_traffic_mngr.c
> +++ b/platform/linux-generic/odp_traffic_mngr.c
> @@ -3918,9 +3918,10 @@ odp_tm_queue_t odp_tm_queue_create(odp_tm_t odp_tm,
> tm_queue_obj->pkt = ODP_PACKET_INVALID;
> odp_ticketlock_init(&tm_wred_node->tm_wred_node_lock);
>
> - tm_queue_obj->tm_qentry.s.type = QUEUE_TYPE_TM;
> - tm_queue_obj->tm_qentry.s.enqueue = queue_tm_reenq;
> - tm_queue_obj->tm_qentry.s.enqueue_multi = queue_tm_reenq_multi;
> + queue_set_type(&tm_queue_obj->tm_qentry, QUEUE_TYPE_TM);
> + queue_set_enq_func(&tm_queue_obj->tm_qentry, queue_tm_reenq);
> + queue_set_enq_multi_func(&tm_queue_obj->tm_qentry,
> + queue_tm_reenq_multi);
>
> tm_system->queue_num_tbl[tm_queue_obj->queue_num - 1] = tm_queue_obj;
> odp_ticketlock_lock(&tm_system->tm_system_lock);
> diff --git a/platform/linux-generic/pktio/loop.c b/platform/linux-generic/pktio/loop.c
> index 70962839..1f2f16c1 100644
> --- a/platform/linux-generic/pktio/loop.c
> +++ b/platform/linux-generic/pktio/loop.c
> @@ -80,11 +80,13 @@ static int loopback_recv(pktio_entry_t *pktio_entry, int index ODP_UNUSED,
>
> for (i = 0; i < nbr; i++) {
> uint32_t pkt_len;
> -
> +#ifdef ODP_SCHEDULE_SCALABLE
> + pkt = _odp_packet_from_buffer((odp_buffer_t)(hdr_tbl[i]));
> +#else
> pkt = _odp_packet_from_buffer(odp_hdr_to_buf(hdr_tbl[i]));
> +#endif
> pkt_len = odp_packet_len(pkt);
>
> -
> if (pktio_cls_enabled(pktio_entry)) {
> odp_packet_t new_pkt;
> odp_pool_t new_pool;
> @@ -162,7 +164,11 @@ static int loopback_send(pktio_entry_t *pktio_entry, int index ODP_UNUSED,
> len = QUEUE_MULTI_MAX;
>
> for (i = 0; i < len; ++i) {
> +#ifdef ODP_SCHEDULE_SCALABLE
> + hdr_tbl[i] = _odp_packet_to_buf_hdr_ptr(pkt_tbl[i]);
> +#else
> hdr_tbl[i] = buf_hdl_to_hdr(_odp_packet_to_buffer(pkt_tbl[i]));
> +#endif
> bytes += odp_packet_len(pkt_tbl[i]);
> }
>
> @@ -176,6 +182,7 @@ static int loopback_send(pktio_entry_t *pktio_entry, int index ODP_UNUSED,
> pktio_entry->s.stats.out_octets += bytes;
> } else {
> ODP_DBG("queue enqueue failed %i\n", ret);
> + odp_ticketlock_unlock(&pktio_entry->s.txl);
> return -1;
> }
>
> diff --git a/test/common_plat/performance/odp_sched_latency.c b/test/common_plat/performance/odp_sched_latency.c
> index 2b28cd7b..5e11a4f3 100644
> --- a/test/common_plat/performance/odp_sched_latency.c
> +++ b/test/common_plat/performance/odp_sched_latency.c
> @@ -28,7 +28,13 @@
> #define MAX_WORKERS 64 /**< Maximum number of worker threads */
> #define MAX_QUEUES 4096 /**< Maximum number of queues */
> #define EVENT_POOL_SIZE (1024 * 1024) /**< Event pool size */
> +
> +#ifdef ODP_SCHEDULE_SP
> #define TEST_ROUNDS (4 * 1024 * 1024) /**< Test rounds for each thread */
> +#else
> +#define TEST_ROUNDS (32 * 1024 * 1024) /**< Test rounds for each thread */
> +#endif
> +
> #define MAIN_THREAD 1 /**< Thread ID performing maintenance tasks */
>
> /* Default values for command line arguments */
> @@ -104,6 +110,9 @@ typedef union {
> typedef struct {
> core_stat_t core_stat[MAX_WORKERS]; /**< Core specific stats */
> odp_barrier_t barrier; /**< Barrier for thread synchronization */
> +#ifdef ODP_SCHEDULE_SCALABLE
> + odp_schedule_group_t schedule_group;
> +#endif
> odp_pool_t pool; /**< Pool for allocating test events */
> test_args_t args; /**< Parsed command line arguments */
> odp_queue_t queue[NUM_PRIOS][MAX_QUEUES]; /**< Scheduled queues */
> @@ -119,7 +128,11 @@ static void clear_sched_queues(void)
> odp_event_t ev;
>
> while (1) {
> - ev = odp_schedule(NULL, ODP_SCHED_NO_WAIT);
> + /* Need a non-zero timeout to ensure we can observe
> + * any non-empty queue made eligible for scheduling
> + * by some other thread.
> + */
> + ev = odp_schedule(NULL, 1000);
>
> if (ev == ODP_EVENT_INVALID)
> break;
> @@ -428,6 +441,20 @@ static int run_thread(void *arg ODP_UNUSED)
> return -1;
> }
>
> +#ifdef ODP_SCHEDULE_SCALABLE
> + int err;
> + odp_thrmask_t thrmask;
> +
> + odp_thrmask_zero(&thrmask);
> + odp_thrmask_set(&thrmask, thr);
> +
> + err = odp_schedule_group_join(globals->schedule_group, &thrmask);
> + if (err != 0) {
> + LOG_ERR("odp_schedule_group_join failed\n");
> + return -1;
> + }
> +#endif
> +
> if (thr == MAIN_THREAD) {
> args = &globals->args;
>
> @@ -452,6 +479,13 @@ static int run_thread(void *arg ODP_UNUSED)
> if (test_schedule(thr, globals))
> return -1;
>
> +#ifdef ODP_SCHEDULE_SCALABLE
> + err = odp_schedule_group_leave(globals->schedule_group, &thrmask);
> + if (err != 0) {
> + LOG_ERR("odp_schedule_group_leave failed\n");
> + return -1;
> + }
> +#endif
> return 0;
> }
>
> @@ -692,6 +726,31 @@ int main(int argc, char *argv[])
> }
> globals->pool = pool;
>
> +#ifdef ODP_SCHEDULE_SCALABLE
> + /*
> + * Create scheduler group
> + */
> + odp_thrmask_t expected_thrmask;
> + int cpu;
> + int thr;
> +
> + odp_thrmask_zero(&expected_thrmask);
> + /* This is a odp_thrmask_from_cpumask() */
> + cpu = odp_cpumask_first(&cpumask);
> + thr = 1;
> + while (0 <= cpu) {
> + odp_thrmask_set(&expected_thrmask, thr++);
> + cpu = odp_cpumask_next(&cpumask, cpu);
> + }
> +
> + globals->schedule_group =
> + odp_schedule_group_create("sg0", &expected_thrmask);
> + if (globals->schedule_group == ODP_SCHED_GROUP_INVALID) {
> + LOG_ERR("odp_schedule_group_create failed\n");
> + return -1;
> + }
> +#endif
> +
> /*
> * Create queues for schedule test
> */
> @@ -713,7 +772,11 @@ int main(int argc, char *argv[])
> param.type = ODP_QUEUE_TYPE_SCHED;
> param.sched.prio = prio;
> param.sched.sync = args.sync_type;
> +#ifdef ODP_SCHEDULE_SCALABLE
> + param.sched.group = globals->schedule_group;
> +#else
> param.sched.group = ODP_SCHED_GROUP_ALL;
> +#endif
>
> for (j = 0; j < args.prio[i].queues; j++) {
> name[9] = '0' + j / 10;
> @@ -758,6 +821,10 @@ int main(int argc, char *argv[])
> }
> }
>
> +
> +#ifdef ODP_SCHEDULE_SCALABLE
> + ret += odp_schedule_group_destroy(globals->schedule_group);
> +#endif
> ret += odp_shm_free(shm);
> ret += odp_pool_destroy(pool);
> ret += odp_term_local();
> diff --git a/test/common_plat/performance/odp_scheduling.c b/test/common_plat/performance/odp_scheduling.c
> index c74a0713..38e76257 100644
> --- a/test/common_plat/performance/odp_scheduling.c
> +++ b/test/common_plat/performance/odp_scheduling.c
> @@ -273,7 +273,7 @@ static int test_plain_queue(int thr, test_globals_t *globals)
> test_message_t *t_msg;
> odp_queue_t queue;
> uint64_t c1, c2, cycles;
> - int i;
> + int i, j;
>
> /* Alloc test message */
> buf = odp_buffer_alloc(globals->pool);
> @@ -307,7 +307,15 @@ static int test_plain_queue(int thr, test_globals_t *globals)
> return -1;
> }
>
> - ev = odp_queue_deq(queue);
> + /* When enqueue and dequeue are decoupled (e.g. not using a
> + * common lock), an enqueued event may not be immediately
> + * visible to dequeue. So we just try again for a while. */
> + for (j = 0; j < 100; j++) {
> + ev = odp_queue_deq(queue);
> + if (ev != ODP_EVENT_INVALID)
> + break;
> + odp_cpu_pause();
> + }
>
> buf = odp_buffer_from_event(ev);
>
> diff --git a/test/common_plat/validation/api/classification/odp_classification_basic.c b/test/common_plat/validation/api/classification/odp_classification_basic.c
> index 9817287e..b5b92851 100644
> --- a/test/common_plat/validation/api/classification/odp_classification_basic.c
> +++ b/test/common_plat/validation/api/classification/odp_classification_basic.c
> @@ -93,10 +93,10 @@ void classification_test_create_pmr_match(void)
> configure_default_cos(pktio, &default_cos,
> &default_queue, &default_pool);
>
> - queue = queue_create("pmr_match", true);
> + queue = queue_create("pmr_match_queue", true);
> CU_ASSERT(queue != ODP_QUEUE_INVALID);
>
> - pool = pool_create("pmr_match");
> + pool = pool_create("pmr_match_pool");
> CU_ASSERT_FATAL(pool != ODP_POOL_INVALID);
>
> odp_cls_cos_param_init(&cls_param);
> @@ -277,10 +277,10 @@ void classification_test_pmr_composite_create(void)
> configure_default_cos(pktio, &default_cos,
> &default_queue, &default_pool);
>
> - queue = queue_create("pmr_match", true);
> + queue = queue_create("pmr_match_queue", true);
> CU_ASSERT(queue != ODP_QUEUE_INVALID);
>
> - pool = pool_create("pmr_match");
> + pool = pool_create("pmr_match_pool");
> CU_ASSERT_FATAL(pool != ODP_POOL_INVALID);
>
> odp_cls_cos_param_init(&cls_param);
> diff --git a/test/common_plat/validation/api/classification/odp_classification_test_pmr.c b/test/common_plat/validation/api/classification/odp_classification_test_pmr.c
> index d9524205..c1da5159 100644
> --- a/test/common_plat/validation/api/classification/odp_classification_test_pmr.c
> +++ b/test/common_plat/validation/api/classification/odp_classification_test_pmr.c
> @@ -121,10 +121,10 @@ void classification_test_pmr_term_tcp_dport(void)
> configure_default_cos(pktio, &default_cos,
> &default_queue, &default_pool);
>
> - queue = queue_create("tcp_dport1", true);
> + queue = queue_create("tcp_dport1_queue", true);
> CU_ASSERT(queue != ODP_QUEUE_INVALID);
>
> - pool = pool_create("tcp_dport1");
> + pool = pool_create("tcp_dport1_pool");
> CU_ASSERT_FATAL(pool != ODP_POOL_INVALID);
>
> sprintf(cosname, "tcp_dport");
> @@ -235,10 +235,10 @@ void classification_test_pmr_term_tcp_sport(void)
> configure_default_cos(pktio, &default_cos,
> &default_queue, &default_pool);
>
> - queue = queue_create("tcp_sport", true);
> + queue = queue_create("tcp_sport_queue", true);
> CU_ASSERT_FATAL(queue != ODP_QUEUE_INVALID);
>
> - pool = pool_create("tcp_sport");
> + pool = pool_create("tcp_sport_pool");
> CU_ASSERT_FATAL(pool != ODP_POOL_INVALID);
>
> sprintf(cosname, "tcp_sport");
> @@ -348,10 +348,10 @@ void classification_test_pmr_term_udp_dport(void)
> configure_default_cos(pktio, &default_cos,
> &default_queue, &default_pool);
>
> - queue = queue_create("udp_dport", true);
> + queue = queue_create("udp_dport_queue", true);
> CU_ASSERT_FATAL(queue != ODP_QUEUE_INVALID);
>
> - pool = pool_create("udp_dport");
> + pool = pool_create("udp_dport_pool");
> CU_ASSERT_FATAL(pool != ODP_POOL_INVALID);
>
> sprintf(cosname, "udp_dport");
> @@ -464,10 +464,10 @@ void classification_test_pmr_term_udp_sport(void)
> configure_default_cos(pktio, &default_cos,
> &default_queue, &default_pool);
>
> - queue = queue_create("udp_sport", true);
> + queue = queue_create("udp_sport_queue", true);
> CU_ASSERT_FATAL(queue != ODP_QUEUE_INVALID);
>
> - pool = pool_create("udp_sport");
> + pool = pool_create("udp_sport_pool");
> CU_ASSERT_FATAL(pool != ODP_POOL_INVALID);
>
> sprintf(cosname, "udp_sport");
> @@ -578,10 +578,10 @@ void classification_test_pmr_term_ipproto(void)
> configure_default_cos(pktio, &default_cos,
> &default_queue, &default_pool);
>
> - queue = queue_create("ipproto", true);
> + queue = queue_create("ipproto_queue", true);
> CU_ASSERT_FATAL(queue != ODP_QUEUE_INVALID);
>
> - pool = pool_create("ipproto");
> + pool = pool_create("ipproto_pool");
> CU_ASSERT_FATAL(pool != ODP_POOL_INVALID);
>
> sprintf(cosname, "ipproto");
> @@ -687,10 +687,10 @@ void classification_test_pmr_term_dmac(void)
> configure_default_cos(pktio, &default_cos,
> &default_queue, &default_pool);
>
> - queue = queue_create("dmac", true);
> + queue = queue_create("dmac_queue", true);
> CU_ASSERT_FATAL(queue != ODP_QUEUE_INVALID);
>
> - pool = pool_create("dmac");
> + pool = pool_create("dmac_pool");
> CU_ASSERT_FATAL(pool != ODP_POOL_INVALID);
>
> sprintf(cosname, "dmac");
> @@ -794,10 +794,10 @@ void classification_test_pmr_term_packet_len(void)
> configure_default_cos(pktio, &default_cos,
> &default_queue, &default_pool);
>
> - queue = queue_create("packet_len", true);
> + queue = queue_create("packet_len_queue", true);
> CU_ASSERT_FATAL(queue != ODP_QUEUE_INVALID);
>
> - pool = pool_create("packet_len");
> + pool = pool_create("packet_len_pool");
> CU_ASSERT_FATAL(pool != ODP_POOL_INVALID);
>
> sprintf(cosname, "packet_len");
> @@ -1355,10 +1355,10 @@ static void classification_test_pmr_pool_set(void)
> configure_default_cos(pktio, &default_cos,
> &default_queue, &default_pool);
>
> - queue = queue_create("ipproto1", true);
> + queue = queue_create("ipproto1_queue", true);
> CU_ASSERT_FATAL(queue != ODP_QUEUE_INVALID);
>
> - pool = pool_create("ipproto1");
> + pool = pool_create("ipproto1_pool");
> CU_ASSERT_FATAL(pool != ODP_POOL_INVALID);
>
> sprintf(cosname, "ipproto1");
> @@ -1454,10 +1454,10 @@ static void classification_test_pmr_queue_set(void)
> configure_default_cos(pktio, &default_cos,
> &default_queue, &default_pool);
>
> - queue = queue_create("ipproto1", true);
> + queue = queue_create("ipproto1_queue", true);
> CU_ASSERT_FATAL(queue != ODP_QUEUE_INVALID);
>
> - pool = pool_create("ipproto1");
> + pool = pool_create("ipproto1_pool");
> CU_ASSERT_FATAL(pool != ODP_POOL_INVALID);
>
> sprintf(cosname, "ipproto1");
> @@ -1469,7 +1469,7 @@ static void classification_test_pmr_queue_set(void)
> cos = odp_cls_cos_create(cosname, &cls_param);
> CU_ASSERT_FATAL(cos != ODP_COS_INVALID);
>
> - queue_new = queue_create("ipproto2", true);
> + queue_new = queue_create("ipproto2_queue", true);
> CU_ASSERT_FATAL(queue_new != ODP_QUEUE_INVALID);
>
> /* new queue is set on CoS */
> @@ -1546,10 +1546,10 @@ static void classification_test_pmr_term_daddr(void)
> configure_default_cos(pktio, &default_cos,
> &default_queue, &default_pool);
>
> - queue = queue_create("daddr", true);
> + queue = queue_create("daddr_queue", true);
> CU_ASSERT_FATAL(queue != ODP_QUEUE_INVALID);
>
> - pool = pool_create("daddr");
> + pool = pool_create("daddr_pool");
> CU_ASSERT_FATAL(pool != ODP_POOL_INVALID);
>
> sprintf(cosname, "daddr");
> diff --git a/test/common_plat/validation/api/scheduler/scheduler.c b/test/common_plat/validation/api/scheduler/scheduler.c
> index 952561cd..bc486192 100644
> --- a/test/common_plat/validation/api/scheduler/scheduler.c
> +++ b/test/common_plat/validation/api/scheduler/scheduler.c
> @@ -251,7 +251,10 @@ void scheduler_test_queue_destroy(void)
> CU_ASSERT_FATAL(u32[0] == MAGIC);
>
> odp_buffer_free(buf);
> - odp_schedule_release_ordered();
> + if (qp.sched.sync == ODP_SCHED_SYNC_ATOMIC)
> + odp_schedule_release_atomic();
> + else if (qp.sched.sync == ODP_SCHED_SYNC_ORDERED)
> + odp_schedule_release_ordered();
>
> CU_ASSERT_FATAL(odp_queue_destroy(queue) == 0);
> }
> @@ -478,6 +481,11 @@ void scheduler_test_groups(void)
> odp_schedule_group_leave(mygrp1, &mymask);
> odp_schedule_group_leave(mygrp2, &mymask);
>
> + if (qp.sched.sync == ODP_SCHED_SYNC_ATOMIC)
> + odp_schedule_release_atomic();
> + else if (qp.sched.sync == ODP_SCHED_SYNC_ORDERED)
> + odp_schedule_release_ordered();
> +
> /* Done with queues for this round */
> CU_ASSERT_FATAL(odp_queue_destroy(queue_grp1) == 0);
> CU_ASSERT_FATAL(odp_queue_destroy(queue_grp2) == 0);
> @@ -959,7 +967,6 @@ static void fill_queues(thread_args_t *args)
> }
>
> ret = odp_queue_enq(queue, ev);
> - CU_ASSERT_FATAL(ret == 0);
>
> if (ret)
> odp_buffer_free(buf);
> diff --git a/test/common_plat/validation/api/timer/timer.c b/test/common_plat/validation/api/timer/timer.c
> index b7d84c64..362f33c8 100644
> --- a/test/common_plat/validation/api/timer/timer.c
> +++ b/test/common_plat/validation/api/timer/timer.c
> @@ -297,8 +297,11 @@ static int worker_entrypoint(void *arg TEST_UNUSED)
> struct timespec ts;
> uint32_t nstale;
> odp_timer_set_t timer_rc;
> + char name[32];
>
> - queue = odp_queue_create("timer_queue", NULL);
> + snprintf(name, sizeof(name), "timer_queue_%d", thr);
> +
> + queue = odp_queue_create(name, NULL);
> if (queue == ODP_QUEUE_INVALID)
> CU_FAIL_FATAL("Queue create failed");
>
> --
> 2.12.1
>
On Tue, Mar 28, 2017 at 2:23 PM, Brian Brooks <brian.brooks@arm.com> wrote: > Signed-off-by: Brian Brooks <brian.brooks@arm.com> > --- > include/odp/api/spec/queue.h | 5 +++++ > 1 file changed, 5 insertions(+) > > diff --git a/include/odp/api/spec/queue.h b/include/odp/api/spec/queue.h > index 7972feac..1cec4773 100644 > --- a/include/odp/api/spec/queue.h > +++ b/include/odp/api/spec/queue.h > @@ -124,6 +124,11 @@ typedef struct odp_queue_param_t { > * the queue type. */ > odp_queue_type_t type; > > + /** Queue size > + * > + * Indicates the max ring size of the ring buffer. */ > + uint32_t ring_size; ODP queues have historically been of unspecified size. If we're going to introduce the notion of explicitly limited sized queues this has additional implications. First, ring_size is an inappropriate choice of name here since a ring is an implementation model, not a specification. The documentation says "Queue size", so uint32_t size; is sufficient here. We should document that size = 0 requests a queue of default size (which may be unbounded). Second, if we're going to allow a queue size to be specified then this needs to be added as an output to odp_queue_capability() so the application knows the max_size supported (again 0 = unbounded). A larger question, however, is why is this being introduced at all since this field is only used in the modified odph_cuckoo_table_create() helper routine and this, in turn, is only used within the cuckootable test module? This seems an extraneous and unnecessary change and has no relationship to the rest of this patch series. So Parts 1 and 3 of this series don't seem to have anything to do with the scalable scheduler. As a minor point, the order of these needs to be reversed to preserve bisectability since Part 1 can't reference the new field before Part 3 defines it. > + > /** Enqueue mode > * > * Default value for both queue types is ODP_QUEUE_OP_MT. Application > -- > 2.12.1 >
On 03/28 19:18:37, Bill Fischofer wrote: > <html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"> It is infinitely better to do patch review in plain text rather than HTML. I thought this was a plain text mailing list? > <meta name="Generator" content="Microsoft Exchange Server"> > <!-- converted from text --> > <style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style></head> > <body> > <font size="2"><span style="font-size:10pt;"><div class="PlainText">On Tue, Mar 28, 2017 at 2:23 PM, Brian Brooks <brian.brooks@arm.com> wrote:<br> > > Signed-off-by: Brian Brooks <brian.brooks@arm.com><br> > > ---<br> > > include/odp/api/spec/queue.h | 5 +++++<br> > > 1 file changed, 5 insertions(+)<br> > ><br> > > diff --git a/include/odp/api/spec/queue.h b/include/odp/api/spec/queue.h<br> > > index 7972feac..1cec4773 100644<br> > > --- a/include/odp/api/spec/queue.h<br> > > +++ b/include/odp/api/spec/queue.h<br> > > @@ -124,6 +124,11 @@ typedef struct odp_queue_param_t {<br> > > * the queue type. */<br> > > odp_queue_type_t type;<br> > ><br> > > + /** Queue size<br> > > + *<br> > > + * Indicates the max ring size of the ring buffer. */<br> > > + uint32_t ring_size;<br> > <br> > ODP queues have historically been of unspecified size. If we're going<br> > to introduce the notion of explicitly limited sized queues this has<br> > additional implications.<br> > > First, ring_size is an inappropriate choice of name here since a ring<br> > is an implementation model, not a specification. The documentation<br> > says "Queue size", so<br> > <br> > uint32_t size;<br> > <br> > is sufficient here. Agree, will change 'ring_size' to a better name. > We should document that size = 0 requests a queue<br> > of default size (which may be unbounded).<br> Unbounded size is not practical or possible. Can we agree that 0 means that the default aka ODP_CONFIG_QUEUE_SIZE is used? Should we allow for greater than ODP_CONFIG_QUEUE_SIZE? E.g. 10,000 queue depth is also not practical or possible. Perhaps we need a max which also acts as the default. > <br> > Second, if we're going to allow a queue size to be specified then this<br> > needs to be added as an output to odp_queue_capability() OK, will look into that. > so the<br> > application knows the max_size supported (again 0 = unbounded).<br> > <br> > A larger question, however, is why is this being introduced at all<br> > since this field is only used in the modified<br> > odph_cuckoo_table_create() helper routine and this, in turn, is only<br> > used within the cuckootable test module? This seems an extraneous and<br> > unnecessary change and has no relationship to the rest of this patch<br> > series.<br> AFAIK, the cuckoo unit test enqueues too many events (millions) to a queue. That sounds like it makes no sense, but is an example of how anything is possible. > <br> > So Parts 1 and 3 of this series don't seem to have anything to do with<br> > the scalable scheduler. As a minor point, the order of these needs to<br> > be reversed to preserve bisectability since Part 1 can't reference the<br> > new field before Part 3 defines it.<br> Agree that there are 2 independent sets of patches, but the order is needed in order to get a sane `make check' on ARM-based chips. Without these fixes going in first, we have no way of knowing whether the scalable scheduler patches caused the issue or not. I can break these into 2 separate sets of patches. > <br> > > +<br> > > /** Enqueue mode<br> > > *<br> > > * Default value for both queue types is ODP_QUEUE_OP_MT. Application<br> > > --<br> > > 2.12.1<br> > ><br> > </div></span></font> > </body> > </html>
On 03/28 18:50:32, Bill Fischofer wrote: > <html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"> > <meta name="Generator" content="Microsoft Exchange Server"> > <!-- converted from text --> > <style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style></head> > <body> > <font size="2"><span style="font-size:10pt;"><div class="PlainText">This part generates numerous checkpatch warnings and errors. Please<br> > run checkpatch and correct for v2.<br> We ran checkpatch.pl and corrected the issues that made sense. We all know that checkpatch.pl is not perfect. Please point out the checkpatch.pl issues that are generated from this patch which are not false positives or not destructive to programmer readability. > <br> > Also, this part introduces a number of errors that result in failure<br> > to compile using clang. This is likely true. Is compilation with Clang a blocker? > Please test with both gcc and clang to ensure<br> > that it compiles cleanly for both (gcc looks fine)<br> > <br> > Specific clang issues:<br> > CC odp_packet.lo<br> > odp_packet.c:656:9: error: cast from 'odp_buffer_t' (aka<br> > '_odp_abi_buffer_t *') to 'odp_buffer_hdr_t *' (aka 'struct<br> > odp_buffer_hdr_t *') increases required alignment from 1 to 8<br> > [-Werror,-Wcast-align]<br> > return (odp_buffer_hdr_t *)(_odp_packet_to_buffer(pkt));<br> > ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br> > <br> > clang requires an intermediate cast to (void *) to avoid these warnings.<br> > <br> > 1 error generated.<br> > Makefile:1004: recipe for target 'odp_packet.lo' failed<br> > <br> > <br> > clang also generates numerous link errors:<br> > <br> > CC odp_bench_packet-odp_bench_packet.o<br> > make[3]: *** No rule to make target '../../../lib/libodp-linux.la',<br> > needed by 'odp_bench_packet'.<br> > <br> > . CC atomic_main.o<br> > make[5]: *** No rule to make target<br> > '../../../../../lib/libodp-linux.la', needed by 'atomic_main'.<br> > <br> > CC classification_main.o<br> > make[5]: *** No rule to make target<br> > '../../../../../lib/libodp-linux.la', needed by 'classification_main'.<br> > <br> > ...and many others like this.<br> > <br> I cannot tell if there are any other comments in this response because it is not in plain text. Did you have more comments? > On Tue, Mar 28, 2017 at 2:23 PM, Brian Brooks <brian.brooks@arm.com> wrote:<br> > > This work derives from Ola Liljedahl's prototype [1] which introduced a<br> > > scalable scheduler design based on primarily lock-free algorithms and<br> > > data structures designed to decrease contention. A thread searches<br> > > through a data structure containing only queues that are both non-empty<br> > > and allowed to be scheduled to that thread. Strict priority scheduling is<br> > > respected, and (W)RR scheduling may be used within queues of the same priority.<br> > > Lastly, pre-scheduling or stashing is not employed since it is optional<br> > > functionality that can be implemented in the application.<br> > ><br> > > In addition to scalable ring buffers, the algorithm also uses unbounded<br> > > concurrent queues. LL/SC and CAS variants exist in cases where absense of<br> > > ABA problem cannot be proved, and also in cases where the compiler's atomic<br> > > built-ins may not be lowered to the desired instruction(s). Finally, a version<br> > > of the algorithm that uses locks is also provided.<br> > ><br> > > See platform/linux-generic/include/odp_config_internal.h for further build<br> > > time configuration.<br> > ><br> > > Use --enable-schedule-scalable to conditionally compile this scheduler<br> > > into the library.<br> > ><br> > > [1] <a href="https://lists.linaro.org/pipermail/lng-odp/2016-September/025682.html">https://lists.linaro.org/pipermail/lng-odp/2016-September/025682.html</a><br> > ><br> > > Signed-off-by: Brian Brooks <brian.brooks@arm.com><br> > > Signed-off-by: Kevin Wang <kevin.wang@arm.com><br> > > Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com><br> > > Signed-off-by: Ola Liljedahl <ola.liljedahl@arm.com><br> > > ---<br> > > platform/linux-generic/Makefile.am | 21 +-<br> > > .../include/odp/api/plat/schedule_types.h | 20 +-<br> > > platform/linux-generic/include/odp_atomic16.h | 244 +++<br> > > platform/linux-generic/include/odp_bitset.h | 155 ++<br> > > .../linux-generic/include/odp_config_internal.h | 90 +-<br> > > platform/linux-generic/include/odp_internal.h | 2 +<br> > > platform/linux-generic/include/odp_llqueue.h | 386 ++++<br> > > platform/linux-generic/include/odp_llsc.h | 325 ++++<br> > > .../linux-generic/include/odp_packet_internal.h | 3 +<br> > > .../linux-generic/include/odp_queue_internal.h | 122 +-<br> > > platform/linux-generic/include/odp_schedule_if.h | 166 +-<br> > > .../include/odp_schedule_ordered_internal.h | 150 ++<br> > > platform/linux-generic/m4/odp_schedule.m4 | 55 +-<br> > > platform/linux-generic/odp_classification.c | 4 +-<br> > > platform/linux-generic/odp_packet.c | 5 +<br> > > platform/linux-generic/odp_packet_io.c | 88 +-<br> > > platform/linux-generic/odp_queue.c | 2 +-<br> > > platform/linux-generic/odp_queue_scalable.c | 944 ++++++++++<br> > > platform/linux-generic/odp_schedule_if.c | 36 +-<br> > > platform/linux-generic/odp_schedule_scalable.c | 1959 ++++++++++++++++++++<br> > > .../linux-generic/odp_schedule_scalable_ordered.c | 298 +++<br> > > platform/linux-generic/odp_traffic_mngr.c | 7 +-<br> > > platform/linux-generic/pktio/loop.c | 11 +-<br> > > test/common_plat/performance/odp_sched_latency.c | 69 +-<br> > > test/common_plat/performance/odp_scheduling.c | 12 +-<br> > > .../api/classification/odp_classification_basic.c | 8 +-<br> > > .../classification/odp_classification_test_pmr.c | 42 +-<br> > > .../validation/api/scheduler/scheduler.c | 11 +-<br> > > test/common_plat/validation/api/timer/timer.c | 5 +-<br> > > 29 files changed, 5123 insertions(+), 117 deletions(-)<br> > > create mode 100644 platform/linux-generic/include/odp_atomic16.h<br> > > create mode 100644 platform/linux-generic/include/odp_bitset.h<br> > > create mode 100644 platform/linux-generic/include/odp_llqueue.h<br> > > create mode 100644 platform/linux-generic/include/odp_llsc.h<br> > > create mode 100644 platform/linux-generic/include/odp_schedule_ordered_internal.h<br> > > create mode 100644 platform/linux-generic/odp_queue_scalable.c<br> > > create mode 100644 platform/linux-generic/odp_schedule_scalable.c<br> > > create mode 100644 platform/linux-generic/odp_schedule_scalable_ordered.c<br> > ><br> > > diff --git a/platform/linux-generic/Makefile.am b/platform/linux-generic/Makefile.am<br> > > index 70683cac..8c263b99 100644<br> > > --- a/platform/linux-generic/Makefile.am<br> > > +++ b/platform/linux-generic/Makefile.am<br> > > @@ -151,6 +151,8 @@ noinst_HEADERS = \<br> > > ${srcdir}/include/odp_debug_internal.h \<br> > > ${srcdir}/include/odp_forward_typedefs_internal.h \<br> > > ${srcdir}/include/odp_internal.h \<br> > > + ${srcdir}/include/odp_llqueue.h \<br> > > + ${srcdir}/include/odp_llsc.h \<br> > > ${srcdir}/include/odp_name_table_internal.h \<br> > > ${srcdir}/include/odp_packet_internal.h \<br> > > ${srcdir}/include/odp_packet_io_internal.h \<br> > > @@ -219,13 +221,9 @@ __LIB__libodp_linux_la_SOURCES = \<br> > > pktio/ring.c \<br> > > odp_pkt_queue.c \<br> > > odp_pool.c \<br> > > - odp_queue.c \<br> > > odp_rwlock.c \<br> > > odp_rwlock_recursive.c \<br> > > - odp_schedule.c \<br> > > odp_schedule_if.c \<br> > > - odp_schedule_sp.c \<br> > > - odp_schedule_iquery.c \<br> > > odp_shared_memory.c \<br> > > odp_sorted_list.c \<br> > > odp_spinlock.c \<br> > > @@ -250,6 +248,21 @@ __LIB__libodp_linux_la_SOURCES = \<br> > > arch/@ARCH_DIR@/odp_cpu_arch.c \<br> > > arch/@ARCH_DIR@/odp_sysinfo_parse.c<br> > ><br> > > +if ODP_SCHEDULE_SP<br> > > +__LIB__libodp_linux_la_SOURCES += odp_schedule_sp.c<br> > > +endif<br> > > +<br> > > +if ODP_SCHEDULE_IQUERY<br> > > +__LIB__libodp_linux_la_SOURCES += odp_schedule_iquery.c<br> > > +endif<br> > > +<br> > > +if ODP_SCHEDULE_SCALABLE<br> > > +__LIB__libodp_linux_la_SOURCES += odp_queue_scalable.c odp_schedule_scalable.c \<br> > > + odp_schedule_scalable_ordered.c<br> > > +else<br> > > +__LIB__libodp_linux_la_SOURCES += odp_queue.c odp_schedule.c<br> > > +endif<br> > > +<br> > > if HAVE_PCAP<br> > > __LIB__libodp_linux_la_SOURCES += pktio/pcap.c<br> > > endif<br> > > diff --git a/platform/linux-generic/include/odp/api/plat/schedule_types.h b/platform/linux-generic/include/odp/api/plat/schedule_types.h<br> > > index 535fd6d0..d9bb39a6 100644<br> > > --- a/platform/linux-generic/include/odp/api/plat/schedule_types.h<br> > > +++ b/platform/linux-generic/include/odp/api/plat/schedule_types.h<br> > > @@ -18,6 +18,8 @@<br> > > extern "C" {<br> > > #endif<br> > ><br> > > +#include <odp/api/std_types.h><br> > > +<br> > > /** @addtogroup odp_scheduler<br> > > * @{<br> > > */<br> > > @@ -27,6 +29,20 @@ extern "C" {<br> > ><br> > > typedef int odp_schedule_prio_t;<br> > ><br> > > +#ifdef ODP_SCHEDULE_SCALABLE<br> > > +<br> > > +#define ODP_SCHED_PRIO_NUM 8<br> > > +<br> > > +#define ODP_SCHED_PRIO_HIGHEST 0<br> > > +<br> > > +#define ODP_SCHED_PRIO_LOWEST (ODP_SCHED_PRIO_NUM - 1)<br> > > +<br> > > +#define ODP_SCHED_PRIO_DEFAULT (ODP_SCHED_PRIO_NUM / 2)<br> > > +<br> > > +#define ODP_SCHED_PRIO_NORMAL ODP_SCHED_PRIO_DEFAULT<br> > > +<br> > > +#else<br> > > +<br> > > #define ODP_SCHED_PRIO_HIGHEST 0<br> > ><br> > > #define ODP_SCHED_PRIO_NORMAL 4<br> > > @@ -35,6 +51,8 @@ typedef int odp_schedule_prio_t;<br> > ><br> > > #define ODP_SCHED_PRIO_DEFAULT ODP_SCHED_PRIO_NORMAL<br> > ><br> > > +#endif<br> > > +<br> > > typedef int odp_schedule_sync_t;<br> > ><br> > > #define ODP_SCHED_SYNC_PARALLEL 0<br> > > @@ -44,7 +62,7 @@ typedef int odp_schedule_sync_t;<br> > > typedef int odp_schedule_group_t;<br> > ><br> > > /* These must be kept in sync with thread_globals_t in odp_thread.c */<br> > > -#define ODP_SCHED_GROUP_INVALID -1<br> > > +#define ODP_SCHED_GROUP_INVALID ((odp_schedule_group_t) -1)<br> > > #define ODP_SCHED_GROUP_ALL 0<br> > > #define ODP_SCHED_GROUP_WORKER 1<br> > > #define ODP_SCHED_GROUP_CONTROL 2<br> > > diff --git a/platform/linux-generic/include/odp_atomic16.h b/platform/linux-generic/include/odp_atomic16.h<br> > > new file mode 100644<br> > > index 00000000..5afd66c5<br> > > --- /dev/null<br> > > +++ b/platform/linux-generic/include/odp_atomic16.h<br> > > @@ -0,0 +1,244 @@<br> > > +/* Copyright (c) 2017, ARM Limited.<br> > > + * All rights reserved.<br> > > + *<br> > > + * SPDX-License-Identifier: BSD-3-Clause<br> > > + */<br> > > +<br> > > +#ifndef _ODP_ATOMIC16_H_<br> > > +#define _ODP_ATOMIC16_H_<br> > > +<br> > > +#include "odp_llsc.h"<br> > > +<br> > > +#if defined __ARM_ARCH && __ARM_ARCH == 8<br> > > +<br> > > +#ifdef __ARM_FEATURE_QRDMX /* Feature only available in v8.1a and beyond */<br> > > +static inline __int128 casp(__int128 *var, __int128 old, __int128 neu, int mo)<br> > > +{<br> > > + if (mo == __ATOMIC_RELAXED) {<br> > > + __asm __volatile("casp %0, %H0, %1, %H1, [%2]"<br> > > + : "+r" (old)<br> > > + : "r" (neu), "r" (var)<br> > > + : "memory");<br> > > + } else if (mo == __ATOMIC_ACQUIRE) {<br> > > + __asm __volatile("caspa %0, %H0, %1, %H1, [%2]"<br> > > + : "+r" (old)<br> > > + : "r" (neu), "r" (var)<br> > > + : "memory");<br> > > + } else if (mo == __ATOMIC_ACQ_REL) {<br> > > + __asm __volatile("caspal %0, %H0, %1, %H1, [%2]"<br> > > + : "+r" (old)<br> > > + : "r" (neu), "r" (var)<br> > > + : "memory");<br> > > + } else if (mo == __ATOMIC_RELEASE) {<br> > > + __asm __volatile("caspl %0, %H0, %1, %H1, [%2]"<br> > > + : "+r" (old)<br> > > + : "r" (neu), "r" (var)<br> > > + : "memory");<br> > > + } else {<br> > > + abort();<br> > > + }<br> > > + return old;<br> > > +}<br> > > +#endif<br> > > +<br> > > +static inline bool __atomic_compare_exchange_16(register __int128 *var,<br> > > + __int128 *exp, register __int128 neu,<br> > > + bool weak, int mo_success, int mo_failure)<br> > > +{<br> > > +#ifdef __ARM_FEATURE_QRDMX /* Feature only available in v8.1a and beyond */<br> > > + (void)weak;<br> > > + (void)mo_failure;<br> > > + __int128 old;<br> > > + __int128 expected;<br> > > +<br> > > + expected = *exp;<br> > > + old = casp(var, expected, neu, mo_success);<br> > > + *exp = old; /* Always update, atomically read value */<br> > > + return old == expected;<br> > > +#else<br> > > + (void)weak; /* Always do strong CAS or we can't perform atomic read */<br> > > + /* Ignore memory ordering for failure, memory order for<br> > > + * success must be stronger or equal<br> > > + */<br> > > + (void)mo_failure;<br> > > + int ll_mo;<br> > > + int sc_mo;<br> > > +<br> > > + register __int128 old;<br> > > + register __int128 expected;<br> > > +<br> > > + ll_mo = mo_success == __ATOMIC_ACQUIRE ||<br> > > + mo_success == __ATOMIC_ACQ_REL ?<br> > > + __ATOMIC_ACQUIRE : __ATOMIC_RELAXED;<br> > > +<br> > > + sc_mo = mo_success == __ATOMIC_RELEASE ||<br> > > + mo_success == __ATOMIC_ACQ_REL ?<br> > > + __ATOMIC_RELEASE : __ATOMIC_RELAXED;<br> > > +<br> > > + expected = *exp;<br> > > + __asm __volatile("" ::: "memory");<br> > > + do {<br> > > + /* Atomicity of LLD is not guaranteed */<br> > > + old = lld(var, ll_mo);<br> > > + /* Must write back neu or old to verify atomicity of LLD */<br> > > + } while (odp_unlikely(scd(var, old == expected ? neu : old, sc_mo)));<br> > > + *exp = old; /* Always update, atomically read value */<br> > > + return old == expected;<br> > > +#endif<br> > > +}<br> > > +<br> > > +static inline bool __atomic_compare_exchange_16_frail(register __int128 *var,<br> > > + __int128 *exp, register __int128 neu, bool weak,<br> > > + int mo_success, int mo_failure)<br> > > +{<br> > > +#ifdef __ARM_FEATURE_QRDMX /* Feature only available in v8.1a and beyond */<br> > > + (void)weak;<br> > > + (void)mo_failure;<br> > > + __int128 old;<br> > > + __int128 expected;<br> > > +<br> > > + expected = *exp;<br> > > + old = casp(var, expected, neu, mo_success);<br> > > + *exp = old; /* Always update, atomically read value */<br> > > + return old == expected;<br> > > +#else<br> > > + (void)weak; /* Weak CAS and non-atomic load on failure */<br> > > + (void)mo_failure; /* Ignore memory ordering for failure */<br> > > + int ll_mo;<br> > > + int sc_mo;<br> > > + register __int128 expected;<br> > > + register __int128 old;<br> > > +<br> > > + /* memory order for success must be stronger or equal */<br> > > + ll_mo = mo_success == __ATOMIC_ACQUIRE ||<br> > > + mo_success == __ATOMIC_ACQ_REL ?<br> > > + __ATOMIC_ACQUIRE : __ATOMIC_RELAXED;<br> > > + sc_mo = mo_success == __ATOMIC_RELEASE ||<br> > > + mo_success == __ATOMIC_ACQ_REL ?<br> > > + __ATOMIC_RELEASE : __ATOMIC_RELAXED;<br> > > + expected = *exp;<br> > > + __asm __volatile("" ::: "memory");<br> > > + /* Atomicity of LLD is not guaranteed */<br> > > + old = lld(var, ll_mo);<br> > > + if (odp_likely(old == expected && !scd(var, neu, sc_mo))) {<br> > > + /* Right value and SC succeeded */<br> > > + __asm __volatile("" ::: "memory");<br> > > + return 1;<br> > > + }<br> > > + __asm __volatile("" ::: "memory");<br> > > + /* Wrong value or SC failed */<br> > > + *exp = old; /* Old possibly torn value */<br> > > + return 0; /* Failure, *exp updated */<br> > > +#endif<br> > > +}<br> > > +<br> > > +static inline __int128 __atomic_load_16(__int128 *var, int mo)<br> > > +{<br> > > + __int128 old = *var; /* Possibly torn read */<br> > > +<br> > > + /* Do CAS to ensure atomicity<br> > > + * Either CAS succeeds (writing back the same value)<br> > > + * Or CAS fails and returns the old value (atomic read)<br> > > + */<br> > > + (void)__atomic_compare_exchange_n(var,<br> > > + &old,<br> > > + old,<br> > > + false, /* weak= */<br> > > + mo,<br> > > + mo);<br> > > + return old;<br> > > +}<br> > > +<br> > > +static inline __int128 __atomic_exchange_16(__int128 *var,<br> > > + __int128 neu, int mo)<br> > > +{<br> > > +#ifdef __ARM_FEATURE_QRDMX /* Feature only available in v8.1a and beyond */<br> > > + __int128 old;<br> > > + __int128 expected;<br> > > +<br> > > + do {<br> > > + expected = *var;<br> > > + old = casp(var, expected, neu, mo);<br> > > + } while (old != expected);<br> > > + return old;<br> > > +#else<br> > > + int ll_mo;<br> > > + int sc_mo;<br> > > + register __int128 old;<br> > > +<br> > > + ll_mo = mo == __ATOMIC_ACQUIRE || mo == __ATOMIC_ACQ_REL ?<br> > > + __ATOMIC_ACQUIRE : __ATOMIC_RELAXED;<br> > > + sc_mo = mo == __ATOMIC_RELEASE || mo == __ATOMIC_ACQ_REL ?<br> > > + __ATOMIC_RELEASE : __ATOMIC_RELAXED;<br> > > + do {<br> > > + /* Atomicity of LLD is not guaranteed */<br> > > + old = lld(var, ll_mo);<br> > > + /* Must successfully write back to verify atomicity of LLD */<br> > > + } while (odp_unlikely(scd(var, neu, sc_mo)));<br> > > + return old;<br> > > +#endif<br> > > +}<br> > > +<br> > > +static inline __int128 __atomic_fetch_and_16(__int128 *var,<br> > > + __int128 mask, int mo)<br> > > +{<br> > > +#ifdef __ARM_FEATURE_QRDMX /* Feature only available in v8.1a and beyond */<br> > > + __int128 old;<br> > > + __int128 expected;<br> > > +<br> > > + do {<br> > > + expected = *var;<br> > > + old = casp(var, expected, expected & mask, mo);<br> > > + } while (old != expected);<br> > > + return old;<br> > > +#else<br> > > + int ll_mo;<br> > > + int sc_mo;<br> > > + register __int128 old;<br> > > +<br> > > + ll_mo = mo == __ATOMIC_ACQUIRE || mo == __ATOMIC_ACQ_REL ?<br> > > + __ATOMIC_ACQUIRE : __ATOMIC_RELAXED;<br> > > + sc_mo = mo == __ATOMIC_RELEASE || mo == __ATOMIC_ACQ_REL ?<br> > > + __ATOMIC_RELEASE : __ATOMIC_RELAXED;<br> > > + do {<br> > > + /* Atomicity of LLD is not guaranteed */<br> > > + old = lld(var, ll_mo);<br> > > + /* Must successfully write back to verify atomicity of LLD */<br> > > + } while (odp_unlikely(scd(var, old & mask, sc_mo)));<br> > > + return old;<br> > > +#endif<br> > > +}<br> > > +<br> > > +static inline __int128 __atomic_fetch_or_16(__int128 *var,<br> > > + __int128 mask,<br> > > + int mo)<br> > > +{<br> > > +#ifdef __ARM_FEATURE_QRDMX /* Feature only available in v8.1a and beyond */<br> > > + __int128 old;<br> > > + __int128 expected;<br> > > +<br> > > + do {<br> > > + expected = *var;<br> > > + old = casp(var, expected, expected | mask, mo);<br> > > + } while (old != expected);<br> > > + return old;<br> > > +#else<br> > > + int ll_mo;<br> > > + int sc_mo;<br> > > + register __int128 old;<br> > > +<br> > > + ll_mo = mo == __ATOMIC_ACQUIRE || mo == __ATOMIC_ACQ_REL ?<br> > > + __ATOMIC_ACQUIRE : __ATOMIC_RELAXED;<br> > > + sc_mo = mo == __ATOMIC_RELEASE || mo == __ATOMIC_ACQ_REL ?<br> > > + __ATOMIC_RELEASE : __ATOMIC_RELAXED;<br> > > + do {<br> > > + /* Atomicity of LLD is not guaranteed */<br> > > + old = lld(var, ll_mo);<br> > > + /* Must successfully write back to verify atomicity of LLD */<br> > > + } while (odp_unlikely(scd(var, old | mask, sc_mo)));<br> > > + return old;<br> > > +#endif<br> > > +}<br> > > +<br> > > +#endif<br> > > +#endif<br> > > diff --git a/platform/linux-generic/include/odp_bitset.h b/platform/linux-generic/include/odp_bitset.h<br> > > new file mode 100644<br> > > index 00000000..db004267<br> > > --- /dev/null<br> > > +++ b/platform/linux-generic/include/odp_bitset.h<br> > > @@ -0,0 +1,155 @@<br> > > +/* Copyright (c) 2017, ARM Limited<br> > > + * All rights reserved.<br> > > + *<br> > > + * SPDX-License-Identifier: BSD-3-Clause<br> > > + */<br> > > +<br> > > +#ifndef _ODP_BITSET_H_<br> > > +#define _ODP_BITSET_H_<br> > > +<br> > > +/******************************************************************************<br> > > + * bitset abstract data type<br> > > + *****************************************************************************/<br> > > +/* This could be a struct of scalars to support larger bit sets */<br> > > +<br> > > +#if ATOM_BITSET_SIZE <= 32<br> > > +<br> > > +typedef uint32_t bitset_t;<br> > > +<br> > > +static inline bitset_t bitset_mask(uint32_t bit)<br> > > +{<br> > > + return 1UL << bit;<br> > > +}<br> > > +<br> > > +/* Return first-bit-set with StdC ffs() semantics */<br> > > +static inline uint32_t bitset_ffs(bitset_t b)<br> > > +{<br> > > + return __builtin_ffsl(b);<br> > > +}<br> > > +<br> > > +/* Load-exclusive with memory ordering */<br> > > +static inline bitset_t bitset_ldex(bitset_t *bs, int mo)<br> > > +{<br> > > + return LDXR32(bs, mo);<br> > > +}<br> > > +<br> > > +#elif ATOM_BITSET_SIZE <= 64<br> > > +<br> > > +typedef uint64_t bitset_t;<br> > > +<br> > > +static inline bitset_t bitset_mask(uint32_t bit)<br> > > +{<br> > > + return 1ULL << bit;<br> > > +}<br> > > +<br> > > +/* Return first-bit-set with StdC ffs() semantics */<br> > > +static inline uint32_t bitset_ffs(bitset_t b)<br> > > +{<br> > > + return __builtin_ffsll(b);<br> > > +}<br> > > +<br> > > +/* Load-exclusive with memory ordering */<br> > > +static inline bitset_t bitset_ldex(bitset_t *bs, int mo)<br> > > +{<br> > > + return LDXR64(bs, mo);<br> > > +}<br> > > +<br> > > +#elif ATOM_BITSET_SIZE <= 128<br> > > +<br> > > +#if __SIZEOF_INT128__ == 16<br> > > +typedef unsigned __int128 bitset_t;<br> > > +<br> > > +static inline bitset_t bitset_mask(uint32_t bit)<br> > > +{<br> > > + if (bit < 64)<br> > > + return 1ULL << bit;<br> > > + else<br> > > + return (unsigned __int128)(1ULL << (bit - 64)) << 64;<br> > > +}<br> > > +<br> > > +/* Return first-bit-set with StdC ffs() semantics */<br> > > +static inline uint32_t bitset_ffs(bitset_t b)<br> > > +{<br> > > + if ((uint64_t)b != 0)<br> > > + return __builtin_ffsll((uint64_t)b);<br> > > + else if ((b >> 64) != 0)<br> > > + return __builtin_ffsll((uint64_t)(b >> 64)) + 64;<br> > > + else<br> > > + return 0;<br> > > +}<br> > > +<br> > > +/* Load-exclusive with memory ordering */<br> > > +static inline bitset_t bitset_ldex(bitset_t *bs, int mo)<br> > > +{<br> > > + return LDXR128(bs, mo);<br> > > +}<br> > > +<br> > > +#else<br> > > +#error __int128 not supported by compiler<br> > > +#endif<br> > > +<br> > > +#else<br> > > +#error Unsupported size of bit sets (ATOM_BITSET_SIZE)<br> > > +#endif<br> > > +<br> > > +/* Atomic load with memory ordering */<br> > > +static inline bitset_t atom_bitset_load(bitset_t *bs, int mo)<br> > > +{<br> > > + return __atomic_load_n(bs, mo);<br> > > +}<br> > > +<br> > > +/* Atomic bit set with memory ordering */<br> > > +static inline void atom_bitset_set(bitset_t *bs, uint32_t bit, int mo)<br> > > +{<br> > > + (void)__atomic_fetch_or(bs, bitset_mask(bit), mo);<br> > > +}<br> > > +<br> > > +/* Atomic bit clear with memory ordering */<br> > > +static inline void atom_bitset_clr(bitset_t *bs, uint32_t bit, int mo)<br> > > +{<br> > > + (void)__atomic_fetch_and(bs, ~bitset_mask(bit), mo);<br> > > +}<br> > > +<br> > > +/* Atomic exchange with memory ordering */<br> > > +static inline bitset_t atom_bitset_xchg(bitset_t *bs, bitset_t neu, int mo)<br> > > +{<br> > > + return __atomic_exchange_n(bs, neu, mo);<br> > > +}<br> > > +<br> > > +/* Return a & ~b */<br> > > +static inline bitset_t bitset_andn(bitset_t a, bitset_t b)<br> > > +{<br> > > + return a & ~b;<br> > > +}<br> > > +<br> > > +static inline bool bitset_is_eql(bitset_t a, bitset_t b)<br> > > +{<br> > > + return a == b;<br> > > +}<br> > > +<br> > > +static inline bitset_t bitset_clr(bitset_t bs, uint32_t bit)<br> > > +{<br> > > + return bs & ~bitset_mask(bit);<br> > > +}<br> > > +<br> > > +static inline bitset_t bitset_set(bitset_t bs, uint32_t bit)<br> > > +{<br> > > + return bs | bitset_mask(bit);<br> > > +}<br> > > +<br> > > +static inline bitset_t bitset_null(void)<br> > > +{<br> > > + return 0U;<br> > > +}<br> > > +<br> > > +static inline bool bitset_is_null(bitset_t a)<br> > > +{<br> > > + return a == 0U;<br> > > +}<br> > > +<br> > > +static inline bool bitset_is_set(bitset_t a, uint32_t bit)<br> > > +{<br> > > + return (a & bitset_mask(bit)) != 0;<br> > > +}<br> > > +<br> > > +#endif<br> > > diff --git a/platform/linux-generic/include/odp_config_internal.h b/platform/linux-generic/include/odp_config_internal.h<br> > > index dadd59e7..b1ee48f2 100644<br> > > --- a/platform/linux-generic/include/odp_config_internal.h<br> > > +++ b/platform/linux-generic/include/odp_config_internal.h<br> > > @@ -22,6 +22,13 @@ extern "C" {<br> > > #define ODP_CONFIG_QUEUES 1024<br> > ><br> > > /*<br> > > + * Maximum queue depth. Maximum number of elements that can be stored in a<br> > > + * queue. This value is used only when the size is not explicitly provided<br> > > + * during queue creation.<br> > > + */<br> > > +#define ODP_CONFIG_QUEUE_SIZE 4096<br> > > +<br> > > +/*<br> > > * Maximum number of ordered locks per queue<br> > > */<br> > > #define CONFIG_QUEUE_MAX_ORD_LOCKS 4<br> > > @@ -104,7 +111,7 @@ extern "C" {<br> > > *<br> > > * This the the number of separate SHM areas that can be reserved concurrently<br> > > */<br> > > -#define ODP_CONFIG_SHM_BLOCKS (ODP_CONFIG_POOLS + 48)<br> > > +#define ODP_CONFIG_SHM_BLOCKS (ODP_CONFIG_POOLS + ODP_CONFIG_QUEUES + 48)<br> > ><br> > > /*<br> > > * Size of the virtual address space pre-reserver for ISHM<br> > > @@ -120,7 +127,7 @@ extern "C" {<br> > > *<br> > > * This the the number of separate SHM areas that can be reserved concurrently<br> > > */<br> > > -#define ODPDRV_CONFIG_SHM_BLOCKS 48<br> > > +#define ODPDRV_CONFIG_SHM_BLOCKS ODP_CONFIG_SHM_BLOCKS<br> > ><br> > > /* Maximum event burst size<br> > > *<br> > > @@ -129,6 +136,21 @@ extern "C" {<br> > > */<br> > > #define CONFIG_BURST_SIZE 16<br> > ><br> > > +/* Default weight (in events) for WRR in scalable scheduler<br> > > + *<br> > > + * This controls the per-queue weight for WRR between queues of the same<br> > > + * priority in the scalable scheduler<br> > > + * A higher value improves throughput while a lower value increases fairness<br> > > + * and thus likely decreases latency<br> > > + *<br> > > + * If WRR is undesired, set the value to ~0 which will use the largest possible<br> > > + * weight<br> > > + *<br> > > + * Note: an API for specifying this on a per-queue basis would be useful but is<br> > > + * not yet available<br> > > + */<br> > > +#define CONFIG_WRR_WEIGHT 64<br> > > +<br> > > /*<br> > > * Maximum number of events in a pool<br> > > */<br> > > @@ -139,6 +161,70 @@ extern "C" {<br> > > */<br> > > #define CONFIG_POOL_CACHE_SIZE 256<br> > ><br> > > +/*<br> > > + * Split queue producer/consumer metadata into separate cache lines.<br> > > + * This is beneficial on e.g. Cortex-A57 but not so much on A53.<br> > > + */<br> > > +#define ODP_CONFIG_USE_SPLIT_PRODCONS<br> > > +<br> > > +/*<br> > > + * Split queue read/write metadata into separate cache lines.<br> > > + * This enhances scalability even further on Cortex-A57.<br> > > + */<br> > > +#define ODP_CONFIG_USE_SPLIT_READWRITE<br> > > +<br> > > +/*<br> > > + * Use locks to protect queue (ring buffer) and scheduler state updates<br> > > + * On x86, this decreases overhead but also degrades scalability<br> > > + */<br> > > +#ifndef __ARM_ARCH<br> > > +#define CONFIG_QSCHST_LOCK<br> > > +#endif<br> > > +<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > +/* Keep all ring buffer/qschst data together when using locks<br> > > + */<br> > > +#undef ODP_CONFIG_USE_SPLIT_PRODCONS<br> > > +#undef ODP_CONFIG_USE_SPLIT_READWRITE<br> > > +#endif<br> > > +<br> > > +/*<br> > > + * Use spin lock instead of (lock-free) atomic operations in llqueue<br> > > + * This is recommended on architectures which implement CAS instead<br> > > + * of LL/SC as CAS is susceptible to the ABA problem<br> > > + */<br> > > +#ifndef __ARM_ARCH<br> > > +#define ODP_CONFIG_LLQ_LOCK<br> > > +#endif<br> > > +<br> > > +/*<br> > > + * Size of atomic bit set. This limits the max number of threads,<br> > > + * scheduler groups and reorder windows. On ARMv8/64-bit and x86-64, the<br> > > + * (lock-free) max is 128<br> > > + */<br> > > +#define ATOM_BITSET_SIZE 64<br> > > +<br> > > +/*<br> > > + * Use LL/SC atomic primitives instead of __atomic_compare_exchange built-ins<br> > > + */<br> > > +#ifdef __ARM_ARCH<br> > > +#define ODP_CONFIG_USE_LLSC<br> > > +#endif<br> > > +<br> > > +/*<br> > > + * Use DMB;STR instead of STRL on ARM<br> > > + */<br> > > +#ifdef __ARM_ARCH<br> > > +#define ODP_CONFIG_USE_DMB<br> > > +#endif<br> > > +<br> > > +/*<br> > > + * Use ARM event signalling mechanism<br> > > + */<br> > > +#ifdef __ARM_ARCH<br> > > +#define ODP_CONFIG_USE_WFE<br> > > +#endif<br> > > +<br> > > #ifdef __cplusplus<br> > > }<br> > > #endif<br> > > diff --git a/platform/linux-generic/include/odp_internal.h b/platform/linux-generic/include/odp_internal.h<br> > > index 05c8a422..60eb9daa 100644<br> > > --- a/platform/linux-generic/include/odp_internal.h<br> > > +++ b/platform/linux-generic/include/odp_internal.h<br> > > @@ -28,6 +28,8 @@ extern __thread int __odp_errno;<br> > ><br> > > #define MAX_CPU_NUMBER 128<br> > ><br> > > +#define IS_POWER_TWO(x) ((((x) - 1) & (x)) == 0)<br> > > +<br> > > typedef struct {<br> > > uint64_t cpu_hz_max[MAX_CPU_NUMBER];<br> > > uint64_t page_size;<br> > > diff --git a/platform/linux-generic/include/odp_llqueue.h b/platform/linux-generic/include/odp_llqueue.h<br> > > new file mode 100644<br> > > index 00000000..6f20c641<br> > > --- /dev/null<br> > > +++ b/platform/linux-generic/include/odp_llqueue.h<br> > > @@ -0,0 +1,386 @@<br> > > +/* Copyright (c) 2017, ARM Limited.<br> > > + * All rights reserved.<br> > > + *<br> > > + * SPDX-License-Identifier: BSD-3-Clause<br> > > + */<br> > > +<br> > > +#ifndef ODP_LLQUEUE_H_<br> > > +#define ODP_LLQUEUE_H_<br> > > +<br> > > +#include <odp/api/cpu.h><br> > > +#include <odp/api/hints.h><br> > > +#include <odp/api/spinlock.h><br> > > +<br> > > +#include <odp_config_internal.h><br> > > +#include <odp_debug_internal.h><br> > > +#include <odp_llsc.h><br> > > +<br> > > +#include <stdint.h><br> > > +#include <stdlib.h><br> > > +<br> > > +/******************************************************************************<br> > > + * Linked list queues<br> > > + *****************************************************************************/<br> > > +<br> > > +#if defined(__ARM_ARCH) && !defined(ODP_CONFIG_USE_LLSC) && \<br> > > + !defined(ODP_CONFIG_LLQ_LOCK)<br> > > +#error ODP_CONFIG_USE_LLSC required on ARM for double-word atomics if not \<br> > > + using spinlock<br> > > +#endif<br> > > +<br> > > +/* The scalar equivalent of a double pointer */<br> > > +#if __SIZEOF_PTRDIFF_T__ == 4<br> > > +typedef uint64_t dintptr_t;<br> > > +#endif<br> > > +#if __SIZEOF_PTRDIFF_T__ == 8<br> > > +typedef __int128 dintptr_t;<br> > > +#endif<br> > > +<br> > > +#define SENTINEL ((void *)~(uintptr_t)0)<br> > > +<br> > > +struct llnode {<br> > > + struct llnode *next;<br> > > +};<br> > > +<br> > > +union llht {<br> > > + struct {<br> > > + struct llnode *head, *tail;<br> > > + } st;<br> > > + dintptr_t ui;<br> > > +};<br> > > +<br> > > +struct llqueue {<br> > > + union llht u;<br> > > +#ifdef ODP_CONFIG_LLQ_LOCK<br> > > + odp_spinlock_t lock;<br> > > +#endif<br> > > +};<br> > > +<br> > > +static inline struct llnode *llq_head(struct llqueue *llq)<br> > > +{<br> > > + return __atomic_load_n(&llq->u.st.head, __ATOMIC_RELAXED);<br> > > +}<br> > > +<br> > > +static inline void llqueue_init(struct llqueue *llq)<br> > > +{<br> > > + llq->u.st.head = NULL;<br> > > + llq->u.st.tail = NULL;<br> > > +#ifdef ODP_CONFIG_LLQ_LOCK<br> > > + odp_spinlock_init(&llq->lock);<br> > > +#endif<br> > > +}<br> > > +<br> > > +#ifndef ODP_CONFIG_LLQ_LOCK<br> > > +<br> > > +static inline void llq_enqueue(struct llqueue *llq, struct llnode *node)<br> > > +{<br> > > + union llht old;<br> > > + union llht neu;<br> > > +<br> > > + ODP_ASSERT(node->next == NULL);<br> > > + node->next = SENTINEL;<br> > > +#ifdef ODP_CONFIG_USE_LLSC<br> > > + /* Prefetch for store for faster LL/SC execution */<br> > > + __builtin_prefetch(&llq->u.ui, 1, 0);<br> > > +retry: /* Failed SC requires new LL */<br> > > + old.ui = lld(&llq->u.ui, __ATOMIC_RELAXED);<br> > > +#else<br> > > + __atomic_load(&llq->u, &old, __ATOMIC_RELAXED);<br> > > +retry: /* Failed CAS returns existing value */<br> > > + (void)0; /* Need statement after label */<br> > > +#endif<br> > > + neu.st.head = old.st.head == NULL ? node : old.st.head;<br> > > + neu.st.tail = node;<br> > > +#ifdef ODP_CONFIG_USE_LLSC<br> > > + if (odp_unlikely(scd(&llq->u.ui, neu.ui, __ATOMIC_RELEASE))) {<br> > > +#else<br> > > + if (odp_unlikely(!__atomic_compare_exchange(&llq->u,<br> > > + &old,<br> > > + &neu,<br> > > + false, /* weak= */<br> > > + __ATOMIC_RELEASE,<br> > > + __ATOMIC_RELAXED))) {<br> > > +#endif<br> > > + odp_cpu_pause();<br> > > + goto retry;<br> > > + }<br> > > + if (old.st.tail != NULL) {<br> > > + /* List was not empty */<br> > > + ODP_ASSERT(old.st.tail->next == SENTINEL);<br> > > + old.st.tail->next = node;<br> > > + }<br> > > +}<br> > > +<br> > > +#else<br> > > +<br> > > +static inline void llq_enqueue(struct llqueue *llq, struct llnode *node)<br> > > +{<br> > > + ODP_ASSERT(node->next == NULL);<br> > > + node->next = SENTINEL;<br> > > +<br> > > + odp_spinlock_lock(&llq->lock);<br> > > + if (llq->u.st.head == NULL) {<br> > > + llq->u.st.head = llq->u.st.tail = node;<br> > > + } else {<br> > > + llq->u.st.tail->next = node;<br> > > + llq->u.st.tail = node;<br> > > + }<br> > > + odp_spinlock_unlock(&llq->lock);<br> > > +}<br> > > +<br> > > +#endif<br> > > +<br> > > +#ifndef ODP_CONFIG_LLQ_LOCK<br> > > +<br> > > +static inline struct llnode *llq_dequeue(struct llqueue *llq)<br> > > +{<br> > > + struct llnode *head;<br> > > + union llht old;<br> > > +<br> > > + /* llq_dequeue() may be used in a busy-waiting fashion<br> > > + * Read head using plain load to avoid disturbing remote LL/SC<br> > > + */<br> > > + head = __atomic_load_n(&llq->u.st.head, __ATOMIC_ACQUIRE);<br> > > + if (head == NULL)<br> > > + return NULL;<br> > > + /* Read head->next before LL to minimize cache miss latency<br> > > + * in LL/SC below<br> > > + */<br> > > + (void)__atomic_load_n(&head->next, __ATOMIC_RELAXED);<br> > > +<br> > > +#ifdef ODP_CONFIG_USE_LLSC<br> > > + /* Prefetch for store for faster LL/SC execution */<br> > > + __builtin_prefetch(&llq->u.ui, 1, 0);<br> > > +retry: /* Failed SC requires new LL */<br> > > + old.ui = lld(&llq->u.ui, __ATOMIC_RELAXED);<br> > > +#else<br> > > + __atomic_load(&llq->u, &old, __ATOMIC_RELAXED);<br> > > +retry: /* Failed CAS returns existing value */<br> > > +#endif<br> > > + if (odp_unlikely(old.st.head == NULL)) {<br> > > + /* Empty list */<br> > > + return NULL;<br> > > + } else if (odp_unlikely(old.st.head == old.st.tail)) {<br> > > + /* Single-element in list */<br> > > + union llht neu;<br> > > +<br> > > + neu.st.head = NULL;<br> > > + neu.st.tail = NULL;<br> > > +#ifdef ODP_CONFIG_USE_LLSC<br> > > + if (odp_unlikely(scd(&llq->u.ui, neu.ui, __ATOMIC_RELAXED))) {<br> > > +#else<br> > > + if (odp_unlikely(!__atomic_compare_exchange(<br> > > + &llq->u,<br> > > + &old,<br> > > + &neu,<br> > > + false, /* weak= */<br> > > + __ATOMIC_RELAXED,<br> > > + __ATOMIC_RELAXED))) {<br> > > +#endif<br> > > + /* Failed */<br> > > + odp_cpu_pause();<br> > > + goto retry;<br> > > + }<br> > > + ODP_ASSERT(old.st.head->next == SENTINEL);<br> > > + } else {<br> > > + /* Multi-element list, dequeue head */<br> > > + struct llnode *next;<br> > > + union llht neu;<br> > > + /* Wait until llq_enqueue() has written true next pointer */<br> > > + while ((next = __atomic_load_n(&old.st.head->next,<br> > > + __ATOMIC_RELAXED)) == SENTINEL)<br> > > + odp_cpu_pause();<br> > > + neu.st.head = next;<br> > > + neu.st.tail = old.st.tail;<br> > > +#ifdef ODP_CONFIG_USE_LLSC<br> > > + if (odp_unlikely(scd(&llq->u.ui, neu.ui, __ATOMIC_RELAXED))) {<br> > > +#else<br> > > + if (odp_unlikely(!__atomic_compare_exchange(<br> > > + &llq->u,<br> > > + &old,<br> > > + &neu,<br> > > + false, /* weak= */<br> > > + __ATOMIC_RELAXED,<br> > > + __ATOMIC_RELAXED))) {<br> > > +#endif<br> > > + /* Failed */<br> > > + odp_cpu_pause();<br> > > + goto retry;<br> > > + }<br> > > + ODP_ASSERT(old.st.head->next != SENTINEL);<br> > > + }<br> > > + old.st.head->next = NULL;<br> > > + return old.st.head;<br> > > +}<br> > > +<br> > > +#else<br> > > +<br> > > +static inline struct llnode *llq_dequeue(struct llqueue *llq)<br> > > +{<br> > > + struct llnode *head;<br> > > + struct llnode *node = NULL;<br> > > +<br> > > + head = __atomic_load_n(&llq->u.st.head, __ATOMIC_RELAXED);<br> > > + if (head == NULL)<br> > > + return NULL;<br> > > +<br> > > + odp_spinlock_lock(&llq->lock);<br> > > + if (llq->u.st.head != NULL) {<br> > > + node = llq->u.st.head;<br> > > + if (llq->u.st.head == llq->u.st.tail) {<br> > > + ODP_ASSERT(node->next == SENTINEL);<br> > > + llq->u.st.head = llq->u.st.tail = NULL;<br> > > + } else {<br> > > + ODP_ASSERT(node->next != SENTINEL);<br> > > + llq->u.st.head = node->next;<br> > > + }<br> > > + node->next = NULL;<br> > > + }<br> > > + odp_spinlock_unlock(&llq->lock);<br> > > + return node;<br> > > +}<br> > > +<br> > > +#endif<br> > > +<br> > > +#ifndef ODP_CONFIG_LLQ_LOCK<br> > > +<br> > > +static inline odp_bool_t llq_dequeue_cond(struct llqueue *llq,<br> > > + struct llnode *exp)<br> > > +{<br> > > + union llht old;<br> > > +<br> > > +#ifdef ODP_CONFIG_USE_LLSC<br> > > + /* Prefetch for store for faster LL/SC execution */<br> > > + __builtin_prefetch(&llq->u.ui, 1, 0);<br> > > +retry: /* Failed SC requires new LL */<br> > > + old.ui = lld(&llq->u.ui, __ATOMIC_RELAXED);<br> > > +#else<br> > > + __atomic_load(&llq->u, &old, __ATOMIC_RELAXED);<br> > > +retry: /* Failed CAS returns existing value */<br> > > +#endif<br> > > + if (odp_unlikely(old.st.head == NULL || old.st.head != exp)) {<br> > > + /* Empty list or wrong head */<br> > > + return false;<br> > > + } else if (odp_unlikely(old.st.head == old.st.tail)) {<br> > > + /* Single-element in list */<br> > > + union llht neu;<br> > > +<br> > > + neu.st.head = NULL;<br> > > + neu.st.tail = NULL;<br> > > +#ifdef ODP_CONFIG_USE_LLSC<br> > > + if (odp_unlikely(scd(&llq->u.ui, neu.ui, __ATOMIC_RELAXED))) {<br> > > +#else<br> > > + if (odp_unlikely(!__atomic_compare_exchange(<br> > > + &llq->u,<br> > > + &old,<br> > > + &neu,<br> > > + false, /* weak= */<br> > > + __ATOMIC_RELAXED,<br> > > + __ATOMIC_RELAXED))) {<br> > > +#endif<br> > > + /* Failed */<br> > > + odp_cpu_pause();<br> > > + goto retry;<br> > > + }<br> > > + ODP_ASSERT(old.st.head->next == SENTINEL);<br> > > + } else {<br> > > + /* Multi-element list, dequeue head */<br> > > + struct llnode *next;<br> > > + union llht neu;<br> > > +<br> > > + /* Wait until llq_enqueue() has written true next pointer */<br> > > + while ((next = __atomic_load_n(&old.st.head->next,<br> > > + __ATOMIC_RELAXED)) == SENTINEL)<br> > > + odp_cpu_pause();<br> > > +<br> > > + neu.st.head = next;<br> > > + neu.st.tail = old.st.tail;<br> > > +#ifdef ODP_CONFIG_USE_LLSC<br> > > + if (odp_unlikely(scd(&llq->u.ui, neu.ui, __ATOMIC_RELAXED))) {<br> > > +#else<br> > > + if (odp_unlikely(!__atomic_compare_exchange(<br> > > + &llq->u,<br> > > + &old,<br> > > + &neu,<br> > > + false, /* weak= */<br> > > + __ATOMIC_RELAXED,<br> > > + __ATOMIC_RELAXED))) {<br> > > +#endif<br> > > + /* Failed */<br> > > + odp_cpu_pause();<br> > > + goto retry;<br> > > + }<br> > > + ODP_ASSERT(old.st.head->next != SENTINEL);<br> > > + }<br> > > + old.st.head->next = NULL;<br> > > + return true;<br> > > +}<br> > > +<br> > > +#else<br> > > +<br> > > +static inline odp_bool_t llq_dequeue_cond(struct llqueue *llq,<br> > > + struct llnode *node)<br> > > +{<br> > > + odp_bool_t success = false;<br> > > +<br> > > + odp_spinlock_lock(&llq->lock);<br> > > + if (odp_likely(llq->u.st.head != NULL && llq->u.st.head == node)) {<br> > > + success = true;<br> > > + if (llq->u.st.head == llq->u.st.tail) {<br> > > + ODP_ASSERT(node->next == SENTINEL);<br> > > + llq->u.st.head = llq->u.st.tail = NULL;<br> > > + } else {<br> > > + ODP_ASSERT(node->next != SENTINEL);<br> > > + llq->u.st.head = node->next;<br> > > + }<br> > > + node->next = NULL;<br> > > + }<br> > > + odp_spinlock_unlock(&llq->lock);<br> > > + return success;<br> > > +}<br> > > +<br> > > +#endif<br> > > +<br> > > +#ifndef ODP_CONFIG_LLQ_LOCK<br> > > +<br> > > +/* If 'node' is a head of llq then move it to tail */<br> > > +static inline odp_bool_t llq_cond_rotate(struct llqueue *llq,<br> > > + struct llnode *node)<br> > > +{<br> > > + /* Difficult to make this into a single atomic operation<br> > > + * Instead use existing primitives.<br> > > + */<br> > > + if (odp_likely(llq_dequeue_cond(llq, node))) {<br> > > + llq_enqueue(llq, node);<br> > > + return true;<br> > > + }<br> > > + return false;<br> > > +}<br> > > +<br> > > +#else<br> > > +<br> > > +/* If 'node' is a head of llq then move it to tail */<br> > > +static inline odp_bool_t llq_cond_rotate(struct llqueue *llq,<br> > > + struct llnode *node)<br> > > +{<br> > > + odp_bool_t success = false;<br> > > +<br> > > + odp_spinlock_lock(&llq->lock);<br> > > + if (odp_likely(llq->u.st.head == node)) {<br> > > + success = true;<br> > > + if (llq->u.st.tail != node) {<br> > > + ODP_ASSERT(node->next != SENTINEL);<br> > > + llq->u.st.head = node->next;<br> > > + llq->u.st.tail->next = node;<br> > > + llq->u.st.tail = node;<br> > > + node->next = SENTINEL;<br> > > + }<br> > > + /* Else 'node' is only element on list => nothing to do */<br> > > + }<br> > > + odp_spinlock_unlock(&llq->lock);<br> > > + return success;<br> > > +}<br> > > +<br> > > +#endif<br> > > +<br> > > +#endif<br> > > diff --git a/platform/linux-generic/include/odp_llsc.h b/platform/linux-generic/include/odp_llsc.h<br> > > new file mode 100644<br> > > index 00000000..a7b6166e<br> > > --- /dev/null<br> > > +++ b/platform/linux-generic/include/odp_llsc.h<br> > > @@ -0,0 +1,325 @@<br> > > +/* Copyright (c) 2017, ARM Limited<br> > > + * All rights reserved.<br> > > + *<br> > > + * SPDX-License-Identifier: BSD-3-Clause<br> > > + */<br> > > +<br> > > +#ifndef ODP_LLSC_H_<br> > > +#define ODP_LLSC_H_<br> > > +<br> > > +#include <odp_config_internal.h><br> > > +<br> > > +/******************************************************************************<br> > > + * LL/SC primitives<br> > > + *****************************************************************************/<br> > > +<br> > > +#if defined __ARM_ARCH<br> > > +#if __ARM_ARCH == 7 || (__ARM_ARCH == 8 && __ARM_64BIT_STATE == 0)<br> > > +static inline void dmb(void)<br> > > +{<br> > > + __asm volatile("dmb" : : : "memory");<br> > > +}<br> > > +<br> > > +static inline uint32_t ll8(uint8_t *var, int mm)<br> > > +{<br> > > + uint8_t old;<br> > > +<br> > > + __asm volatile("ldrexb %0, [%1]"<br> > > + : "=&r" (old)<br> > > + : "r" (var)<br> > > + : );<br> > > + /* Barrier after an acquiring load */<br> > > + if (mm == __ATOMIC_ACQUIRE)<br> > > + dmb();<br> > > + return old;<br> > > +}<br> > > +<br> > > +static inline uint32_t ll16(uint16_t *var, int mm)<br> > > +{<br> > > + uint16_t old;<br> > > +<br> > > + __asm volatile("ldrexh %0, [%1]"<br> > > + : "=&r" (old)<br> > > + : "r" (var)<br> > > + : );<br> > > + /* Barrier after an acquiring load */<br> > > + if (mm == __ATOMIC_ACQUIRE)<br> > > + dmb();<br> > > + return old;<br> > > +}<br> > > +<br> > > +static inline uint32_t ll(uint32_t *var, int mm)<br> > > +{<br> > > + uint32_t old;<br> > > +<br> > > + __asm volatile("ldrex %0, [%1]"<br> > > + : "=&r" (old)<br> > > + : "r" (var)<br> > > + : );<br> > > + /* Barrier after an acquiring load */<br> > > + if (mm == __ATOMIC_ACQUIRE)<br> > > + dmb();<br> > > + return old;<br> > > +}<br> > > +#define ll32(a, b) ll((a), (b))<br> > > +<br> > > +/* Return 0 on success, 1 on failure */<br> > > +static inline uint32_t sc(uint32_t *var, uint32_t neu, int mm)<br> > > +{<br> > > + uint32_t ret;<br> > > +<br> > > + /* Barrier before a releasing store */<br> > > + if (mm == __ATOMIC_RELEASE)<br> > > + dmb();<br> > > + __asm volatile("strex %0, %1, [%2]"<br> > > + : "=&r" (ret)<br> > > + : "r" (neu), "r" (var)<br> > > + : );<br> > > + return ret;<br> > > +}<br> > > +#define sc32(a, b, c) sc((a), (b), (c))<br> > > +<br> > > +static inline uint64_t lld(uint64_t *var, int mm)<br> > > +{<br> > > + uint64_t old;<br> > > +<br> > > + __asm volatile("ldrexd %0, %H0, [%1]"<br> > > + : "=&r" (old)<br> > > + : "r" (var)<br> > > + : );<br> > > + /* Barrier after an acquiring load */<br> > > + if (mm == __ATOMIC_ACQUIRE)<br> > > + dmb();<br> > > + return old;<br> > > +}<br> > > +#define ll64(a, b) lld((a), (b))<br> > > +<br> > > +/* Return 0 on success, 1 on failure */<br> > > +static inline uint32_t scd(uint64_t *var, uint64_t neu, int mm)<br> > > +{<br> > > + uint32_t ret;<br> > > +<br> > > + /* Barrier before a releasing store */<br> > > + if (mm == __ATOMIC_RELEASE)<br> > > + dmb();<br> > > + __asm volatile("strexd %0, %1, %H1, [%2]"<br> > > + : "=&r" (ret)<br> > > + : "r" (neu), "r" (var)<br> > > + : );<br> > > + return ret;<br> > > +}<br> > > +#define sc64(a, b, c) scd((a), (b), (c))<br> > > +<br> > > +#endif<br> > > +<br> > > +#if __ARM_ARCH == 8 && __ARM_64BIT_STATE == 1<br> > > +static inline uint16_t ll8(uint8_t *var, int mm)<br> > > +{<br> > > + uint16_t old;<br> > > +<br> > > + if (mm == __ATOMIC_ACQUIRE)<br> > > + __asm volatile("ldaxrb %w0, [%1]"<br> > > + : "=&r" (old)<br> > > + : "r" (var)<br> > > + : "memory");<br> > > + else if (mm == __ATOMIC_RELAXED)<br> > > + __asm volatile("ldxrb %w0, [%1]"<br> > > + : "=&r" (old)<br> > > + : "r" (var)<br> > > + : );<br> > > + else<br> > > + ODP_ABORT();<br> > > + return old;<br> > > +}<br> > > +<br> > > +static inline uint16_t ll16(uint16_t *var, int mm)<br> > > +{<br> > > + uint16_t old;<br> > > +<br> > > + if (mm == __ATOMIC_ACQUIRE)<br> > > + __asm volatile("ldaxrh %w0, [%1]"<br> > > + : "=&r" (old)<br> > > + : "r" (var)<br> > > + : "memory");<br> > > + else if (mm == __ATOMIC_RELAXED)<br> > > + __asm volatile("ldxrh %w0, [%1]"<br> > > + : "=&r" (old)<br> > > + : "r" (var)<br> > > + : );<br> > > + else<br> > > + ODP_ABORT();<br> > > + return old;<br> > > +}<br> > > +<br> > > +static inline uint32_t ll32(uint32_t *var, int mm)<br> > > +{<br> > > + uint32_t old;<br> > > +<br> > > + if (mm == __ATOMIC_ACQUIRE)<br> > > + __asm volatile("ldaxr %w0, [%1]"<br> > > + : "=&r" (old)<br> > > + : "r" (var)<br> > > + : "memory");<br> > > + else if (mm == __ATOMIC_RELAXED)<br> > > + __asm volatile("ldxr %w0, [%1]"<br> > > + : "=&r" (old)<br> > > + : "r" (var)<br> > > + : );<br> > > + else<br> > > + ODP_ABORT();<br> > > + return old;<br> > > +}<br> > > +<br> > > +/* Return 0 on success, 1 on failure */<br> > > +static inline uint32_t sc32(uint32_t *var, uint32_t neu, int mm)<br> > > +{<br> > > + uint32_t ret;<br> > > +<br> > > + if (mm == __ATOMIC_RELEASE)<br> > > + __asm volatile("stlxr %w0, %w1, [%2]"<br> > > + : "=&r" (ret)<br> > > + : "r" (neu), "r" (var)<br> > > + : "memory");<br> > > + else if (mm == __ATOMIC_RELAXED)<br> > > + __asm volatile("stxr %w0, %w1, [%2]"<br> > > + : "=&r" (ret)<br> > > + : "r" (neu), "r" (var)<br> > > + : );<br> > > + else<br> > > + ODP_ABORT();<br> > > + return ret;<br> > > +}<br> > > +<br> > > +static inline uint64_t ll(uint64_t *var, int mm)<br> > > +{<br> > > + uint64_t old;<br> > > +<br> > > + if (mm == __ATOMIC_ACQUIRE)<br> > > + __asm volatile("ldaxr %0, [%1]"<br> > > + : "=&r" (old)<br> > > + : "r" (var)<br> > > + : "memory");<br> > > + else if (mm == __ATOMIC_RELAXED)<br> > > + __asm volatile("ldxr %0, [%1]"<br> > > + : "=&r" (old)<br> > > + : "r" (var)<br> > > + : );<br> > > + else<br> > > + ODP_ABORT();<br> > > + return old;<br> > > +}<br> > > +#define ll64(a, b) ll((a), (b))<br> > > +<br> > > +/* Return 0 on success, 1 on failure */<br> > > +static inline uint32_t sc(uint64_t *var, uint64_t neu, int mm)<br> > > +{<br> > > + uint32_t ret;<br> > > +<br> > > + if (mm == __ATOMIC_RELEASE)<br> > > + __asm volatile("stlxr %w0, %1, [%2]"<br> > > + : "=&r" (ret)<br> > > + : "r" (neu), "r" (var)<br> > > + : "memory");<br> > > + else if (mm == __ATOMIC_RELAXED)<br> > > + __asm volatile("stxr %w0, %1, [%2]"<br> > > + : "=&r" (ret)<br> > > + : "r" (neu), "r" (var)<br> > > + : );<br> > > + else<br> > > + ODP_ABORT();<br> > > + return ret;<br> > > +}<br> > > +#define sc64(a, b, c) sc((a), (b), (c))<br> > > +static inline __int128 lld(__int128 *var, int mm)<br> > > +{<br> > > + __int128 old;<br> > > +<br> > > + if (mm == __ATOMIC_ACQUIRE)<br> > > + __asm volatile("ldaxp %0, %H0, [%1]"<br> > > + : "=&r" (old)<br> > > + : "r" (var)<br> > > + : "memory");<br> > > + else if (mm == __ATOMIC_RELAXED)<br> > > + __asm volatile("ldxp %0, %H0, [%1]"<br> > > + : "=&r" (old)<br> > > + : "r" (var)<br> > > + : );<br> > > + else<br> > > + ODP_ABORT();<br> > > + return old;<br> > > +}<br> > > +<br> > > +/* Return 0 on success, 1 on failure */<br> > > +static inline uint32_t scd(__int128 *var, __int128 neu, int mm)<br> > > +{<br> > > + uint32_t ret;<br> > > +<br> > > + if (mm == __ATOMIC_RELEASE)<br> > > + __asm volatile("stlxp %w0, %1, %H1, [%2]"<br> > > + : "=&r" (ret)<br> > > + : "r" (neu), "r" (var)<br> > > + : "memory");<br> > > + else if (mm == __ATOMIC_RELAXED)<br> > > + __asm volatile("stxp %w0, %1, %H1, [%2]"<br> > > + : "=&r" (ret)<br> > > + : "r" (neu), "r" (var)<br> > > + : );<br> > > + else<br> > > + ODP_ABORT();<br> > > + return ret;<br> > > +}<br> > > +#endif<br> > > +#endif<br> > > +<br> > > +static inline void sevl(void)<br> > > +{<br> > > +#if defined __ARM_ARCH<br> > > + __asm volatile("sevl" : : : );<br> > > +#endif<br> > > +}<br> > > +<br> > > +static inline void sev(void)<br> > > +{<br> > > +#if defined __ARM_ARCH<br> > > + __asm volatile("sev" : : : "memory");<br> > > +#endif<br> > > +}<br> > > +<br> > > +static inline int wfe(void)<br> > > +{<br> > > +#if defined __ARM_ARCH<br> > > + __asm volatile("wfe" : : : "memory");<br> > > +#endif<br> > > + return 1;<br> > > +}<br> > > +<br> > > +#ifdef ODP_CONFIG_USE_WFE<br> > > +#define SEVL() sevl()<br> > > +#define WFE() wfe()<br> > > +#define SEV() do { __asm volatile("dsb ish" ::: "memory"); sev(); } while (0)<br> > > +#if defined __ARM_ARCH && __ARM_ARCH == 8 && __ARM_64BIT_STATE == 1<br> > > +#define LDXR128(addr, mo) lld((addr), (mo))<br> > > +#endif<br> > > +#define LDXR64(addr, mo) ll64((addr), (mo))<br> > > +#define LDXR32(addr, mo) ll32((addr), (mo))<br> > > +#define LDXR16(addr, mo) ll16((addr), (mo))<br> > > +#define LDXR8(addr, mo) ll8((addr), (mo))<br> > > +/* When using WFE do not stall the pipeline using other means */<br> > > +#define DOZE() (void)0<br> > > +#else<br> > > +#define SEVL() (void)0<br> > > +#define WFE() 1<br> > > +#define SEV() (void)0<br> > > +#define LDXR128(addr, mo) __atomic_load_n((addr), (mo))<br> > > +#define LDXR64(addr, mo) __atomic_load_n((addr), (mo))<br> > > +#define LDXR32(addr, mo) __atomic_load_n((addr), (mo))<br> > > +#define LDXR16(addr, mo) __atomic_load_n((addr), (mo))<br> > > +#define LDXR8(addr, mo) __atomic_load_n((addr), (mo))<br> > > +#if defined __ARM_ARCH<br> > > +#define DOZE() __asm volatile("isb" ::: "memory")<br> > > +#else<br> > > +#define DOZE() __asm volatile("pause" ::: "memory")<br> > > +#endif<br> > > +#endif<br> > > +<br> > > +#endif<br> > > diff --git a/platform/linux-generic/include/odp_packet_internal.h b/platform/linux-generic/include/odp_packet_internal.h<br> > > index 0a9f1779..c9d14766 100644<br> > > --- a/platform/linux-generic/include/odp_packet_internal.h<br> > > +++ b/platform/linux-generic/include/odp_packet_internal.h<br> > > @@ -235,6 +235,9 @@ odp_buffer_t _odp_packet_to_buffer(odp_packet_t pkt);<br> > > /* Convert a buffer handle to a packet handle */<br> > > odp_packet_t _odp_packet_from_buffer(odp_buffer_t buf);<br> > ><br> > > +/* Convert a packet handle to a buffer hdr handle pointer */<br> > > +odp_buffer_hdr_t *_odp_packet_to_buf_hdr_ptr(odp_packet_t pkt);<br> > > +<br> > > static inline int packet_hdr_has_l2(odp_packet_hdr_t *pkt_hdr)<br> > > {<br> > > return pkt_hdr->p.input_flags.l2;<br> > > diff --git a/platform/linux-generic/include/odp_queue_internal.h b/platform/linux-generic/include/odp_queue_internal.h<br> > > index 560f826e..ebfc0f76 100644<br> > > --- a/platform/linux-generic/include/odp_queue_internal.h<br> > > +++ b/platform/linux-generic/include/odp_queue_internal.h<br> > > @@ -19,16 +19,24 @@ extern "C" {<br> > > #endif<br> > ><br> > > #include <odp/api/queue.h><br> > > -#include <odp_forward_typedefs_internal.h><br> > > -#include <odp_schedule_if.h><br> > > -#include <odp_buffer_internal.h><br> > > -#include <odp_align_internal.h><br> > > +#include <odp/api/std_types.h><br> > > +#include <odp/api/buffer.h><br> > > #include <odp/api/packet_io.h><br> > > #include <odp/api/align.h><br> > > #include <odp/api/hints.h><br> > > #include <odp/api/ticketlock.h><br> > > +<br> > > #include <odp_config_internal.h><br> > ><br> > > +#include <odp_align_internal.h><br> > > +#include <odp_buffer_internal.h><br> > > +#include <odp_forward_typedefs_internal.h><br> > > +#ifdef ODP_SCHEDULE_SCALABLE<br> > > +#include <odp_llsc.h><br> > > +#include <odp_schedule_ordered_internal.h><br> > > +#endif<br> > > +#include <odp_schedule_if.h><br> > > +<br> > > #define QUEUE_MULTI_MAX CONFIG_BURST_SIZE<br> > ><br> > > #define QUEUE_STATUS_FREE 0<br> > > @@ -37,8 +45,6 @@ extern "C" {<br> > > #define QUEUE_STATUS_NOTSCHED 3<br> > > #define QUEUE_STATUS_SCHED 4<br> > ><br> > > -<br> > > -/* forward declaration */<br> > > union queue_entry_u;<br> > ><br> > > typedef int (*enq_func_t)(union queue_entry_u *, odp_buffer_hdr_t *);<br> > > @@ -49,6 +55,37 @@ typedef int (*enq_multi_func_t)(union queue_entry_u *,<br> > > typedef int (*deq_multi_func_t)(union queue_entry_u *,<br> > > odp_buffer_hdr_t **, int);<br> > ><br> > > +#ifdef ODP_SCHEDULE_SCALABLE<br> > > +#define BUFFER_HDR_INVALID ((odp_buffer_hdr_t *)ODP_EVENT_INVALID)<br> > > +<br> > > +struct queue_entry_s {<br> > > + sched_elem_t sched_elem;<br> > > + odp_shm_t shm;<br> > > + odp_shm_t rwin_shm;<br> > > +<br> > > + odp_ticketlock_t lock ODP_ALIGNED_CACHE;<br> > > + int status;<br> > > +<br> > > + enq_func_t enqueue ODP_ALIGNED_CACHE;<br> > > + deq_func_t dequeue;<br> > > + enq_multi_func_t enqueue_multi;<br> > > + deq_multi_func_t dequeue_multi;<br> > > +<br> > > + uint32_t index;<br> > > + odp_queue_t handle;<br> > > + odp_queue_type_t type;<br> > > + odp_queue_param_t param;<br> > > + odp_pktin_queue_t pktin;<br> > > + odp_pktout_queue_t pktout;<br> > > + char name[ODP_QUEUE_NAME_LEN];<br> > > +};<br> > > +<br> > > +int _odp_queue_deq(sched_elem_t *q, odp_event_t *evp, int num);<br> > > +int _odp_queue_deq_sc(sched_elem_t *q, odp_event_t *evp, int num);<br> > > +<br> > > +#else<br> > > +#define BUFFER_HDR_INVALID NULL<br> > > +<br> > > struct queue_entry_s {<br> > > odp_ticketlock_t lock ODP_ALIGNED_CACHE;<br> > ><br> > > @@ -77,6 +114,8 @@ struct queue_entry_s {<br> > > char name[ODP_QUEUE_NAME_LEN];<br> > > };<br> > ><br> > > +#endif<br> > > +<br> > > union queue_entry_u {<br> > > struct queue_entry_s s;<br> > > uint8_t pad[ROUNDUP_CACHE_LINE(sizeof(struct queue_entry_s))];<br> > > @@ -94,6 +133,12 @@ int queue_deq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[], int num);<br> > > void queue_lock(queue_entry_t *queue);<br> > > void queue_unlock(queue_entry_t *queue);<br> > ><br> > > +int queue_pktout_enq(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr);<br> > > +int queue_pktout_enq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[],<br> > > + int num);<br> > > +<br> > > +int queue_tm_reorder(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr);<br> > > +<br> > > static inline uint32_t queue_to_id(odp_queue_t handle)<br> > > {<br> > > return _odp_typeval(handle) - 1;<br> > > @@ -107,6 +152,71 @@ static inline queue_entry_t *queue_to_qentry(odp_queue_t handle)<br> > > return get_qentry(queue_id);<br> > > }<br> > ><br> > > +static inline odp_queue_t queue_get_handle(queue_entry_t *queue)<br> > > +{<br> > > + return queue->s.handle;<br> > > +}<br> > > +<br> > > +static inline odp_pktout_queue_t queue_get_pktout(queue_entry_t *queue)<br> > > +{<br> > > + return queue->s.pktout;<br> > > +}<br> > > +<br> > > +static inline void queue_set_pktout(queue_entry_t *queue,<br> > > + odp_pktio_t pktio,<br> > > + int index)<br> > > +{<br> > > + queue->s.pktout.pktio = pktio;<br> > > + queue->s.pktout.index = index;<br> > > +}<br> > > +<br> > > +static inline odp_pktin_queue_t queue_get_pktin(queue_entry_t *queue)<br> > > +{<br> > > + return queue->s.pktin;<br> > > +}<br> > > +<br> > > +static inline void queue_set_pktin(queue_entry_t *queue,<br> > > + odp_pktio_t pktio,<br> > > + int index)<br> > > +{<br> > > + queue->s.pktin.pktio = pktio;<br> > > + queue->s.pktin.index = index;<br> > > +}<br> > > +<br> > > +static inline void queue_set_enq_func(queue_entry_t *queue, enq_func_t func)<br> > > +{<br> > > + queue->s.enqueue = func;<br> > > +}<br> > > +<br> > > +static inline void queue_set_enq_multi_func(queue_entry_t *queue,<br> > > + enq_multi_func_t func)<br> > > +{<br> > > + queue->s.enqueue_multi = func;<br> > > +}<br> > > +<br> > > +static inline void queue_set_deq_func(queue_entry_t *queue, deq_func_t func)<br> > > +{<br> > > + queue->s.dequeue = func;<br> > > +}<br> > > +<br> > > +static inline void queue_set_deq_multi_func(queue_entry_t *queue,<br> > > + deq_multi_func_t func)<br> > > +{<br> > > + queue->s.dequeue_multi = func;<br> > > +}<br> > > +<br> > > +static inline void queue_set_type(queue_entry_t *queue, odp_queue_type_t type)<br> > > +{<br> > > + queue->s.type = type;<br> > > +}<br> > > +<br> > > +#ifdef ODP_SCHEDULE_SCALABLE<br> > > +static inline reorder_window_t *queue_get_rwin(queue_entry_t *queue)<br> > > +{<br> > > + return queue->s.sched_elem.rwin;<br> > > +}<br> > > +#endif<br> > > +<br> > > #ifdef __cplusplus<br> > > }<br> > > #endif<br> > > diff --git a/platform/linux-generic/include/odp_schedule_if.h b/platform/linux-generic/include/odp_schedule_if.h<br> > > index 530d157f..41998833 100644<br> > > --- a/platform/linux-generic/include/odp_schedule_if.h<br> > > +++ b/platform/linux-generic/include/odp_schedule_if.h<br> > > @@ -4,6 +4,12 @@<br> > > * SPDX-License-Identifier: BSD-3-Clause<br> > > */<br> > ><br> > > +/* Copyright (c) 2017, ARM Limited<br> > > + * All rights reserved.<br> > > + *<br> > > + * SPDX-License-Identifier: BSD-3-Clause<br> > > + */<br> > > +<br> > > #ifndef ODP_SCHEDULE_IF_H_<br> > > #define ODP_SCHEDULE_IF_H_<br> > ><br> > > @@ -11,18 +17,168 @@<br> > > extern "C" {<br> > > #endif<br> > ><br> > > +#include <odp/api/align.h><br> > > #include <odp/api/queue.h><br> > > -#include <odp_queue_internal.h><br> > > #include <odp/api/schedule.h><br> > > +#include <odp/api/ticketlock.h><br> > > +<br> > > +#include <odp_forward_typedefs_internal.h><br> > > +#ifdef ODP_SCHEDULE_SCALABLE<br> > > +#include <odp_config_internal.h><br> > > +#include <odp_llqueue.h><br> > > +#include <odp_schedule_ordered_internal.h><br> > > +<br> > > +#include <limits.h><br> > > +#endif<br> > > +<br> > > +/* Number of ordered locks per queue */<br> > > +#define SCHEDULE_ORDERED_LOCKS_PER_QUEUE 2<br> > > +<br> > > +#ifdef ODP_SCHEDULE_SCALABLE<br> > > +<br> > > +typedef struct {<br> > > + union {<br> > > + struct {<br> > > + struct llqueue llq;<br> > > + uint32_t prio;<br> > > + };<br> > > + char line[ODP_CACHE_LINE_SIZE];<br> > > + };<br> > > +} sched_queue_t ODP_ALIGNED_CACHE;<br> > > +<br> > > +#define TICKET_INVALID (uint16_t)(~0U)<br> > > +<br> > > +typedef struct {<br> > > + int32_t numevts;<br> > > + uint16_t wrr_budget;<br> > > + uint8_t cur_ticket;<br> > > + uint8_t nxt_ticket;<br> > > +} qschedstate_t ODP_ALIGNED(sizeof(uint64_t));<br> > > +<br> > > +typedef uint32_t ringidx_t;<br> > > +<br> > > +#ifdef ODP_CONFIG_USE_SPLIT_PRODCONS<br> > > +#define SPLIT_PC ODP_ALIGNED_CACHE<br> > > +#else<br> > > +#define SPLIT_PC<br> > > +#endif<br> > > +<br> > > +#ifdef ODP_CONFIG_USE_SPLIT_READWRITE<br> > > +#define SPLIT_RW ODP_ALIGNED_CACHE<br> > > +#else<br> > > +#define SPLIT_RW<br> > > +#endif<br> > > +<br> > > +#define ODP_NO_SCHED_QUEUE (ODP_SCHED_SYNC_ORDERED + 1)<br> > > +<br> > > +typedef struct {<br> > > + /* (ll)node must be first */<br> > > + struct llnode node; /* 8: 0.. 7 */<br> > > + sched_queue_t *schedq; /* 8: 8..15 */<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + odp_ticketlock_t qschlock;<br> > > +#endif<br> > > + qschedstate_t qschst; /* 8: 16..23 */<br> > > + uint16_t pop_deficit; /* 2: 24..25 */<br> > > + uint16_t qschst_type; /* 2: 26..27 */<br> > > +<br> > > + ringidx_t prod_read SPLIT_PC; /* 4: 28..31 */<br> > > +<br> > > + ringidx_t prod_write SPLIT_RW;/* 4: 32..35 */<br> > > + ringidx_t prod_mask; /* 4: 36..39 */<br> > > + odp_buffer_hdr_t **prod_ring; /* 8: 40..47 */<br> > > +<br> > > + ringidx_t cons_write SPLIT_PC;/* 4: 48..51 */<br> > > +<br> > > + ringidx_t cons_read SPLIT_RW; /* 4: 52..55 */<br> > > +<br> > > + reorder_window_t *rwin; /* 8: 56..63 */<br> > > + void *user_ctx; /* 8: 64..71 */<br> > > +#ifdef ODP_CONFIG_USE_SPLIT_PRODCONS<br> > > + odp_buffer_hdr_t **cons_ring;<br> > > + ringidx_t cons_mask;<br> > > + uint16_t cons_type;<br> > > +#else<br> > > +#define cons_mask prod_mask<br> > > +#define cons_ring prod_ring<br> > > +#define cons_type qschst_type<br> > > +#endif<br> > > +} sched_elem_t ODP_ALIGNED_CACHE;<br> > > +<br> > > +/*<br> > > + * Scheduler group related declarations.<br> > > + */<br> > > +typedef bitset_t sched_group_mask_t;<br> > > +<br> > > +/* Number of scheduling groups */<br> > > +#define MAX_SCHED_GROUP (sizeof(sched_group_mask_t) * CHAR_BIT)<br> > > +<br> > > +typedef struct {<br> > > + /* Threads currently associated with the sched group */<br> > > + bitset_t thr_actual[ODP_SCHED_PRIO_NUM] ODP_ALIGNED_CACHE;<br> > > + bitset_t thr_wanted;<br> > > + /* Used to spread queues over schedq's */<br> > > + uint32_t xcount[ODP_SCHED_PRIO_NUM];<br> > > + /* Number of schedq's per prio */<br> > > + uint32_t xfactor;<br> > > +<br> > > + char name[ODP_SCHED_GROUP_NAME_LEN];<br> > > + /* shm handle for the group */<br> > > + odp_shm_t shm;<br> > > +<br> > > + /* ODP_SCHED_PRIO_NUM * xfactor.<br> > > + * Must be the last memember in this struct.<br> > > + */<br> > > + sched_queue_t schedq[1] ODP_ALIGNED_CACHE;<br> > > +} sched_group_t;<br> > > +<br> > > +/*<br> > > + * Per thread state<br> > > + */<br> > > +/* Number of reorder contexts per thread */<br> > > +#define TS_RVEC_SIZE 16<br> > > +typedef struct {<br> > > + /* Atomic queue currently being processed or NULL */<br> > > + sched_elem_t *atomq;<br> > > + /* Current reorder context or NULL */<br> > > + reorder_context_t *rctx;<br> > > + uint8_t pause;<br> > > + uint8_t out_of_order;<br> > > + uint8_t tidx;<br> > > + uint8_t pad;<br> > > + uint32_t dequeued; /* Number of events dequeued from atomic queue */<br> > > + uint16_t pktin_next;/* Next pktin tag to poll */<br> > > + uint16_t pktin_poll_cnts;<br> > > + uint16_t ticket; /* Ticket for atomic queue or TICKET_INVALID */<br> > > + uint16_t num_schedq;<br> > > + uint16_t sg_sem; /* Set when sg_wanted is modified by other thread */<br> > > +#define SCHEDQ_PER_THREAD (MAX_SCHED_GROUP * ODP_SCHED_PRIO_NUM)<br> > > + sched_queue_t *schedq_list[SCHEDQ_PER_THREAD];<br> > > + /* Current sched_group membership */<br> > > + sched_group_mask_t sg_actual[ODP_SCHED_PRIO_NUM];<br> > > + /* Future sched_group membership. */<br> > > + sched_group_mask_t sg_wanted[ODP_SCHED_PRIO_NUM];<br> > > + bitset_t priv_rvec_free;<br> > > + /* Bitset of free entries in rvec[] */<br> > > + bitset_t rvec_free ODP_ALIGNED_CACHE;<br> > > + /* Reordering contexts to allocate from */<br> > > + reorder_context_t rvec[TS_RVEC_SIZE] ODP_ALIGNED_CACHE;<br> > > +} sched_scalable_thread_state_t ODP_ALIGNED_CACHE;<br> > > +<br> > > +void sched_update_enq(sched_elem_t *q, uint32_t actual);<br> > > +void sched_update_enq_sp(sched_elem_t *q, uint32_t actual);<br> > > +sched_queue_t *schedq_from_sched_group(odp_schedule_group_t grp, uint32_t prio);<br> > > +void sched_group_xcount_dec(odp_schedule_group_t grp, uint32_t prio);<br> > > +<br> > > +#endif /* ODP_SCHEDULE_SCALABLE */<br> > ><br> > > typedef void (*schedule_pktio_start_fn_t)(int pktio_index, int num_in_queue,<br> > > int in_queue_idx[]);<br> > > typedef int (*schedule_thr_add_fn_t)(odp_schedule_group_t group, int thr);<br> > > typedef int (*schedule_thr_rem_fn_t)(odp_schedule_group_t group, int thr);<br> > > typedef int (*schedule_num_grps_fn_t)(void);<br> > > -typedef int (*schedule_init_queue_fn_t)(uint32_t queue_index,<br> > > - const odp_schedule_param_t *sched_param<br> > > - );<br> > > +typedef int (*schedule_init_queue_fn_t)(<br> > > + uint32_t queue_index, const odp_schedule_param_t *sched_param);<br> > > typedef void (*schedule_destroy_queue_fn_t)(uint32_t queue_index);<br> > > typedef int (*schedule_sched_queue_fn_t)(uint32_t queue_index);<br> > > typedef int (*schedule_unsched_queue_fn_t)(uint32_t queue_index);<br> > > @@ -64,6 +220,7 @@ extern const schedule_fn_t *sched_fn;<br> > > int sched_cb_pktin_poll(int pktio_index, int num_queue, int index[]);<br> > > void sched_cb_pktio_stop_finalize(int pktio_index);<br> > > int sched_cb_num_pktio(void);<br> > > +#ifndef ODP_SCHEDULE_SCALABLE<br> > > int sched_cb_num_queues(void);<br> > > int sched_cb_queue_prio(uint32_t queue_index);<br> > > int sched_cb_queue_grp(uint32_t queue_index);<br> > > @@ -73,6 +230,7 @@ odp_queue_t sched_cb_queue_handle(uint32_t queue_index);<br> > > void sched_cb_queue_destroy_finalize(uint32_t queue_index);<br> > > int sched_cb_queue_deq_multi(uint32_t queue_index, odp_event_t ev[], int num);<br> > > int sched_cb_queue_empty(uint32_t queue_index);<br> > > +#endif<br> > ><br> > > /* API functions */<br> > > typedef struct {<br> > > diff --git a/platform/linux-generic/include/odp_schedule_ordered_internal.h b/platform/linux-generic/include/odp_schedule_ordered_internal.h<br> > > new file mode 100644<br> > > index 00000000..a16deca8<br> > > --- /dev/null<br> > > +++ b/platform/linux-generic/include/odp_schedule_ordered_internal.h<br> > > @@ -0,0 +1,150 @@<br> > > +/* Copyright (c) 2017, ARM Limited<br> > > + * All rights reserved.<br> > > + *<br> > > + * SPDX-License-Identifier: BSD-3-Clause<br> > > + */<br> > > +<br> > > +#ifndef ODP_SCHEDULE_ORDERED_INTERNAL_H_<br> > > +#define ODP_SCHEDULE_ORDERED_INTERNAL_H_<br> > > +<br> > > +#ifdef __cplusplus<br> > > +extern "C" {<br> > > +#endif<br> > > +<br> > > +#ifdef ODP_SCHEDULE_SCALABLE<br> > > +<br> > > +#include <odp/api/shared_memory.h><br> > > +<br> > > +#include <odp_internal.h><br> > > +<br> > > +#include <odp_align_internal.h><br> > > +#include <odp_bitset.h><br> > > +#include <odp_llsc.h><br> > > +<br> > > +/* High level functioning of reordering<br> > > + * Datastructures -<br> > > + * Reorder Window - Every ordered queue is associated with a reorder window.<br> > > + * Reorder window stores reorder contexts from threads that<br> > > + * have completed processing out-of-order.<br> > > + * Reorder Context - Reorder context consists of events that a thread<br> > > + * wants to enqueue while processing a batch of events<br> > > + * from an ordered queue.<br> > > + *<br> > > + * Algorithm -<br> > > + * 1) Thread identifies the ordered queue.<br> > > + * 2) It 'reserves a slot in the reorder window and dequeues the<br> > > + * events' atomically. Atomicity is achieved by using a ticket-lock<br> > > + * like design where the reorder window slot is the ticket.<br> > > + * 3a) Upon order-release/next schedule call, the thread<br> > > + * checks if it's slot (ticket) equals the head of the reorder window.<br> > > + * If yes, enqueues the events to the destination queue till<br> > > + * i) the reorder window is empty or<br> > > + * ii) there is a gap in the reorder window<br> > > + * If no, the reorder context is stored in the reorder window at<br> > > + * the reserved slot.<br> > > + * 3b) Upon the first enqueue, the thread checks if it's slot (ticket)<br> > > + * equals the head of the reorder window.<br> > > + * If yes, enqueues the events immediately to the destination queue<br> > > + * If no, these (and subsequent) events are stored in the reorder context<br> > > + * (in the application given order)<br> > > + */<br> > > +<br> > > +/* Head and change indicator variables are used to synchronise between<br> > > + * concurrent insert operations in the reorder window. A thread performing<br> > > + * an in-order insertion must be notified about the newly inserted<br> > > + * reorder contexts so that it doesn’t halt the retire process too early.<br> > > + * A thread performing an out-of-order insertion must correspondingly<br> > > + * notify the thread doing in-order insertion of the new waiting reorder<br> > > + * context, which may need to be handled by that thread.<br> > > + *<br> > > + * Also, an out-of-order insertion may become an in-order insertion if the<br> > > + * thread doing an in-order insertion completes before this thread completes.<br> > > + * We need a point of synchronisation where this knowledge and potential state<br> > > + * change can be transferred between threads.<br> > > + */<br> > > +typedef struct hc {<br> > > + /* First missing context */<br> > > + uint32_t head;<br> > > + /* Change indicator */<br> > > + uint32_t chgi;<br> > > +} hc_t ODP_ALIGNED(sizeof(uint64_t));<br> > > +<br> > > +/* Number of reorder contects in the reorder window.<br> > > + * Should be at least one per CPU.<br> > > + */<br> > > +#define RWIN_SIZE 32<br> > > +ODP_STATIC_ASSERT(IS_POWER_TWO(RWIN_SIZE), "RWIN_SIZE is not a power of 2");<br> > > +<br> > > +#define NUM_OLOCKS 2<br> > > +<br> > > +typedef struct reorder_context reorder_context_t;<br> > > +<br> > > +typedef struct reorder_window {<br> > > + /* head and change indicator */<br> > > + hc_t hc;<br> > > + uint32_t winmask;<br> > > + uint32_t tail;<br> > > + uint32_t turn;<br> > > + uint32_t olock[NUM_OLOCKS];<br> > > + uint16_t lock_count;<br> > > + /* Reorder contexts in this window */<br> > > + reorder_context_t *ring[RWIN_SIZE];<br> > > +} reorder_window_t;<br> > > +<br> > > +/* Number of events that can be stored in a reorder context.<br> > > + * This size is chosen so that there is no space left unused at the end<br> > > + * of the last cache line (for 64b architectures and 64b handles).<br> > > + */<br> > > +#define RC_EVT_SIZE 18<br> > > +<br> > > +typedef struct reorder_context {<br> > > + /* Reorder window to which this context belongs */<br> > > + reorder_window_t *rwin;<br> > > + /* Pointer to TS->rvec_free */<br> > > + bitset_t *rvec_free;<br> > > + /* Our slot number in the reorder window */<br> > > + uint32_t sn;<br> > > + uint8_t olock_flags;<br> > > + /* Our index in thread_state rvec array */<br> > > + uint8_t idx;<br> > > + /* Use to link reorder contexts together */<br> > > + uint8_t next_idx;<br> > > + /* Current reorder context to save events in */<br> > > + uint8_t cur_idx;<br> > > + /* Number of events stored in this reorder context */<br> > > + uint8_t numevts;<br> > > + /* Events stored in this context */<br> > > + odp_buffer_hdr_t *events[RC_EVT_SIZE];<br> > > + queue_entry_t *destq[RC_EVT_SIZE];<br> > > +} reorder_context_t ODP_ALIGNED_CACHE;<br> > > +<br> > > +reorder_window_t *rwin_alloc(int rwin_id, odp_shm_t *shm, unsigned lock_count);<br> > > +bool rwin_reserve(reorder_window_t *rwin, uint32_t *sn);<br> > > +void rwin_insert(reorder_window_t *rwin,<br> > > + reorder_context_t *rctx,<br> > > + uint32_t sn,<br> > > + void (*callback)(reorder_context_t *));<br> > > +void rctx_init(reorder_context_t *rctx, uint16_t idx,<br> > > + reorder_window_t *rwin, uint32_t sn);<br> > > +void rctx_free(const reorder_context_t *rctx);<br> > > +void olock_unlock(const reorder_context_t *rctx, reorder_window_t *rwin,<br> > > + uint32_t lock_index);<br> > > +void olock_release(const reorder_context_t *rctx);<br> > > +void rctx_retire(reorder_context_t *first);<br> > > +void rctx_release(reorder_context_t *rctx);<br> > > +<br> > > +#else<br> > > +<br> > > +#define SUSTAIN_ORDER 1<br> > > +<br> > > +int schedule_ordered_queue_enq(uint32_t queue_index, void *p_buf_hdr,<br> > > + int sustain, int *ret);<br> > > +int schedule_ordered_queue_enq_multi(uint32_t queue_index, void *p_buf_hdr[],<br> > > + int num, int sustain, int *ret);<br> > > +#endif<br> > > +<br> > > +#ifdef __cplusplus<br> > > +}<br> > > +#endif<br> > > +<br> > > +#endif<br> > > diff --git a/platform/linux-generic/m4/odp_schedule.m4 b/platform/linux-generic/m4/odp_schedule.m4<br> > > index 91c19f21..d862b8b2 100644<br> > > --- a/platform/linux-generic/m4/odp_schedule.m4<br> > > +++ b/platform/linux-generic/m4/odp_schedule.m4<br> > > @@ -1,13 +1,44 @@<br> > > -AC_ARG_ENABLE([schedule-sp],<br> > > - [ --enable-schedule-sp enable strict priority scheduler],<br> > > - [if test x$enableval = xyes; then<br> > > - schedule_sp_enabled=yes<br> > > - ODP_CFLAGS="$ODP_CFLAGS -DODP_SCHEDULE_SP"<br> > > - fi])<br> > > +# Checks for --enable-schedule-sp and defines ODP_SCHEDULE_SP and adds<br> > > +# -DODP_SCHEDULE_SP to CFLAGS.<br> > > +AC_ARG_ENABLE(<br> > > + [schedule_sp],<br> > > + [AC_HELP_STRING([--enable-schedule-sp],<br> > > + [enable strict priority scheduler])],<br> > > + [if test "x$enableval" = xyes; then<br> > > + schedule_sp=true<br> > > + ODP_CFLAGS="$ODP_CFLAGS -DODP_SCHEDULE_SP"<br> > > + else<br> > > + schedule_sp=false<br> > > + fi],<br> > > + [schedule_sp=false])<br> > > +AM_CONDITIONAL([ODP_SCHEDULE_SP], [test x$schedule_sp = xtrue])<br> > ><br> > > -AC_ARG_ENABLE([schedule-iquery],<br> > > - [ --enable-schedule-iquery enable interests query (sparse bitmap) scheduler],<br> > > - [if test x$enableval = xyes; then<br> > > - schedule_iquery_enabled=yes<br> > > - ODP_CFLAGS="$ODP_CFLAGS -DODP_SCHEDULE_IQUERY"<br> > > - fi])<br> > > +# Checks for --enable-schedule-iquery and defines ODP_SCHEDULE_IQUERY and adds<br> > > +# -DODP_SCHEDULE_IQUERY to CFLAGS.<br> > > +AC_ARG_ENABLE(<br> > > + [schedule_iquery],<br> > > + [AC_HELP_STRING([--enable-schedule-iquery],<br> > > + [enable interests query (sparse bitmap) scheduler])],<br> > > + [if test "x$enableval" = xyes; then<br> > > + schedule_iquery=true<br> > > + ODP_CFLAGS="$ODP_CFLAGS -DODP_SCHEDULE_IQUERY"<br> > > + else<br> > > + schedule_iquery=false<br> > > + fi],<br> > > + [schedule_iquery=false])<br> > > +AM_CONDITIONAL([ODP_SCHEDULE_IQUERY], [test x$schedule_iquery = xtrue])<br> > > +<br> > > +# Checks for --enable-schedule-scalable and defines ODP_SCHEDULE_SCALABLE and<br> > > +# adds -DODP_SCHEDULE_SCALABLE to CFLAGS.<br> > > +AC_ARG_ENABLE(<br> > > + [schedule_scalable],<br> > > + [AC_HELP_STRING([--enable-schedule-scalable],<br> > > + [enable scalable scheduler])],<br> > > + [if test "x$enableval" = xyes; then<br> > > + schedule_scalable=true<br> > > + ODP_CFLAGS="$ODP_CFLAGS -DODP_SCHEDULE_SCALABLE"<br> > > + else<br> > > + schedule_scalable=false<br> > > + fi],<br> > > + [schedule_scalable=false])<br> > > +AM_CONDITIONAL([ODP_SCHEDULE_SCALABLE], [test x$schedule_scalable = xtrue])<br> > > diff --git a/platform/linux-generic/odp_classification.c b/platform/linux-generic/odp_classification.c<br> > > index 5d96b00b..8fb5c32f 100644<br> > > --- a/platform/linux-generic/odp_classification.c<br> > > +++ b/platform/linux-generic/odp_classification.c<br> > > @@ -282,7 +282,7 @@ odp_queue_t odp_cos_queue(odp_cos_t cos_id)<br> > > if (!cos->s.queue)<br> > > return ODP_QUEUE_INVALID;<br> > ><br> > > - return cos->s.queue->s.handle;<br> > > + return queue_get_handle(cos->s.queue);<br> > > }<br> > ><br> > > int odp_cos_drop_set(odp_cos_t cos_id, odp_cls_drop_t drop_policy)<br> > > @@ -849,7 +849,7 @@ int cls_classify_packet(pktio_entry_t *entry, const uint8_t *base,<br> > ><br> > > *pool = cos->s.pool;<br> > > pkt_hdr->p.input_flags.dst_queue = 1;<br> > > - pkt_hdr->dst_queue = cos->s.queue->s.handle;<br> > > + pkt_hdr->dst_queue = queue_get_handle(cos->s.queue);<br> > ><br> > > return 0;<br> > > }<br> > > diff --git a/platform/linux-generic/odp_packet.c b/platform/linux-generic/odp_packet.c<br> > > index b8aac6bf..8ae35867 100644<br> > > --- a/platform/linux-generic/odp_packet.c<br> > > +++ b/platform/linux-generic/odp_packet.c<br> > > @@ -651,6 +651,11 @@ odp_buffer_t _odp_packet_to_buffer(odp_packet_t pkt)<br> > > return buffer_handle(packet_hdr(pkt));<br> > > }<br> > ><br> > > +odp_buffer_hdr_t *_odp_packet_to_buf_hdr_ptr(odp_packet_t pkt)<br> > > +{<br> > > + return (odp_buffer_hdr_t *)(_odp_packet_to_buffer(pkt));<br> > > +}<br> > > +<br> > > odp_packet_t odp_packet_from_event(odp_event_t ev)<br> > > {<br> > > if (odp_unlikely(ev == ODP_EVENT_INVALID))<br> > > diff --git a/platform/linux-generic/odp_packet_io.c b/platform/linux-generic/odp_packet_io.c<br> > > index 5e783d83..a267ee97 100644<br> > > --- a/platform/linux-generic/odp_packet_io.c<br> > > +++ b/platform/linux-generic/odp_packet_io.c<br> > > @@ -5,23 +5,25 @@<br> > > */<br> > > #include <odp_posix_extensions.h><br> > ><br> > > -#include <odp/api/packet_io.h><br> > > -#include <odp_packet_io_internal.h><br> > > -#include <odp_packet_io_queue.h><br> > > #include <odp/api/packet.h><br> > > -#include <odp_packet_internal.h><br> > > -#include <odp_internal.h><br> > > +#include <odp/api/packet_io.h><br> > > #include <odp/api/spinlock.h><br> > > #include <odp/api/ticketlock.h><br> > > #include <odp/api/shared_memory.h><br> > > -#include <odp_packet_socket.h><br> > > +#include <odp/api/time.h><br> > > +<br> > > +#include <odp_internal.h><br> > > #include <odp_config_internal.h><br> > > -#include <odp_queue_internal.h><br> > > -#include <odp_schedule_if.h><br> > > -#include <odp_classification_internal.h><br> > > #include <odp_debug_internal.h><br> > > +<br> > > +#include <odp_classification_internal.h><br> > > +#include <odp_queue_internal.h><br> > > #include <odp_packet_io_ipc_internal.h><br> > > -#include <odp/api/time.h><br> > > +#include <odp_packet_io_internal.h><br> > > +#include <odp_packet_io_queue.h><br> > > +#include <odp_packet_internal.h><br> > > +#include <odp_packet_socket.h><br> > > +#include <odp_schedule_if.h><br> > ><br> > > #include <string.h><br> > > #include <inttypes.h><br> > > @@ -470,7 +472,6 @@ int odp_pktio_start(odp_pktio_t hdl)<br> > > return -1;<br> > > }<br> > > }<br> > > -<br> > > sched_fn->pktio_start(pktio_to_id(hdl), num, index);<br> > > }<br> > ><br> > > @@ -552,7 +553,6 @@ static inline int pktin_recv_buf(odp_pktin_queue_t queue,<br> > > odp_packet_t packets[num];<br> > > odp_packet_hdr_t *pkt_hdr;<br> > > odp_buffer_hdr_t *buf_hdr;<br> > > - odp_buffer_t buf;<br> > > int i;<br> > > int pkts;<br> > > int num_rx = 0;<br> > > @@ -562,9 +562,11 @@ static inline int pktin_recv_buf(odp_pktin_queue_t queue,<br> > > for (i = 0; i < pkts; i++) {<br> > > pkt = packets[i];<br> > > pkt_hdr = odp_packet_hdr(pkt);<br> > > - buf = _odp_packet_to_buffer(pkt);<br> > > - buf_hdr = buf_hdl_to_hdr(buf);<br> > > -<br> > > +#ifdef ODP_SCHEDULE_SCALABLE<br> > > + buf_hdr = _odp_packet_to_buf_hdr_ptr(pkt);<br> > > +#else<br> > > + buf_hdr = buf_hdl_to_hdr(_odp_packet_to_buffer(pkt));<br> > > +#endif<br> > > if (pkt_hdr->p.input_flags.dst_queue) {<br> > > queue_entry_t *dst_queue;<br> > > int ret;<br> > > @@ -582,11 +584,17 @@ static inline int pktin_recv_buf(odp_pktin_queue_t queue,<br> > ><br> > > int pktout_enqueue(queue_entry_t *qentry, odp_buffer_hdr_t *buf_hdr)<br> > > {<br> > > - odp_packet_t pkt = _odp_packet_from_buffer(buf_hdr->handle.handle);<br> > > + odp_packet_t pkt;<br> > > +<br> > > +#ifdef ODP_SCHEDULE_SCALABLE<br> > > + pkt = _odp_packet_from_buffer((odp_buffer_t)buf_hdr);<br> > > +#else<br> > > + pkt = _odp_packet_from_buffer(buf_hdr->handle.handle);<br> > > +#endif<br> > > int len = 1;<br> > > int nbr;<br> > ><br> > > - nbr = odp_pktout_send(qentry->s.pktout, &pkt, len);<br> > > + nbr = odp_pktout_send(queue_get_pktout(qentry), &pkt, len);<br> > > return (nbr == len ? 0 : -1);<br> > > }<br> > ><br> > > @@ -604,9 +612,13 @@ int pktout_enq_multi(queue_entry_t *qentry, odp_buffer_hdr_t *buf_hdr[],<br> > > int i;<br> > ><br> > > for (i = 0; i < num; ++i)<br> > > +#ifdef ODP_SCHEDULE_SCALABLE<br> > > + pkt_tbl[i] = _odp_packet_from_buffer((odp_buffer_t)buf_hdr[i]);<br> > > +#else<br> > > pkt_tbl[i] = _odp_packet_from_buffer(buf_hdr[i]->handle.handle);<br> > > +#endif<br> > ><br> > > - nbr = odp_pktout_send(qentry->s.pktout, pkt_tbl, num);<br> > > + nbr = odp_pktout_send(queue_get_pktout(qentry), pkt_tbl, num);<br> > > return nbr;<br> > > }<br> > ><br> > > @@ -632,13 +644,14 @@ odp_buffer_hdr_t *pktin_dequeue(queue_entry_t *qentry)<br> > > int pkts;<br> > ><br> > > buf_hdr = queue_deq(qentry);<br> > > - if (buf_hdr != NULL)<br> > > + if (buf_hdr != BUFFER_HDR_INVALID)<br> > > return buf_hdr;<br> > ><br> > > - pkts = pktin_recv_buf(qentry->s.pktin, hdr_tbl, QUEUE_MULTI_MAX);<br> > > + pkts = pktin_recv_buf(queue_get_pktin(qentry),<br> > > + hdr_tbl, QUEUE_MULTI_MAX);<br> > ><br> > > if (pkts <= 0)<br> > > - return NULL;<br> > > + return BUFFER_HDR_INVALID;<br> > ><br> > > if (pkts > 1)<br> > > queue_enq_multi(qentry, &hdr_tbl[1], pkts - 1);<br> > > @@ -669,7 +682,8 @@ int pktin_deq_multi(queue_entry_t *qentry, odp_buffer_hdr_t *buf_hdr[], int num)<br> > > if (nbr == num)<br> > > return nbr;<br> > ><br> > > - pkts = pktin_recv_buf(qentry->s.pktin, hdr_tbl, QUEUE_MULTI_MAX);<br> > > + pkts = pktin_recv_buf(queue_get_pktin(qentry),<br> > > + hdr_tbl, QUEUE_MULTI_MAX);<br> > > if (pkts <= 0)<br> > > return nbr;<br> > ><br> > > @@ -684,7 +698,6 @@ int pktin_deq_multi(queue_entry_t *qentry, odp_buffer_hdr_t *buf_hdr[], int num)<br> > > queue_enq_multi(qentry, hdr_tbl, j);<br> > > return nbr;<br> > > }<br> > > -<br> > > int sched_cb_pktin_poll(int pktio_index, int num_queue, int index[])<br> > > {<br> > > odp_buffer_hdr_t *hdr_tbl[QUEUE_MULTI_MAX];<br> > > @@ -1266,13 +1279,14 @@ int odp_pktin_queue_config(odp_pktio_t pktio,<br> > > queue_entry_t *qentry;<br> > ><br> > > qentry = queue_to_qentry(queue);<br> > > - qentry->s.pktin.index = i;<br> > > - qentry->s.pktin.pktio = pktio;<br> > > -<br> > > - qentry->s.enqueue = pktin_enqueue;<br> > > - qentry->s.dequeue = pktin_dequeue;<br> > > - qentry->s.enqueue_multi = pktin_enq_multi;<br> > > - qentry->s.dequeue_multi = pktin_deq_multi;<br> > > + queue_set_pktin(qentry, pktio, i);<br> > > +<br> > > + queue_set_enq_func(qentry, pktin_enqueue);<br> > > + queue_set_deq_func(qentry, pktin_dequeue);<br> > > + queue_set_enq_multi_func(qentry,<br> > > + pktin_enq_multi);<br> > > + queue_set_deq_multi_func(qentry,<br> > > + pktin_deq_multi);<br> > > }<br> > ><br> > > entry->s.in_queue[i].queue = queue;<br> > > @@ -1390,14 +1404,12 @@ int odp_pktout_queue_config(odp_pktio_t pktio,<br> > > }<br> > ><br> > > qentry = queue_to_qentry(queue);<br> > > - qentry->s.pktout.index = i;<br> > > - qentry->s.pktout.pktio = pktio;<br> > > -<br> > > - /* Override default enqueue / dequeue functions */<br> > > - qentry->s.enqueue = pktout_enqueue;<br> > > - qentry->s.dequeue = pktout_dequeue;<br> > > - qentry->s.enqueue_multi = pktout_enq_multi;<br> > > - qentry->s.dequeue_multi = pktout_deq_multi;<br> > > + queue_set_pktout(qentry, pktio, i);<br> > > +<br> > > + queue_set_enq_func(qentry, pktout_enqueue);<br> > > + queue_set_deq_func(qentry, pktout_dequeue);<br> > > + queue_set_enq_multi_func(qentry, pktout_enq_multi);<br> > > + queue_set_deq_multi_func(qentry, pktout_deq_multi);<br> > ><br> > > entry->s.out_queue[i].queue = queue;<br> > > }<br> > > diff --git a/platform/linux-generic/odp_queue.c b/platform/linux-generic/odp_queue.c<br> > > index fcf4bf5b..42b58c44 100644<br> > > --- a/platform/linux-generic/odp_queue.c<br> > > +++ b/platform/linux-generic/odp_queue.c<br> > > @@ -291,11 +291,11 @@ void sched_cb_queue_destroy_finalize(uint32_t queue_index)<br> > > int odp_queue_destroy(odp_queue_t handle)<br> > > {<br> > > queue_entry_t *queue;<br> > > - queue = queue_to_qentry(handle);<br> > ><br> > > if (handle == ODP_QUEUE_INVALID)<br> > > return -1;<br> > ><br> > > + queue = queue_to_qentry(handle);<br> > > LOCK(&queue->s.lock);<br> > > if (queue->s.status == QUEUE_STATUS_FREE) {<br> > > UNLOCK(&queue->s.lock);<br> > > diff --git a/platform/linux-generic/odp_queue_scalable.c b/platform/linux-generic/odp_queue_scalable.c<br> > > new file mode 100644<br> > > index 00000000..c08dd06a<br> > > --- /dev/null<br> > > +++ b/platform/linux-generic/odp_queue_scalable.c<br> > > @@ -0,0 +1,944 @@<br> > > +/* Copyright (c) 2017, ARM Limited<br> > > + * All rights reserved.<br> > > + *<br> > > + * SPDX-License-Identifier: BSD-3-Clause<br> > > + */<br> > > +<br> > > +#include <odp/api/hints.h><br> > > +#include <odp/api/plat/ticketlock_inlines.h><br> > > +#include <odp/api/queue.h><br> > > +#include <odp/api/schedule.h><br> > > +#include <odp/api/shared_memory.h><br> > > +#include <odp/api/sync.h><br> > > +#include <odp/api/traffic_mngr.h><br> > > +<br> > > +#include <odp_internal.h><br> > > +#include <odp_config_internal.h><br> > > +#include <odp_debug_internal.h><br> > > +<br> > > +#include <odp_buffer_inlines.h><br> > > +#include <odp_packet_io_internal.h><br> > > +#include <odp_packet_io_queue.h><br> > > +#include <odp_pool_internal.h><br> > > +#include <odp_queue_internal.h><br> > > +#include <odp_schedule_if.h><br> > > +<br> > > +#include <string.h><br> > > +#include <inttypes.h><br> > > +<br> > > +#define NUM_INTERNAL_QUEUES 64<br> > > +<br> > > +#define MIN(a, b) \<br> > > + ({ \<br> > > + __typeof__(a) tmp_a = (a); \<br> > > + __typeof__(b) tmp_b = (b); \<br> > > + tmp_a < tmp_b ? tmp_a : tmp_b; \<br> > > + })<br> > > +<br> > > +#define LOCK(a) _odp_ticketlock_lock(a)<br> > > +#define UNLOCK(a) _odp_ticketlock_unlock(a)<br> > > +#define LOCK_INIT(a) odp_ticketlock_init(a)<br> > > +<br> > > +extern __thread sched_scalable_thread_state_t *sched_ts;<br> > > +<br> > > +typedef struct queue_table_t {<br> > > + queue_entry_t queue[ODP_CONFIG_QUEUES];<br> > > +} queue_table_t;<br> > > +<br> > > +static queue_table_t *queue_tbl;<br> > > +<br> > > +static inline odp_queue_t queue_from_id(uint32_t queue_id)<br> > > +{<br> > > + return _odp_cast_scalar(odp_queue_t, queue_id + 1);<br> > > +}<br> > > +<br> > > +queue_entry_t *get_qentry(uint32_t queue_id)<br> > > +{<br> > > + return &queue_tbl->queue[queue_id];<br> > > +}<br> > > +<br> > > +static int _odp_queue_disable_enq(sched_elem_t *q)<br> > > +{<br> > > + ringidx_t old_read, old_write, new_write;<br> > > + uint32_t size;<br> > > +<br> > > +#ifndef ODP_CONFIG_USE_LLSC<br> > > + old_write = q->prod_write;<br> > > +#endif<br> > > + size = q->prod_mask + 1;<br> > > + do {<br> > > + /* Need __atomic_load to avoid compiler reordering */<br> > > + old_read = __atomic_load_n(&q->prod_read, __ATOMIC_ACQUIRE);<br> > > +#ifdef ODP_CONFIG_USE_LLSC<br> > > + old_write = ll32(&q->prod_write, __ATOMIC_RELAXED);<br> > > +#endif<br> > > + if (old_write != old_read) {<br> > > + /* Queue is not empty, cannot claim all elements<br> > > + * Cannot disable enqueue.<br> > > + */<br> > > + return -1;<br> > > + }<br> > > + /* Claim all elements in ring */<br> > > + new_write = old_write + size;<br> > > +#ifdef ODP_CONFIG_USE_LLSC<br> > > + } while (odp_unlikely(sc32(&q->prod_write,<br> > > + new_write,<br> > > + __ATOMIC_RELAXED)));<br> > > +#else<br> > > + } while (!__atomic_compare_exchange_n(&q->prod_write,<br> > > + &old_write, /* Updated on failure */<br> > > + new_write,<br> > > + true,<br> > > + __ATOMIC_RELAXED,<br> > > + __ATOMIC_RELAXED));<br> > > +#endif<br> > > + /* All remaining elements claimed, noone else can enqueue */<br> > > + return 0;<br> > > +}<br> > > +<br> > > +static uint32_t roundup_pow2(uint32_t v)<br> > > +{<br> > > + uint32_t i;<br> > > +<br> > > + /* Check if already power of 2 */<br> > > + if ((v & (v - 1)) == 0)<br> > > + return v;<br> > > +<br> > > + for (i = 0; (i < (sizeof(v) * CHAR_BIT)) && (v != 0); i++)<br> > > + v = v >> 1;<br> > > +<br> > > + return 1<<i;<br> > > +}<br> > > +<br> > > +static int queue_init(int queue_idx, queue_entry_t *queue, const char *name,<br> > > + const odp_queue_param_t *param)<br> > > +{<br> > > + ringidx_t ring_idx;<br> > > + sched_elem_t *sched_elem;<br> > > + odp_shm_t shm;<br> > > + uint32_t ring_size;<br> > > + odp_buffer_hdr_t **ring;<br> > > +<br> > > + sched_elem = &queue->s.sched_elem;<br> > > + ring_size = param->ring_size > 0 ?<br> > > + roundup_pow2(param->ring_size) :<br> > > + ODP_CONFIG_QUEUE_SIZE;<br> > > + strncpy(queue->s.name, name ? name : "", ODP_QUEUE_NAME_LEN - 1);<br> > > + queue->s.name[ODP_QUEUE_NAME_LEN - 1] = 0;<br> > > + memcpy(&queue->s.param, param, sizeof(odp_queue_param_t));<br> > > +<br> > > + shm = odp_shm_reserve(name,<br> > > + ring_size * sizeof(odp_buffer_hdr_t *),<br> > > + ODP_CACHE_LINE_SIZE,<br> > > + ODP_SHM_PROC);<br> > > + if (ODP_SHM_INVALID == shm)<br> > > + return -1;<br> > > +<br> > > + ring = (odp_buffer_hdr_t **)odp_shm_addr(shm);<br> > > +<br> > > + for (ring_idx = 0; ring_idx < ring_size; ring_idx++)<br> > > + ring[ring_idx] = NULL;<br> > > +<br> > > + queue->s.shm = shm;<br> > > + queue->s.type = queue->s.param.type;<br> > > + queue->s.enqueue = queue_enq;<br> > > + queue->s.dequeue = queue_deq;<br> > > + queue->s.enqueue_multi = queue_enq_multi;<br> > > + queue->s.dequeue_multi = queue_deq_multi;<br> > > + queue->s.pktin = PKTIN_INVALID;<br> > > +<br> > > + sched_elem->node.next = NULL;<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + LOCK_INIT(&sched_elem->qschlock);<br> > > +#endif<br> > > + sched_elem->qschst.numevts = 0;<br> > > + sched_elem->qschst.wrr_budget = CONFIG_WRR_WEIGHT;<br> > > + sched_elem->qschst.cur_ticket = 0;<br> > > + sched_elem->qschst.nxt_ticket = 0;<br> > > + sched_elem->pop_deficit = 0;<br> > > + if (queue->s.type == ODP_QUEUE_TYPE_SCHED)<br> > > + sched_elem->qschst_type = queue->s.param.sched.sync;<br> > > + else<br> > > + sched_elem->qschst_type = ODP_NO_SCHED_QUEUE;<br> > > + /* 2nd cache line - enqueue */<br> > > + sched_elem->prod_read = 0;<br> > > + sched_elem->prod_write = 0;<br> > > + sched_elem->prod_ring = ring;<br> > > + sched_elem->prod_mask = ring_size - 1;<br> > > + /* 3rd cache line - dequeue */<br> > > + sched_elem->cons_read = 0;<br> > > + sched_elem->cons_write = 0;<br> > > + sched_elem->rwin = NULL;<br> > > + sched_elem->schedq = NULL;<br> > > + sched_elem->user_ctx = queue->s.param.context;<br> > > +#ifdef ODP_CONFIG_USE_SPLIT_PRODCONS<br> > > + sched_elem->cons_ring = ring;<br> > > + sched_elem->cons_mask = ring_size - 1;<br> > > + sched_elem->cons_type = sched_elem->qschst_type;<br> > > +#endif<br> > > +<br> > > + /* Queue initialized successfully, add it to the sched group */<br> > > + if (queue->s.type == ODP_QUEUE_TYPE_SCHED) {<br> > > + if (queue->s.param.sched.sync == ODP_SCHED_SYNC_ORDERED) {<br> > > + sched_elem->rwin =<br> > > + rwin_alloc(queue_idx, &queue->s.rwin_shm,<br> > > + queue->s.param.sched.lock_count);<br> > > + if (sched_elem->rwin == NULL) {<br> > > + ODP_ERR("Reorder window not created\n");<br> > > + goto rwin_create_failed;<br> > > + }<br> > > + }<br> > > + sched_elem->schedq =<br> > > + schedq_from_sched_group(param->sched.group,<br> > > + param->sched.prio);<br> > > + }<br> > > +<br> > > + return 0;<br> > > +<br> > > +rwin_create_failed:<br> > > + odp_shm_free(queue->s.shm);<br> > > + return -1;<br> > > +}<br> > > +<br> > > +int odp_queue_init_global(void)<br> > > +{<br> > > + uint32_t i;<br> > > + odp_shm_t shm;<br> > > +<br> > > + ODP_DBG("Queue init ... ");<br> > > +<br> > > + shm = odp_shm_reserve("odp_queues",<br> > > + sizeof(queue_table_t),<br> > > + sizeof(queue_entry_t), 0);<br> > > +<br> > > + queue_tbl = odp_shm_addr(shm);<br> > > +<br> > > + if (queue_tbl == NULL)<br> > > + return -1;<br> > > +<br> > > + memset(queue_tbl, 0, sizeof(queue_table_t));<br> > > +<br> > > + for (i = 0; i < ODP_CONFIG_QUEUES; i++) {<br> > > + /* init locks */<br> > > + queue_entry_t *queue;<br> > > +<br> > > + queue = get_qentry(i);<br> > > + LOCK_INIT(&queue->s.lock);<br> > > + queue->s.index = i;<br> > > + queue->s.handle = queue_from_id(i);<br> > > + }<br> > > +<br> > > + ODP_DBG("done\n");<br> > > + ODP_DBG("Queue init global\n");<br> > > + ODP_DBG(" struct queue_entry_s size %zu\n",<br> > > + sizeof(struct queue_entry_s));<br> > > + ODP_DBG(" queue_entry_t size %zu\n",<br> > > + sizeof(queue_entry_t));<br> > > + ODP_DBG("\n");<br> > > +<br> > > + return 0;<br> > > +}<br> > > +<br> > > +int odp_queue_term_global(void)<br> > > +{<br> > > + int ret = 0;<br> > > + int rc = 0;<br> > > + queue_entry_t *queue;<br> > > + int i;<br> > > +<br> > > + for (i = 0; i < ODP_CONFIG_QUEUES; i++) {<br> > > + queue = &queue_tbl->queue[i];<br> > > + if (__atomic_load_n(&queue->s.status,<br> > > + __ATOMIC_RELAXED) != QUEUE_STATUS_FREE) {<br> > > + ODP_ERR("Not destroyed queue: %s\n", queue->s.name);<br> > > + rc = -1;<br> > > + }<br> > > + }<br> > > +<br> > > + ret = odp_shm_free(odp_shm_lookup("odp_queues"));<br> > > + if (ret < 0) {<br> > > + ODP_ERR("shm free failed for odp_queues");<br> > > + rc = -1;<br> > > + }<br> > > +<br> > > + return rc;<br> > > +}<br> > > +<br> > > +int odp_queue_capability(odp_queue_capability_t *capa)<br> > > +{<br> > > + memset(capa, 0, sizeof(odp_queue_capability_t));<br> > > +<br> > > + /* Reserve some queues for internal use */<br> > > + capa->max_queues = ODP_CONFIG_QUEUES - NUM_INTERNAL_QUEUES;<br> > > + capa->max_ordered_locks = SCHEDULE_ORDERED_LOCKS_PER_QUEUE;<br> > > + capa->max_sched_groups = sched_fn->num_grps();<br> > > + capa->sched_prios = odp_schedule_num_prio();<br> > > +<br> > > + return 0;<br> > > +}<br> > > +<br> > > +odp_queue_type_t odp_queue_type(odp_queue_t handle)<br> > > +{<br> > > + return queue_to_qentry(handle)->s.type;<br> > > +}<br> > > +<br> > > +odp_schedule_sync_t odp_queue_sched_type(odp_queue_t handle)<br> > > +{<br> > > + return queue_to_qentry(handle)->s.param.sched.sync;<br> > > +}<br> > > +<br> > > +odp_schedule_prio_t odp_queue_sched_prio(odp_queue_t handle)<br> > > +{<br> > > + return queue_to_qentry(handle)->s.param.sched.prio;<br> > > +}<br> > > +<br> > > +odp_schedule_group_t odp_queue_sched_group(odp_queue_t handle)<br> > > +{<br> > > + return queue_to_qentry(handle)->s.param.sched.group;<br> > > +}<br> > > +<br> > > +int odp_queue_lock_count(odp_queue_t handle)<br> > > +{<br> > > + queue_entry_t *queue = queue_to_qentry(handle);<br> > > +<br> > > + return queue->s.param.sched.sync == ODP_SCHED_SYNC_ORDERED ?<br> > > + (int)queue->s.param.sched.lock_count : -1;<br> > > +}<br> > > +<br> > > +odp_queue_t odp_queue_create(const char *name, const odp_queue_param_t *param)<br> > > +{<br> > > + int queue_idx;<br> > > + odp_queue_t handle = ODP_QUEUE_INVALID;<br> > > + queue_entry_t *queue;<br> > > + odp_queue_param_t default_param;<br> > > +<br> > > + if (param == NULL) {<br> > > + odp_queue_param_init(&default_param);<br> > > + param = &default_param;<br> > > + }<br> > > +<br> > > + for (queue_idx = 0; queue_idx < ODP_CONFIG_QUEUES; queue_idx++) {<br> > > + queue = &queue_tbl->queue[queue_idx];<br> > > +<br> > > + if (queue->s.status != QUEUE_STATUS_FREE)<br> > > + continue;<br> > > +<br> > > + LOCK(&queue->s.lock);<br> > > + if (queue->s.status == QUEUE_STATUS_FREE) {<br> > > + if (queue_init(queue_idx, queue, name, param)) {<br> > > + UNLOCK(&queue->s.lock);<br> > > + return handle;<br> > > + }<br> > > + queue->s.status = QUEUE_STATUS_READY;<br> > > + handle = queue->s.handle;<br> > > + UNLOCK(&queue->s.lock);<br> > > + break;<br> > > + }<br> > > + UNLOCK(&queue->s.lock);<br> > > + }<br> > > + return handle;<br> > > +}<br> > > +<br> > > +int odp_queue_destroy(odp_queue_t handle)<br> > > +{<br> > > + queue_entry_t *queue;<br> > > + sched_elem_t *q;<br> > > +<br> > > + if (handle == ODP_QUEUE_INVALID)<br> > > + return -1;<br> > > +<br> > > + queue = queue_to_qentry(handle);<br> > > + LOCK(&queue->s.lock);<br> > > + if (queue->s.status != QUEUE_STATUS_READY) {<br> > > + UNLOCK(&queue->s.lock);<br> > > + return -1;<br> > > + }<br> > > + q = &queue->s.sched_elem;<br> > > +<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + LOCK(&q->qschlock);<br> > > +#endif<br> > > + if (_odp_queue_disable_enq(q)) {<br> > > + /* Producer side not empty */<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + UNLOCK(&q->qschlock);<br> > > +#endif<br> > > + UNLOCK(&queue->s.lock);<br> > > + return -1;<br> > > + }<br> > > + /* Enqueue is now disabled */<br> > > + if (q->cons_read != q->cons_write) {<br> > > + /* Consumer side is not empty<br> > > + * Roll back previous change, enable enqueue again.<br> > > + */<br> > > + uint32_t size;<br> > > +<br> > > + size = q->prod_mask + 1;<br> > > + __atomic_fetch_sub(&q->prod_write, size, __ATOMIC_RELAXED);<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + UNLOCK(&q->qschlock);<br> > > +#endif<br> > > + UNLOCK(&queue->s.lock);<br> > > + return -1;<br> > > + }<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + UNLOCK(&q->qschlock);<br> > > +#endif<br> > > + /* Producer and consumer sides empty, enqueue disabled<br> > > + * Now wait until schedq state is empty and no outstanding tickets<br> > > + */<br> > > + while (__atomic_load_n(&q->qschst.numevts, __ATOMIC_RELAXED) != 0 ||<br> > > + __atomic_load_n(&q->qschst.cur_ticket, __ATOMIC_RELAXED) !=<br> > > + __atomic_load_n(&q->qschst.nxt_ticket, __ATOMIC_RELAXED)) {<br> > > + SEVL();<br> > > + while (WFE() && LDXR32((uint32_t *)&q->qschst.numevts,<br> > > + __ATOMIC_RELAXED) != 0)<br> > > + DOZE();<br> > > + }<br> > > +<br> > > + /* Adjust the spread factor for the queues in the schedule group */<br> > > + if (queue->s.type == ODP_QUEUE_TYPE_SCHED)<br> > > + sched_group_xcount_dec(queue->s.param.sched.group,<br> > > + queue->s.param.sched.prio);<br> > > +<br> > > + if (odp_shm_free(queue->s.shm) < 0) {<br> > > + UNLOCK(&queue->s.lock);<br> > > + return -1;<br> > > + }<br> > > + if (queue->s.param.sched.sync == ODP_SCHED_SYNC_ORDERED) {<br> > > + if (odp_shm_free(queue->s.rwin_shm) < 0) {<br> > > + ODP_ERR("Failed to free reorder window shm\n");<br> > > + UNLOCK(&queue->s.lock);<br> > > + return -1;<br> > > + }<br> > > + }<br> > > + queue->s.status = QUEUE_STATUS_FREE;<br> > > + UNLOCK(&queue->s.lock);<br> > > + return 0;<br> > > +}<br> > > +<br> > > +int odp_queue_context_set(odp_queue_t handle, void *context,<br> > > + uint32_t len ODP_UNUSED)<br> > > +{<br> > > + odp_mb_full();<br> > > + queue_to_qentry(handle)->s.param.context = context;<br> > > + odp_mb_full();<br> > > + return 0;<br> > > +}<br> > > +<br> > > +void *odp_queue_context(odp_queue_t handle)<br> > > +{<br> > > + return queue_to_qentry(handle)->s.param.context;<br> > > +}<br> > > +<br> > > +odp_queue_t odp_queue_lookup(const char *name)<br> > > +{<br> > > + uint32_t i;<br> > > +<br> > > + for (i = 0; i < ODP_CONFIG_QUEUES; i++) {<br> > > + queue_entry_t *queue = &queue_tbl->queue[i];<br> > > +<br> > > + if (queue->s.status == QUEUE_STATUS_FREE ||<br> > > + queue->s.status == QUEUE_STATUS_DESTROYED)<br> > > + continue;<br> > > +<br> > > + LOCK(&queue->s.lock);<br> > > + if (strcmp(name, queue->s.name) == 0) {<br> > > + /* found it */<br> > > + UNLOCK(&queue->s.lock);<br> > > + return queue->s.handle;<br> > > + }<br> > > + UNLOCK(&queue->s.lock);<br> > > + }<br> > > +<br> > > + return ODP_QUEUE_INVALID;<br> > > +}<br> > > +<br> > > +static inline int _odp_queue_enq(sched_elem_t *q,<br> > > + odp_buffer_hdr_t *buf_hdr[],<br> > > + int num)<br> > > +{<br> > > + ringidx_t old_read;<br> > > + ringidx_t old_write;<br> > > + ringidx_t new_write;<br> > > + int actual;<br> > > + uint32_t mask;<br> > > + odp_buffer_hdr_t **ring;<br> > > +<br> > > + mask = q->prod_mask;<br> > > + ring = q->prod_ring;<br> > > +<br> > > + /* Load producer ring state (read & write index) */<br> > > +#ifndef ODP_CONFIG_USE_LLSC<br> > > + old_write = __atomic_load_n(&q->prod_write, __ATOMIC_RELAXED);<br> > > +#endif<br> > > + do {<br> > > + /* Consumer does store-release prod_read, we need<br> > > + * load-acquire.<br> > > + */<br> > > + old_read = __atomic_load_n(&q->prod_read, __ATOMIC_ACQUIRE);<br> > > +#ifdef ODP_CONFIG_USE_LLSC<br> > > + old_write = ll32(&q->prod_write, __ATOMIC_RELAXED);<br> > > +#endif<br> > > +<br> > > + actual = MIN(num, (int)((mask + 1) - (old_write - old_read)));<br> > > + if (odp_unlikely(actual <= 0))<br> > > + return 0;<br> > > +<br> > > + new_write = old_write + actual;<br> > > +#ifdef ODP_CONFIG_USE_LLSC<br> > > + } while (odp_unlikely(sc32(&q->prod_write,<br> > > + new_write,<br> > > + __ATOMIC_RELAXED)));<br> > > +#else<br> > > + } while (!__atomic_compare_exchange_n(&q->prod_write,<br> > > + &old_write, /* Updated on failure */<br> > > + new_write,<br> > > + true,<br> > > + __ATOMIC_RELAXED,<br> > > + __ATOMIC_RELAXED));<br> > > +#endif<br> > > +<br> > > +#ifdef ODP_CONFIG_USE_SPLIT_PRODCONS<br> > > + __builtin_prefetch(&q->cons_write, 0, 0);<br> > > +#endif<br> > > + /* Store our event(s) in the ring */<br> > > + do {<br> > > + ring[old_write & mask] = *buf_hdr++;<br> > > + } while (++old_write != new_write);<br> > > + old_write -= actual;<br> > > +<br> > > +#ifdef ODP_CONFIG_USE_SPLIT_PRODCONS<br> > > + __builtin_prefetch(&q->node, 1, 0);<br> > > +#endif<br> > > + /* Wait for our turn to signal consumers */<br> > > + if (odp_unlikely(__atomic_load_n(&q->cons_write,<br> > > + __ATOMIC_RELAXED) != old_write)) {<br> > > + SEVL();<br> > > + while (WFE() && LDXR32(&q->cons_write,<br> > > + __ATOMIC_RELAXED) != old_write)<br> > > + DOZE();<br> > > + }<br> > > +<br> > > + /* Signal consumers that events are available (release events)<br> > > + * Enable other producers to continue<br> > > + */<br> > > +#ifdef ODP_CONFIG_USE_DMB<br> > > + /* Wait for writes (to ring slots) to complete */<br> > > + __asm volatile("dmb ishst" ::: "memory");<br> > > + q->cons_write = new_write;<br> > > +#else<br> > > + __atomic_store_n(&q->cons_write, new_write, __ATOMIC_RELEASE);<br> > > +#endif<br> > > +<br> > > + return actual;<br> > > +}<br> > > +<br> > > +static inline int _odp_queue_enq_sp(sched_elem_t *q,<br> > > + odp_buffer_hdr_t *buf_hdr[],<br> > > + int num)<br> > > +{<br> > > + ringidx_t old_read;<br> > > + ringidx_t old_write;<br> > > + ringidx_t new_write;<br> > > + int actual;<br> > > + uint32_t mask;<br> > > + odp_buffer_hdr_t **ring;<br> > > +<br> > > + mask = q->prod_mask;<br> > > + ring = q->prod_ring;<br> > > +<br> > > + /* Load producer ring state (read & write index) */<br> > > + old_write = q->prod_write;<br> > > + /* Consumer does store-release prod_read, we need load-acquire */<br> > > + old_read = __atomic_load_n(&q->prod_read, __ATOMIC_ACQUIRE);<br> > > + actual = MIN(num, (int)((mask + 1) - (old_write - old_read)));<br> > > + if (odp_unlikely(actual <= 0))<br> > > + return 0;<br> > > +<br> > > + new_write = old_write + actual;<br> > > + q->prod_write = new_write;<br> > > +<br> > > + /* Store our event(s) in the ring */<br> > > + do {<br> > > + ring[old_write & mask] = *buf_hdr++;<br> > > + } while (++old_write != new_write);<br> > > + old_write -= actual;<br> > > +<br> > > +#ifdef ODP_CONFIG_USE_SPLIT_PRODCONS<br> > > + __builtin_prefetch(&q->node, 1, 0);<br> > > +#endif<br> > > +<br> > > + /* Signal consumers that events are available (release events)<br> > > + * Enable other producers to continue<br> > > + */<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + q->cons_write = new_write;<br> > > +#else<br> > > +#ifdef ODP_CONFIG_USE_DMB<br> > > + __asm volatile("dmb ishst" ::: "memory");<br> > > + q->cons_write = new_write;<br> > > +#else<br> > > + __atomic_store_n(&q->cons_write, new_write, __ATOMIC_RELEASE);<br> > > +#endif<br> > > +#endif<br> > > +<br> > > + return actual;<br> > > +}<br> > > +<br> > > +int queue_enq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[], int num)<br> > > +{<br> > > + int actual;<br> > > + int i;<br> > > + sched_scalable_thread_state_t *ts;<br> > > + reorder_context_t *first;<br> > > + reorder_context_t *cur;<br> > > + bitset_t next_idx;<br> > > +<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + LOCK(&queue->s.sched_elem.qschlock);<br> > > +#endif<br> > > + ts = sched_ts;<br> > > + if (ts && odp_unlikely(ts->out_of_order)) {<br> > > + first = ts->rctx;<br> > > + ODP_ASSERT(ts->rctx != NULL);<br> > > + cur = &first[(int)first->cur_idx - (int)first->idx];<br> > > + for (i = 0; i < num; i++) {<br> > > + if (odp_unlikely(cur->numevts == RC_EVT_SIZE)) {<br> > > + /* No more space in current reorder context<br> > > + * Try to allocate another.<br> > > + */<br> > > + if (odp_unlikely(<br> > > + bitset_is_null(ts->priv_rvec_free))) {<br> > > + ts->priv_rvec_free =<br> > > + atom_bitset_xchg(<br> > > + &ts->rvec_free,<br> > > + 0,<br> > > + __ATOMIC_RELAXED);<br> > > + if (odp_unlikely(bitset_is_null(<br> > > + ts->priv_rvec_free)))<br> > > + /* Out of reorder contexts.<br> > > + * Return the number of events<br> > > + * stored so far.<br> > > + */<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + UNLOCK(&queue->s.sched_elem.<br> > > + qschlock);<br> > > +#endif<br> > > + return i;<br> > > + }<br> > > + next_idx = bitset_ffs(ts->priv_rvec_free) - 1;<br> > > + ts->priv_rvec_free =<br> > > + bitset_clr(ts->priv_rvec_free,<br> > > + next_idx);<br> > > + /* Link current to next (for eventual<br> > > + * retiring)<br> > > + */<br> > > + cur->next_idx = next_idx;<br> > > + /* Link first to next (for next call to<br> > > + * queue_enq_multi())<br> > > + */<br> > > + first->cur_idx = next_idx;<br> > > + /* Update current to next */<br> > > + cur = &ts->rvec[next_idx];<br> > > + rctx_init(cur, next_idx, NULL, 0);<br> > > + /* The last rctx (so far) */<br> > > + cur->next_idx = first->idx;<br> > > + }<br> > > + cur->events[cur->numevts] = buf_hdr[i];<br> > > + cur->destq[cur->numevts] = queue;<br> > > + cur->numevts++;<br> > > + }<br> > > + /* All events stored. */<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + UNLOCK(&queue->s.sched_elem.qschlock);<br> > > +#endif<br> > > + return num;<br> > > + }<br> > > +<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + actual = _odp_queue_enq_sp(&queue->s.sched_elem, buf_hdr, num);<br> > > +#else<br> > > + actual = _odp_queue_enq(&queue->s.sched_elem, buf_hdr, num);<br> > > +#endif<br> > > +<br> > > + if (odp_likely(queue->s.sched_elem.schedq != NULL && actual != 0)) {<br> > > + /* Perform scheduler related updates. */<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + sched_update_enq_sp(&queue->s.sched_elem, actual);<br> > > +#else<br> > > + sched_update_enq(&queue->s.sched_elem, actual);<br> > > +#endif<br> > > + }<br> > > +<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + UNLOCK(&queue->s.sched_elem.qschlock);<br> > > +#endif<br> > > + return actual;<br> > > +}<br> > > +<br> > > +int queue_enq(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr)<br> > > +{<br> > > + return odp_likely(<br> > > + queue_enq_multi(queue, &buf_hdr, 1) == 1) ? 0 : -1;<br> > > +}<br> > > +<br> > > +int odp_queue_enq_multi(odp_queue_t handle, const odp_event_t ev[], int num)<br> > > +{<br> > > + odp_buffer_hdr_t *buf_hdr[QUEUE_MULTI_MAX];<br> > > + queue_entry_t *queue;<br> > > + int i;<br> > > +<br> > > + if (num > QUEUE_MULTI_MAX)<br> > > + num = QUEUE_MULTI_MAX;<br> > > +<br> > > + queue = queue_to_qentry(handle);<br> > > +<br> > > + for (i = 0; i < num; i++)<br> > > + buf_hdr[i] = (odp_buffer_hdr_t *)ev[i];<br> > > +<br> > > + return queue->s.enqueue_multi(queue, buf_hdr, num);<br> > > +}<br> > > +<br> > > +int odp_queue_enq(odp_queue_t handle, odp_event_t ev)<br> > > +{<br> > > + queue_entry_t *queue;<br> > > +<br> > > + queue = queue_to_qentry(handle);<br> > > + return queue->s.enqueue(queue, (odp_buffer_hdr_t *)ev);<br> > > +}<br> > > +<br> > > +/* Single-consumer dequeue. */<br> > > +int _odp_queue_deq_sc(sched_elem_t *q, odp_event_t *evp, int num)<br> > > +{<br> > > + int actual;<br> > > + int i;<br> > > + ringidx_t old_read;<br> > > + ringidx_t old_write;<br> > > + ringidx_t new_read;<br> > > + uint32_t mask;<br> > > + odp_event_t *ring;<br> > > +<br> > > + /* Load consumer ring state (read & write index). */<br> > > + old_read = q->cons_read;<br> > > + /* Producer does store-release cons_write, we need load-acquire */<br> > > + old_write = __atomic_load_n(&q->cons_write, __ATOMIC_ACQUIRE);<br> > > + actual = MIN(num, (int)(old_write - old_read));<br> > > +<br> > > + if (odp_unlikely(actual <= 0))<br> > > + return 0;<br> > > +<br> > > +#ifdef ODP_CONFIG_USE_SPLIT_PRODCONS<br> > > + __builtin_prefetch(&q->node, 1, 0);<br> > > +#endif<br> > > + new_read = old_read + actual;<br> > > + q->cons_read = new_read;<br> > > +<br> > > + mask = q->cons_mask;<br> > > + ring = (odp_event_t *)q->cons_ring;<br> > > + do {<br> > > + *evp++ = ring[old_read & mask];<br> > > + } while (++old_read != new_read);<br> > > +<br> > > + /* Signal producers that empty slots are available<br> > > + * (release ring slots). Enable other consumers to continue.<br> > > + */<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + q->prod_read = new_read;<br> > > +#else<br> > > +#ifdef ODP_CONFIG_USE_DMB<br> > > + /* Wait for loads (from ring slots) to complete. */<br> > > + __asm volatile("dmb ishld" ::: "memory");<br> > > + q->prod_read = new_read;<br> > > +#else<br> > > + __atomic_store_n(&q->prod_read, new_read, __ATOMIC_RELEASE);<br> > > +#endif<br> > > +#endif<br> > > + /* Prefetch events after we have release ring buffer slots */<br> > > + for (i = 0; i < actual; i++) {<br> > > + odp_event_t evt = *--evp;<br> > > +<br> > > + __builtin_prefetch(<br> > > + odp_buffer_addr(odp_buffer_from_event(evt)), 0, 0);<br> > > + }<br> > > + return actual;<br> > > +}<br> > > +<br> > > +inline int _odp_queue_deq(sched_elem_t *q, odp_event_t *evp, int num)<br> > > +{<br> > > + int actual;<br> > > + ringidx_t old_read;<br> > > + ringidx_t old_write;<br> > > + ringidx_t new_read;<br> > > + uint32_t mask;<br> > > + odp_buffer_hdr_t **ring;<br> > > + odp_buffer_hdr_t **p_buf_hdr;<br> > > +<br> > > + mask = q->cons_mask;<br> > > + ring = q->cons_ring;<br> > > +<br> > > + /* Load consumer ring state (read & write index) */<br> > > +#ifndef ODP_CONFIG_USE_LLSC<br> > > + old_read = __atomic_load_n(&q->cons_read, __ATOMIC_RELAXED);<br> > > +#endif<br> > > + do {<br> > > + /* Need __atomic_load to avoid compiler reordering<br> > > + * Producer does store-release cons_write, we need<br> > > + * load-acquire.<br> > > + */<br> > > + old_write = __atomic_load_n(&q->cons_write, __ATOMIC_ACQUIRE);<br> > > +#ifdef ODP_CONFIG_USE_LLSC<br> > > + old_read = ll32(&q->cons_read, __ATOMIC_RELAXED);<br> > > +#endif<br> > > + /* Prefetch ring buffer array */<br> > > + __builtin_prefetch(&q->cons_ring[old_read & mask], 0, 0);<br> > > +<br> > > + actual = MIN(num, (int)(old_write - old_read));<br> > > + if (odp_unlikely(actual <= 0))<br> > > + return 0;<br> > > +<br> > > + /* Attempt to free ring slot(s) */<br> > > + new_read = old_read + actual;<br> > > +#ifdef ODP_CONFIG_USE_LLSC<br> > > + } while (odp_unlikely(sc32(&q->cons_read, new_read, __ATOMIC_RELAXED)));<br> > > +#else<br> > > + } while (!__atomic_compare_exchange_n(&q->cons_read,<br> > > + &old_read, /* Updated on failure */<br> > > + new_read,<br> > > + true,<br> > > + __ATOMIC_RELAXED,<br> > > + __ATOMIC_RELAXED));<br> > > +#endif<br> > > +#ifdef ODP_CONFIG_USE_SPLIT_PRODCONS<br> > > + __builtin_prefetch(&q->prod_read, 0, 0);<br> > > +#endif<br> > > + p_buf_hdr = (odp_buffer_hdr_t **)evp;<br> > > + do {<br> > > + odp_buffer_hdr_t *p_buf;<br> > > +<br> > > + p_buf = ring[old_read & mask];<br> > > + __builtin_prefetch(<br> > > + odp_buffer_addr(<br> > > + odp_buffer_from_event((odp_event_t)p_buf)),<br> > > + 0, 0);<br> > > + *p_buf_hdr++ = p_buf;<br> > > + } while (++old_read != new_read);<br> > > + old_read -= actual;<br> > > +<br> > > +#ifdef ODP_CONFIG_USE_SPLIT_PRODCONS<br> > > + __builtin_prefetch(&q->node, 1, 0);<br> > > +#endif<br> > > + /* Wait for our turn to signal producers */<br> > > + if (odp_unlikely(__atomic_load_n(&q->prod_read, __ATOMIC_RELAXED) !=<br> > > + old_read)) {<br> > > + SEVL();<br> > > + while (WFE() && LDXR32(&q->prod_read,<br> > > + __ATOMIC_RELAXED) != old_read)<br> > > + DOZE();<br> > > + }<br> > > +<br> > > + /* Signal producers that empty slots are available<br> > > + * (release ring slots)<br> > > + * Enable other consumers to continue<br> > > + */<br> > > +#ifdef ODP_CONFIG_USE_DMB<br> > > + /* Wait for loads (from ring slots) to complete */<br> > > + __asm volatile("dmb ishld" ::: "memory");<br> > > + q->prod_read = new_read;<br> > > +#else<br> > > + __atomic_store_n(&q->prod_read, new_read, __ATOMIC_RELEASE);<br> > > +#endif<br> > > +<br> > > + return actual;<br> > > +}<br> > > +<br> > > +int queue_deq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[], int num)<br> > > +{<br> > > + sched_elem_t *q;<br> > > +<br> > > + q = &queue->s.sched_elem;<br> > > + return _odp_queue_deq(q, (odp_event_t *)buf_hdr, num);<br> > > +}<br> > > +<br> > > +odp_buffer_hdr_t *queue_deq(queue_entry_t *queue)<br> > > +{<br> > > + sched_elem_t *q;<br> > > + odp_buffer_hdr_t *buf_hdr;<br> > > +<br> > > + q = &queue->s.sched_elem;<br> > > + if (_odp_queue_deq(q, (odp_event_t *)&buf_hdr, 1) == 1)<br> > > + return buf_hdr;<br> > > + else<br> > > + return BUFFER_HDR_INVALID;<br> > > +}<br> > > +<br> > > +int odp_queue_deq_multi(odp_queue_t handle, odp_event_t events[], int num)<br> > > +{<br> > > + queue_entry_t *queue;<br> > > +<br> > > + queue = queue_to_qentry(handle);<br> > > + return queue->s.dequeue_multi(queue, (odp_buffer_hdr_t **)events, num);<br> > > +}<br> > > +<br> > > +odp_event_t odp_queue_deq(odp_queue_t handle)<br> > > +{<br> > > + queue_entry_t *queue;<br> > > +<br> > > + queue = queue_to_qentry(handle);<br> > > + return (odp_event_t)queue->s.dequeue(queue);<br> > > +}<br> > > +<br> > > +void odp_queue_param_init(odp_queue_param_t *params)<br> > > +{<br> > > + memset(params, 0, sizeof(odp_queue_param_t));<br> > > + params->type = ODP_QUEUE_TYPE_PLAIN;<br> > > + params->enq_mode = ODP_QUEUE_OP_MT;<br> > > + params->deq_mode = ODP_QUEUE_OP_MT;<br> > > + params->sched.prio = ODP_SCHED_PRIO_DEFAULT;<br> > > + params->sched.sync = ODP_SCHED_SYNC_PARALLEL;<br> > > + params->sched.group = ODP_SCHED_GROUP_ALL;<br> > > +}<br> > > +<br> > > +int odp_queue_info(odp_queue_t handle, odp_queue_info_t *info)<br> > > +{<br> > > + uint32_t queue_id;<br> > > + queue_entry_t *queue;<br> > > + int status;<br> > > +<br> > > + if (odp_unlikely(info == NULL)) {<br> > > + ODP_ERR("Unable to store info, NULL ptr given\n");<br> > > + return -1;<br> > > + }<br> > > +<br> > > + queue_id = queue_to_id(handle);<br> > > +<br> > > + if (odp_unlikely(queue_id >= ODP_CONFIG_QUEUES)) {<br> > > + ODP_ERR("Invalid queue handle:%" PRIu64 "\n",<br> > > + odp_queue_to_u64(handle));<br> > > + return -1;<br> > > + }<br> > > +<br> > > + queue = get_qentry(queue_id);<br> > > +<br> > > + LOCK(&queue->s.lock);<br> > > + status = queue->s.status;<br> > > +<br> > > + if (odp_unlikely(status == QUEUE_STATUS_FREE ||<br> > > + status == QUEUE_STATUS_DESTROYED)) {<br> > > + UNLOCK(&queue->s.lock);<br> > > + ODP_ERR("Invalid queue status:%d\n", status);<br> > > + return -1;<br> > > + }<br> > > +<br> > > + info->name = queue->s.name;<br> > > + info->param = queue->s.param;<br> > > +<br> > > + UNLOCK(&queue->s.lock);<br> > > +<br> > > + return 0;<br> > > +}<br> > > +<br> > > +uint64_t odp_queue_to_u64(odp_queue_t hdl)<br> > > +{<br> > > + return _odp_pri(hdl);<br> > > +}<br> > > diff --git a/platform/linux-generic/odp_schedule_if.c b/platform/linux-generic/odp_schedule_if.c<br> > > index a9ede98d..bf92333e 100644<br> > > --- a/platform/linux-generic/odp_schedule_if.c<br> > > +++ b/platform/linux-generic/odp_schedule_if.c<br> > > @@ -6,24 +6,38 @@<br> > ><br> > > #include <odp_schedule_if.h><br> > ><br> > > -extern const schedule_fn_t schedule_sp_fn;<br> > > -extern const schedule_api_t schedule_sp_api;<br> > > +#if defined(ODP_SCHEDULE_SCALABLE)<br> > ><br> > > -extern const schedule_fn_t schedule_default_fn;<br> > > -extern const schedule_api_t schedule_default_api;<br> > > +extern const schedule_fn_t schedule_scalable_fn;<br> > > +extern const schedule_api_t schedule_scalable_api;<br> > ><br> > > -extern const schedule_fn_t schedule_iquery_fn;<br> > > -extern const schedule_api_t schedule_iquery_api;<br> > > +const schedule_fn_t *sched_fn = &schedule_scalable_fn;<br> > > +const schedule_api_t *sched_api = &schedule_scalable_api;<br> > ><br> > > -#ifdef ODP_SCHEDULE_SP<br> > > -const schedule_fn_t *sched_fn = &schedule_sp_fn;<br> > > -const schedule_api_t *sched_api = &schedule_sp_api;<br> > > #elif defined(ODP_SCHEDULE_IQUERY)<br> > > -const schedule_fn_t *sched_fn = &schedule_iquery_fn;<br> > > +<br> > > +extern const schedule_fn_t schedule_iquery_fn;<br> > > +extern const schedule_api_t schedule_iquery_api;<br> > > +<br> > > +const schedule_fn_t *sched_fn = &schedule_iquery_fn;<br> > > const schedule_api_t *sched_api = &schedule_iquery_api;<br> > > -#else<br> > > +<br> > > +#elif defined(ODP_SCHEDULE_SP)<br> > > +<br> > > +extern const schedule_fn_t schedule_sp_fn;<br> > > +extern const schedule_api_t schedule_sp_api;<br> > > +<br> > > +const schedule_fn_t *sched_fn = &schedule_sp_fn;<br> > > +const schedule_api_t *sched_api = &schedule_sp_api;<br> > > +<br> > > +#else /* default scheduler */<br> > > +<br> > > +extern const schedule_fn_t schedule_default_fn;<br> > > +extern const schedule_api_t schedule_default_api;<br> > > +<br> > > const schedule_fn_t *sched_fn = &schedule_default_fn;<br> > > const schedule_api_t *sched_api = &schedule_default_api;<br> > > +<br> > > #endif<br> > ><br> > > uint64_t odp_schedule_wait_time(uint64_t ns)<br> > > diff --git a/platform/linux-generic/odp_schedule_scalable.c b/platform/linux-generic/odp_schedule_scalable.c<br> > > new file mode 100644<br> > > index 00000000..1e503f5e<br> > > --- /dev/null<br> > > +++ b/platform/linux-generic/odp_schedule_scalable.c<br> > > @@ -0,0 +1,1959 @@<br> > > +/* Copyright (c) 2017, ARM Limited<br> > > + * All rights reserved.<br> > > + *<br> > > + * SPDX-License-Identifier: BSD-3-Clause<br> > > + */<br> > > +<br> > > +#include <odp/api/align.h><br> > > +#include <odp/api/atomic.h><br> > > +#include <odp/api/cpu.h><br> > > +#include <odp/api/hints.h><br> > > +#include <odp/api/schedule.h><br> > > +#include <odp/api/shared_memory.h><br> > > +#include <odp/api/sync.h><br> > > +#include <odp/api/thread.h><br> > > +#include <odp/api/thrmask.h><br> > > +#include <odp/api/time.h><br> > > +<br> > > +#include <odp_internal.h><br> > > +#include <odp_config_internal.h><br> > > +#include <odp_debug_internal.h><br> > > +<br> > > +#include <odp_align_internal.h><br> > > +#include <odp_buffer_inlines.h><br> > > +#include <odp_llqueue.h><br> > > +#include <odp_queue_internal.h><br> > > +#include <odp_schedule_if.h><br> > > +#include <odp_llsc.h><br> > > +#include <odp_bitset.h><br> > > +#include <odp_atomic16.h><br> > > +#include <odp_packet_io_internal.h><br> > > +<br> > > +#include <limits.h><br> > > +#include <stdbool.h><br> > > +#include <string.h><br> > > +<br> > > +#include <odp/api/plat/ticketlock_inlines.h><br> > > +#define LOCK(a) _odp_ticketlock_lock((a))<br> > > +#define UNLOCK(a) _odp_ticketlock_unlock((a))<br> > > +<br> > > +#define TAG_EMPTY 0U<br> > > +#define TAG_USED (1U << 15)<br> > > +#define TAG_BUSY (1U << 31)<br> > > +#define PKTIO_QUEUE_2_TAG(p, q) ((p) << 16 | (q) | TAG_USED)<br> > > +#define TAG_2_PKTIO(t) (((t) >> 16) & 0x7FFF)<br> > > +#define TAG_2_QUEUE(t) ((t) & 0x7FFF)<br> > > +#define TAG_IS_READY(t) (((t) & (TAG_USED | TAG_BUSY)) == TAG_USED)<br> > > +#define PKTIN_MAX (ODP_CONFIG_PKTIO_ENTRIES * PKTIO_MAX_QUEUES)<br> > > +#define MAXTHREADS ATOM_BITSET_SIZE<br> > > +<br> > > +static uint32_t pktin_num;<br> > > +static uint32_t pktin_hi;<br> > > +static uint16_t pktin_count[ODP_CONFIG_PKTIO_ENTRIES];<br> > > +static uint32_t pktin_tags[PKTIN_MAX] ODP_ALIGNED_CACHE;<br> > > +<br> > > +#define __atomic_fetch_max(var, v, mo) do { \<br> > > + /* Evalulate 'v' once */ \<br> > > + __typeof__(v) tmp_v = (v); \<br> > > + __typeof__(*var) old_var = \<br> > > + __atomic_load_n((var), __ATOMIC_RELAXED); \<br> > > + while (tmp_v > old_var) { \<br> > > + /* Attempt to store 'v' in '*var' */ \<br> > > + if (__atomic_compare_exchange_n((var), &old_var, \<br> > > + tmp_v, true, (mo), \<br> > > + (mo))) \<br> > > + break; \<br> > > + /* Else failure, try again (with updated value of \<br> > > + * old_var). \<br> > > + */ \<br> > > + } \<br> > > + /* v <= old_var, nothing to do */ \<br> > > + } while (0)<br> > > +<br> > > +ODP_STATIC_ASSERT(ODP_SCHED_PRIO_LOWEST == (ODP_SCHED_PRIO_NUM - 1),<br> > > + "lowest_prio_does_not_match_with_num_prios");<br> > > +<br> > > +ODP_STATIC_ASSERT((ODP_SCHED_PRIO_NORMAL > 0) &&<br> > > + (ODP_SCHED_PRIO_NORMAL < (ODP_SCHED_PRIO_NUM - 1)),<br> > > + "normal_prio_is_not_between_highest_and_lowest");<br> > > +<br> > > +ODP_STATIC_ASSERT(IS_POWER_TWO(ODP_CONFIG_QUEUES),<br> > > + "Number_of_queues_is_not_power_of_two");<br> > > +<br> > > +/*<br> > > + * Scheduler group related variables.<br> > > + */<br> > > +/* Currently used scheduler groups */<br> > > +static sched_group_mask_t sg_free;<br> > > +static sched_group_t *sg_vec[MAX_SCHED_GROUP];<br> > > +/* Group lock for MT-safe APIs */<br> > > +odp_spinlock_t sched_grp_lock;<br> > > +<br> > > +#define SCHED_GROUP_JOIN 0<br> > > +#define SCHED_GROUP_LEAVE 1<br> > > +<br> > > +/*<br> > > + * Per thread state<br> > > + */<br> > > +static sched_scalable_thread_state_t thread_state[MAXTHREADS];<br> > > +__thread sched_scalable_thread_state_t *sched_ts;<br> > > +<br> > > +/*<br> > > + * Forward declarations.<br> > > + */<br> > > +static int thread_state_init(int tidx)<br> > > +{<br> > > + sched_scalable_thread_state_t *ts;<br> > > + uint32_t i;<br> > > +<br> > > + ODP_ASSERT(tidx < MAXTHREADS);<br> > > + ts = &thread_state[tidx];<br> > > + ts->atomq = NULL;<br> > > + ts->rctx = NULL;<br> > > + ts->pause = false;<br> > > + ts->out_of_order = false;<br> > > + ts->tidx = tidx;<br> > > + ts->dequeued = 0;<br> > > + ts->pktin_next = 0;<br> > > + ts->pktin_poll_cnts = 0;<br> > > + ts->ticket = TICKET_INVALID;<br> > > + ts->priv_rvec_free = 0;<br> > > + ts->rvec_free = (1ULL << TS_RVEC_SIZE) - 1;<br> > > + ts->num_schedq = 0;<br> > > + ts->sg_sem = 1; /* Start with sched group semaphore changed */<br> > > + memset(ts->sg_actual, 0, sizeof(ts->sg_actual));<br> > > + for (i = 0; i < TS_RVEC_SIZE; i++) {<br> > > + ts->rvec[i].rvec_free = &ts->rvec_free;<br> > > + ts->rvec[i].idx = i;<br> > > + }<br> > > + sched_ts = ts;<br> > > +<br> > > + return 0;<br> > > +}<br> > > +<br> > > +static void insert_schedq_in_list(sched_scalable_thread_state_t *ts,<br> > > + sched_queue_t *schedq)<br> > > +{<br> > > + /* Find slot for schedq */<br> > > + for (uint32_t i = 0; i < ts->num_schedq; i++) {<br> > > + /* Lower value is higher priority and closer to start of list */<br> > > + if (schedq->prio <= ts->schedq_list[i]->prio) {<br> > > + /* This is the slot! */<br> > > + sched_queue_t *tmp;<br> > > +<br> > > + tmp = ts->schedq_list[i];<br> > > + ts->schedq_list[i] = schedq;<br> > > + schedq = tmp;<br> > > + /* Continue the insertion procedure with the<br> > > + * new schedq.<br> > > + */<br> > > + }<br> > > + }<br> > > + if (ts->num_schedq == SCHEDQ_PER_THREAD)<br> > > + ODP_ABORT("Too many schedqs\n");<br> > > + ts->schedq_list[ts->num_schedq++] = schedq;<br> > > +}<br> > > +<br> > > +static void remove_schedq_from_list(sched_scalable_thread_state_t *ts,<br> > > + sched_queue_t *schedq)<br> > > +{<br> > > + /* Find schedq */<br> > > + for (uint32_t i = 0; i < ts->num_schedq; i++)<br> > > + if (ts->schedq_list[i] == schedq) {<br> > > + /* Move remaining schedqs */<br> > > + for (uint32_t j = i + 1; j < ts->num_schedq; j++)<br> > > + ts->schedq_list[j - 1] = ts->schedq_list[j];<br> > > + ts->num_schedq--;<br> > > + return;<br> > > + }<br> > > + ODP_ABORT("Cannot find schedq\n");<br> > > +}<br> > > +<br> > > +/*******************************************************************************<br> > > + * Scheduler queues<br> > > + ******************************************************************************/<br> > > +#ifndef odp_container_of<br> > > +#define odp_container_of(pointer, type, member) \<br> > > + ((type *)(void *)(((char *)pointer) - offsetof(type, member)))<br> > > +#endif<br> > > +<br> > > +static inline void schedq_init(sched_queue_t *schedq, uint32_t prio)<br> > > +{<br> > > + llqueue_init(&schedq->llq);<br> > > + schedq->prio = prio;<br> > > +}<br> > > +<br> > > +static inline sched_elem_t *schedq_peek(sched_queue_t *schedq)<br> > > +{<br> > > + struct llnode *ptr;<br> > > +<br> > > + ptr = llq_head(&schedq->llq);<br> > > + return odp_container_of(ptr, sched_elem_t, node);<br> > > +}<br> > > +<br> > > +static inline odp_bool_t schedq_cond_pop(sched_queue_t *schedq,<br> > > + sched_elem_t *elem)<br> > > +{<br> > > + return llq_dequeue_cond(&schedq->llq, &elem->node);<br> > > +}<br> > > +<br> > > +static inline void schedq_push(sched_queue_t *schedq, sched_elem_t *elem)<br> > > +{<br> > > + llq_enqueue(&schedq->llq, &elem->node);<br> > > +}<br> > > +<br> > > +static inline odp_bool_t schedq_cond_rotate(sched_queue_t *schedq,<br> > > + sched_elem_t *elem)<br> > > +{<br> > > + return llq_cond_rotate(&schedq->llq, &elem->node);<br> > > +}<br> > > +<br> > > +static inline bool schedq_elem_on_queue(sched_elem_t *elem)<br> > > +{<br> > > + return elem->node.next != NULL;<br> > > +}<br> > > +<br> > > +/*******************************************************************************<br> > > + * Shared metadata btwn scheduler and queue<br> > > + ******************************************************************************/<br> > > +<br> > > +void sched_update_enq(sched_elem_t *q, uint32_t actual)<br> > > +{<br> > > + union {<br> > > + qschedstate_t ss;<br> > > + uint64_t ui;<br> > > + } oss, nss;<br> > > + uint32_t ticket;<br> > > +<br> > > +#ifndef ODP_CONFIG_USE_LLSC<br> > > + oss.ss = q->qschst;<br> > > +#endif<br> > > + /* Update event counter, optionally taking a ticket. */<br> > > + do {<br> > > + ticket = TICKET_INVALID;<br> > > +#ifdef ODP_CONFIG_USE_LLSC<br> > > + oss.ui = ll64((uint64_t *)&q->qschst, __ATOMIC_RELAXED);<br> > > +#endif<br> > > + nss = oss;<br> > > + nss.ss.numevts += actual;<br> > > + if (odp_unlikely(oss.ss.numevts <= 0 && nss.ss.numevts > 0))<br> > > + /* E -> NE transition */<br> > > + if (q->qschst_type != ODP_SCHED_SYNC_ATOMIC ||<br> > > + oss.ss.cur_ticket == oss.ss.nxt_ticket)<br> > > + /* Parallel or ordered queues: always take<br> > > + * ticket.<br> > > + * Atomic queue: only take ticket if one is<br> > > + * immediately available.<br> > > + * Otherwise ticket already taken => queue<br> > > + * processed by some thread.<br> > > + */<br> > > + ticket = nss.ss.nxt_ticket++;<br> > > + /* Else queue already was non-empty. */<br> > > + /* Attempt to update numevts counter and optionally take ticket. */<br> > > +#ifdef ODP_CONFIG_USE_LLSC<br> > > + } while (odp_unlikely(<br> > > + sc64((uint64_t *)&q->qschst, nss.ui, __ATOMIC_RELAXED)));<br> > > +#else<br> > > + } while (!__atomic_compare_exchange(<br> > > + &q->qschst, &oss.ss, &nss.ss,<br> > > + true, __ATOMIC_RELAXED, __ATOMIC_RELAXED));<br> > > +#endif<br> > > +<br> > > + if (odp_unlikely(ticket != TICKET_INVALID)) {<br> > > + /* Wait for our turn to update schedq. */<br> > > + if (odp_unlikely(<br> > > + __atomic_load_n(&q->qschst.cur_ticket,<br> > > + __ATOMIC_ACQUIRE) != ticket)) {<br> > > + SEVL();<br> > > + while (WFE() &&<br> > > + LDXR8(&q->qschst.cur_ticket,<br> > > + __ATOMIC_ACQUIRE) != ticket)<br> > > + DOZE();<br> > > + }<br> > > + /* Enqueue at end of scheduler queue */<br> > > + /* We are here because of empty-to-non-empty transition<br> > > + * This means queue must be pushed to schedq if possible<br> > > + * but we can't do that if it already is on the schedq<br> > > + */<br> > > + if (odp_likely(!schedq_elem_on_queue(q) &&<br> > > + q->pop_deficit == 0)) {<br> > > + /* Queue not already on schedq and no pop deficit means<br> > > + * we can push queue to schedq */<br> > > + schedq_push(q->schedq, q);<br> > > + } else {<br> > > + /* Missed push => cancels one missed pop */<br> > > + q->pop_deficit--;<br> > > + }<br> > > +#ifdef ODP_CONFIG_USE_DMB<br> > > + __atomic_thread_fence(__ATOMIC_RELEASE);<br> > > + __atomic_store_n(&q->qschst.cur_ticket, ticket + 1,<br> > > + __ATOMIC_RELAXED);<br> > > +#else<br> > > + __atomic_store_n(&q->qschst.cur_ticket, ticket + 1,<br> > > + __ATOMIC_RELEASE);<br> > > +#endif<br> > > + }<br> > > + /* Else queue was not empty or atomic queue already busy. */<br> > > +}<br> > > +<br> > > +void sched_update_enq_sp(sched_elem_t *q, uint32_t actual)<br> > > +{<br> > > + qschedstate_t oss, nss;<br> > > + uint32_t ticket;<br> > > +<br> > > + oss = q->qschst;<br> > > + /* Update event counter, optionally taking a ticket. */<br> > > + ticket = TICKET_INVALID;<br> > > + nss = oss;<br> > > + nss.numevts += actual;<br> > > + if (odp_unlikely(oss.numevts <= 0 && nss.numevts > 0)) {<br> > > + /* E -> NE transition */<br> > > + if (q->qschst_type != ODP_SCHED_SYNC_ATOMIC ||<br> > > + oss.cur_ticket == oss.nxt_ticket) {<br> > > + /* Parallel or ordered queues: always take<br> > > + * ticket.<br> > > + * Atomic queue: only take ticket if one is<br> > > + * immediately available. Otherwise ticket already<br> > > + * taken => queue owned/processed by some thread<br> > > + */<br> > > + ticket = nss.nxt_ticket++;<br> > > + }<br> > > + }<br> > > + /* Else queue already was non-empty. */<br> > > + /* Attempt to update numevts counter and optionally take ticket. */<br> > > + q->qschst = nss;<br> > > +<br> > > + if (odp_unlikely(ticket != TICKET_INVALID)) {<br> > > + /* Enqueue at end of scheduler queue */<br> > > + /* We are here because of empty-to-non-empty transition<br> > > + * This means queue must be pushed to schedq if possible<br> > > + * but we can't do that if it already is on the schedq<br> > > + */<br> > > + if (odp_likely(!schedq_elem_on_queue(q) &&<br> > > + q->pop_deficit == 0)) {<br> > > + /* Queue not already on schedq and no pop deficit means<br> > > + * we can push queue to schedq */<br> > > + schedq_push(q->schedq, q);<br> > > + } else {<br> > > + /* Missed push => cancels one missed pop */<br> > > + q->pop_deficit--;<br> > > + }<br> > > + q->qschst.cur_ticket = ticket + 1;<br> > > + }<br> > > + /* Else queue was not empty or atomic queue already busy. */<br> > > +}<br> > > +<br> > > +/* The scheduler is the only entity that performs the dequeue from a queue. */<br> > > +static void sched_update_deq(sched_elem_t *q,<br> > > + uint32_t actual, bool atomic) __attribute__((always_inline));<br> > > +static inline void sched_update_deq(sched_elem_t *q,<br> > > + uint32_t actual, bool atomic)<br> > > +{<br> > > + union {<br> > > + qschedstate_t ss;<br> > > + uint64_t ui;<br> > > + } oss, nss;<br> > > + uint32_t ticket;<br> > > +<br> > > + if (atomic) {<br> > > + qschedstate_t oss, nss;<br> > > + bool pushed = false;<br> > > +<br> > > + /* We own this atomic queue, only we can dequeue from it and<br> > > + * thus decrease numevts. Other threads may enqueue and thus<br> > > + * increase numevts.<br> > > + * This means that numevts can't unexpectedly become 0 and<br> > > + * invalidate a push operation already performed<br> > > + */<br> > > + oss = q->qschst;<br> > > + do {<br> > > + ODP_ASSERT(oss.cur_ticket == sched_ts->ticket);<br> > > + nss = oss;<br> > > + nss.numevts -= actual;<br> > > + if (nss.numevts > 0 && !pushed) {<br> > > + schedq_push(q->schedq, q);<br> > > + pushed = true;<br> > > + }<br> > > + /* Attempt to release ticket expecting our view of<br> > > + * numevts to be correct<br> > > + * Unfortunately nxt_ticket will also be included in<br> > > + * the CAS operation<br> > > + */<br> > > + nss.cur_ticket = sched_ts->ticket + 1;<br> > > + } while (odp_unlikely(!__atomic_compare_exchange(<br> > > + &q->qschst,<br> > > + &oss, &nss,<br> > > + true,<br> > > + __ATOMIC_RELEASE,<br> > > + __ATOMIC_RELAXED)));<br> > > + return;<br> > > + }<br> > > +<br> > > +#ifndef ODP_CONFIG_USE_LLSC<br> > > + oss.ss = q->qschst;<br> > > +#endif<br> > > + do {<br> > > + ticket = TICKET_INVALID;<br> > > +#ifdef ODP_CONFIG_USE_LLSC<br> > > + oss.ui = ll64((uint64_t *)&q->qschst, __ATOMIC_RELAXED);<br> > > +#endif<br> > > + nss = oss;<br> > > + nss.ss.numevts -= actual;<br> > > + nss.ss.wrr_budget -= actual;<br> > > + if ((oss.ss.numevts > 0 && nss.ss.numevts <= 0) ||<br> > > + oss.ss.wrr_budget <= actual) {<br> > > + /* If we have emptied parallel/ordered queue or<br> > > + * exchausted its WRR budget, we need a ticket<br> > > + * for a later pop.<br> > > + */<br> > > + ticket = nss.ss.nxt_ticket++;<br> > > + /* Reset wrr_budget as we might also push the<br> > > + * queue to the schedq.<br> > > + */<br> > > + nss.ss.wrr_budget = CONFIG_WRR_WEIGHT;<br> > > + }<br> > > + /* Attempt to update numevts and optionally take ticket. */<br> > > +#ifdef ODP_CONFIG_USE_LLSC<br> > > + } while (odp_unlikely(<br> > > + sc64((uint64_t *)&q->qschst, nss.ui, __ATOMIC_RELAXED)));<br> > > +#else<br> > > + } while (!__atomic_compare_exchange(<br> > > + &q->qschst, &oss.ss, &nss.ss,<br> > > + true, __ATOMIC_RELAXED, __ATOMIC_RELAXED));<br> > > +#endif<br> > > +<br> > > + if (odp_unlikely(ticket != TICKET_INVALID)) {<br> > > + ODP_ASSERT(q->qschst_type != ODP_SCHED_SYNC_ATOMIC);<br> > > + /* Wait for our turn to update schedq. */<br> > > + if (odp_unlikely(<br> > > + __atomic_load_n(&q->qschst.cur_ticket,<br> > > + __ATOMIC_ACQUIRE) != ticket)) {<br> > > + SEVL();<br> > > + while (WFE() &&<br> > > + LDXR8(&q->qschst.cur_ticket,<br> > > + __ATOMIC_ACQUIRE) != ticket)<br> > > + DOZE();<br> > > + }<br> > > + /* We are here because of non-empty-to-empty transition or<br> > > + * WRR budget exhausted<br> > > + * This means the queue must be popped from the schedq, now or<br> > > + * later<br> > > + * If there was no NE->E transition but instead the WRR budget<br> > > + * was exhausted, the queue needs to be moved (popped and<br> > > + * pushed) to the tail of the schedq<br> > > + */<br> > > + if (oss.ss.numevts > 0 && nss.ss.numevts <= 0) {<br> > > + /* NE->E transition, need to pop */<br> > > + if (!schedq_elem_on_queue(q) ||<br> > > + !schedq_cond_pop(q->schedq, q)) {<br> > > + /* Queue not at head, failed to dequeue<br> > > + * Missed a pop.<br> > > + */<br> > > + q->pop_deficit++;<br> > > + }<br> > > + } else {<br> > > + /* WRR budget exhausted<br> > > + * Need to move queue to tail of schedq if possible<br> > > + */<br> > > + if (odp_likely(schedq_elem_on_queue(q))) {<br> > > + /* Queue is on schedq, try to move it to<br> > > + * the tail<br> > > + */<br> > > + (void)schedq_cond_rotate(q->schedq, q);<br> > > + }<br> > > + /* Else queue not on schedq or not at head of schedq<br> > > + * No pop => no push<br> > > + */<br> > > + }<br> > > +#ifdef ODP_CONFIG_USE_DMB<br> > > + __atomic_thread_fence(__ATOMIC_RELEASE);<br> > > + __atomic_store_n(&q->qschst.cur_ticket, ticket + 1,<br> > > + __ATOMIC_RELAXED);<br> > > +#else<br> > > + __atomic_store_n(&q->qschst.cur_ticket, ticket + 1,<br> > > + __ATOMIC_RELEASE);<br> > > +#endif<br> > > + }<br> > > +}<br> > > +<br> > > +static void sched_update_deq_sc(sched_elem_t *q,<br> > > + uint32_t actual, bool atomic) __attribute__((always_inline));<br> > > +static inline void sched_update_deq_sc(sched_elem_t *q,<br> > > + uint32_t actual, bool atomic)<br> > > +{<br> > > + qschedstate_t oss, nss;<br> > > + uint32_t ticket;<br> > > +<br> > > + if (atomic) {<br> > > + ODP_ASSERT(q->qschst.cur_ticket == sched_ts->ticket);<br> > > + ODP_ASSERT(q->qschst.cur_ticket != q->qschst.nxt_ticket);<br> > > + q->qschst.numevts -= actual;<br> > > + q->qschst.cur_ticket = sched_ts->ticket + 1;<br> > > + if (q->qschst.numevts > 0)<br> > > + schedq_push(q->schedq, q);<br> > > + return;<br> > > + }<br> > > +<br> > > + oss = q->qschst;<br> > > + ticket = TICKET_INVALID;<br> > > + nss = oss;<br> > > + nss.numevts -= actual;<br> > > + nss.wrr_budget -= actual;<br> > > + if ((oss.numevts > 0 && nss.numevts <= 0) || oss.wrr_budget <= actual) {<br> > > + /* If we emptied the queue or<br> > > + * if we have served the maximum number of events<br> > > + * then we need a ticket for a later pop.<br> > > + */<br> > > + ticket = nss.nxt_ticket++;<br> > > + /* Also reset wrr_budget as we might also push the<br> > > + * queue to the schedq.<br> > > + */<br> > > + nss.wrr_budget = CONFIG_WRR_WEIGHT;<br> > > + }<br> > > + q->qschst = nss;<br> > > +<br> > > + if (ticket != TICKET_INVALID) {<br> > > + if (oss.numevts > 0 && nss.numevts <= 0) {<br> > > + /* NE->E transition, need to pop */<br> > > + if (!schedq_elem_on_queue(q) ||<br> > > + !schedq_cond_pop(q->schedq, q)) {<br> > > + /* Queue not at head, failed to dequeue.<br> > > + * Missed a pop.<br> > > + */<br> > > + q->pop_deficit++;<br> > > + }<br> > > + } else {<br> > > + /* WRR budget exhausted<br> > > + * Need to move queue to tail of schedq if possible<br> > > + */<br> > > + if (odp_likely(schedq_elem_on_queue(q))) {<br> > > + /* Queue is on schedq, try to move it to<br> > > + * the tail<br> > > + */<br> > > + (void)schedq_cond_rotate(q->schedq, q);<br> > > + }<br> > > + /* Else queue not on schedq or not at head of schedq<br> > > + * No pop => no push<br> > > + */<br> > > + }<br> > > + q->qschst.cur_ticket = ticket + 1;<br> > > + }<br> > > +}<br> > > +<br> > > +static inline void sched_update_popd_sc(sched_elem_t *elem)<br> > > +{<br> > > + if (elem->pop_deficit != 0 &&<br> > > + schedq_elem_on_queue(elem) &&<br> > > + schedq_cond_pop(elem->schedq, elem))<br> > > + elem->pop_deficit--;<br> > > +}<br> > > +<br> > > +static inline void sched_update_popd(sched_elem_t *elem)<br> > > +{<br> > > + uint32_t ticket = __atomic_fetch_add(&elem->qschst.nxt_ticket,<br> > > + 1,<br> > > + __ATOMIC_RELAXED);<br> > > + if (odp_unlikely(__atomic_load_n(&elem->qschst.cur_ticket,<br> > > + __ATOMIC_ACQUIRE) != ticket)) {<br> > > + SEVL();<br> > > + while (WFE() && LDXR8(&elem->qschst.cur_ticket,<br> > > + __ATOMIC_ACQUIRE) != ticket)<br> > > + DOZE();<br> > > + }<br> > > + sched_update_popd_sc(elem);<br> > > +#ifdef ODP_CONFIG_USE_DMB<br> > > + __atomic_thread_fence(__ATOMIC_RELEASE);<br> > > + __atomic_store_n(&elem->qschst.cur_ticket,<br> > > + ticket + 1,<br> > > + __ATOMIC_RELAXED);<br> > > +#else<br> > > + __atomic_store_n(&elem->qschst.cur_ticket,<br> > > + ticket + 1,<br> > > + __ATOMIC_RELEASE);<br> > > +#endif<br> > > +}<br> > > +<br> > > +sched_queue_t *schedq_from_sched_group(odp_schedule_group_t grp, uint32_t prio)<br> > > +{<br> > > + uint32_t sgi;<br> > > + sched_group_t *sg;<br> > > + uint32_t x;<br> > > +<br> > > + ODP_ASSERT(grp >= 0 && grp <= (odp_schedule_group_t)MAX_SCHED_GROUP);<br> > > + ODP_ASSERT((sg_free & (1ULL << grp)) == 0);<br> > > + ODP_ASSERT(prio < ODP_SCHED_PRIO_NUM);<br> > > +<br> > > + sgi = grp;<br> > > + sg = sg_vec[sgi];<br> > > +<br> > > + /* Use xcount to spread queues over the xfactor schedq's<br> > > + * per priority.<br> > > + */<br> > > + x = __atomic_fetch_add(&sg->xcount[prio], 1, __ATOMIC_RELAXED);<br> > > + if (x == 0) {<br> > > + /* First ODP queue for this priority<br> > > + * Notify all threads in sg->thr_wanted that they<br> > > + * should join.<br> > > + */<br> > > + sched_group_mask_t thrds = sg->thr_wanted;<br> > > +<br> > > + while (!bitset_is_null(thrds)) {<br> > > + uint32_t thr;<br> > > +<br> > > + thr = bitset_ffs(thrds) - 1;<br> > > + thrds = bitset_clr(thrds, thr);<br> > > + /* Notify the thread about membership in this<br> > > + * group/priority.<br> > > + */<br> > > + atom_bitset_set(&thread_state[thr].sg_wanted[prio],<br> > > + sgi, __ATOMIC_RELEASE);<br> > > + __atomic_store_n(&thread_state[thr].sg_sem, 1,<br> > > + __ATOMIC_RELEASE);<br> > > + }<br> > > + }<br> > > + return &sg->schedq[prio * sg->xfactor + x % sg->xfactor];<br> > > +}<br> > > +<br> > > +void sched_group_xcount_dec(odp_schedule_group_t grp, uint32_t prio)<br> > > +{<br> > > + uint32_t sgi;<br> > > + sched_group_t *sg;<br> > > + uint32_t x;<br> > > +<br> > > + ODP_ASSERT(grp >= 0 && grp <= (odp_schedule_group_t)MAX_SCHED_GROUP);<br> > > + ODP_ASSERT((sg_free & (1ULL << grp)) == 0);<br> > > + ODP_ASSERT(prio < ODP_SCHED_PRIO_NUM);<br> > > +<br> > > + sgi = grp;<br> > > + sg = sg_vec[sgi];<br> > > + x = __atomic_sub_fetch(&sg->xcount[prio], 1, __ATOMIC_RELAXED);<br> > > +<br> > > + if (x == 0) {<br> > > + /* Last ODP queue for this priority<br> > > + * Notify all threads in sg->thr_wanted that they<br> > > + * should leave.<br> > > + */<br> > > + sched_group_mask_t thrds = sg->thr_wanted;<br> > > +<br> > > + while (!bitset_is_null(thrds)) {<br> > > + uint32_t thr;<br> > > +<br> > > + thr = bitset_ffs(thrds) - 1;<br> > > + thrds = bitset_clr(thrds, thr);<br> > > + /* Notify the thread about membership in this<br> > > + * group/priority.<br> > > + */<br> > > + atom_bitset_clr(&thread_state[thr].sg_wanted[prio],<br> > > + sgi, __ATOMIC_RELEASE);<br> > > + __atomic_store_n(&thread_state[thr].sg_sem, 1,<br> > > + __ATOMIC_RELEASE);<br> > > + }<br> > > + }<br> > > +}<br> > > +<br> > > +static void update_sg_membership(sched_scalable_thread_state_t *ts)<br> > > +{<br> > > + uint32_t p;<br> > > + sched_group_mask_t sg_wanted;<br> > > + sched_group_mask_t added;<br> > > + sched_group_mask_t removed;<br> > > + uint32_t sgi;<br> > > + sched_group_t *sg;<br> > > + uint32_t x;<br> > > +<br> > > + for (p = 0; p < ODP_SCHED_PRIO_NUM; p++) {<br> > > + sg_wanted = __atomic_load_n(&ts->sg_wanted[p],<br> > > + __ATOMIC_ACQUIRE);<br> > > + if (!bitset_is_eql(ts->sg_actual[p], sg_wanted)) {<br> > > + /* Our sched_group membership has changed */<br> > > + added = bitset_andn(sg_wanted, ts->sg_actual[p]);<br> > > + while (!bitset_is_null(added)) {<br> > > + sgi = bitset_ffs(added) - 1;<br> > > + sg = sg_vec[sgi];<br> > > + for (x = 0; x < sg->xfactor; x++) {<br> > > + /* Include our thread index to shift<br> > > + * (rotate) the order of schedq's<br> > > + */<br> > > + insert_schedq_in_list(ts,<br> > > + &sg->schedq[p * sg->xfactor +<br> > > + (x + ts->tidx) % sg->xfactor]);<br> > > + }<br> > > + atom_bitset_set(&sg->thr_actual[p], ts->tidx,<br> > > + __ATOMIC_RELAXED);<br> > > + added = bitset_clr(added, sgi);<br> > > + }<br> > > + removed = bitset_andn(ts->sg_actual[p], sg_wanted);<br> > > + while (!bitset_is_null(removed)) {<br> > > + sgi = bitset_ffs(removed) - 1;<br> > > + sg = sg_vec[sgi];<br> > > + for (x = 0; x < sg->xfactor; x++) {<br> > > + remove_schedq_from_list(ts,<br> > > + &sg->schedq[p * sg->xfactor + x]);<br> > > + }<br> > > + atom_bitset_clr(&sg->thr_actual[p], ts->tidx,<br> > > + __ATOMIC_RELAXED);<br> > > + removed = bitset_clr(removed, sgi);<br> > > + }<br> > > + ts->sg_actual[p] = sg_wanted;<br> > > + }<br> > > + }<br> > > +}<br> > > +<br> > > +/*******************************************************************************<br> > > + * Scheduler<br> > > + ******************************************************************************/<br> > > +<br> > > +static inline void _schedule_release_atomic(sched_scalable_thread_state_t *ts)<br> > > +{<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + sched_update_deq_sc(ts->atomq, ts->dequeued, true);<br> > > + ODP_ASSERT(ts->atomq->qschst.cur_ticket != ts->ticket);<br> > > + ODP_ASSERT(ts->atomq->qschst.cur_ticket ==<br> > > + ts->atomq->qschst.nxt_ticket);<br> > > +#else<br> > > + sched_update_deq(ts->atomq, ts->dequeued, true);<br> > > +#endif<br> > > + ts->atomq = NULL;<br> > > + ts->ticket = TICKET_INVALID;<br> > > +}<br> > > +<br> > > +static inline void _schedule_release_ordered(sched_scalable_thread_state_t *ts)<br> > > +{<br> > > + ts->out_of_order = false;<br> > > + rctx_release(ts->rctx);<br> > > + ts->rctx = NULL;<br> > > +}<br> > > +<br> > > +static void pktin_poll(sched_scalable_thread_state_t *ts)<br> > > +{<br> > > + uint32_t i, tag, hi, npolls = 0;<br> > > + int pktio_index, queue_index;<br> > > +<br> > > + hi = __atomic_load_n(&pktin_hi, __ATOMIC_RELAXED);<br> > > + if (hi == 0)<br> > > + return;<br> > > +<br> > > + for (i = ts->pktin_next; npolls != hi; i = (i + 1) % hi, npolls++) {<br> > > + tag = __atomic_load_n(&pktin_tags[i], __ATOMIC_RELAXED);<br> > > + if (!TAG_IS_READY(tag))<br> > > + continue;<br> > > + if (!__atomic_compare_exchange_n(&pktin_tags[i], &tag,<br> > > + tag | TAG_BUSY,<br> > > + true,<br> > > + __ATOMIC_ACQUIRE,<br> > > + __ATOMIC_RELAXED))<br> > > + continue;<br> > > + /* Tag grabbed */<br> > > + pktio_index = TAG_2_PKTIO(tag);<br> > > + queue_index = TAG_2_QUEUE(tag);<br> > > + if (odp_unlikely(sched_cb_pktin_poll(pktio_index,<br> > > + 1, &queue_index))) {<br> > > + /* Pktio stopped or closed<br> > > + * Remove tag from pktin_tags<br> > > + */<br> > > + __atomic_store_n(&pktin_tags[i],<br> > > + TAG_EMPTY, __ATOMIC_RELAXED);<br> > > + __atomic_fetch_sub(&pktin_num,<br> > > + 1, __ATOMIC_RELEASE);<br> > > + /* Call stop_finalize when all queues<br> > > + * of the pktio have been removed<br> > > + */<br> > > + if (__atomic_sub_fetch(&pktin_count[pktio_index], 1,<br> > > + __ATOMIC_RELAXED) == 0)<br> > > + sched_cb_pktio_stop_finalize(pktio_index);<br> > > + } else {<br> > > + /* We don't know whether any packets were found and enqueued<br> > > + * Write back original tag value to release pktin queue<br> > > + */<br> > > + __atomic_store_n(&pktin_tags[i], tag, __ATOMIC_RELAXED);<br> > > + /* Do not iterate through all pktin queues every time */<br> > > + if ((ts->pktin_poll_cnts & 0xf) != 0)<br> > > + break;<br> > > + }<br> > > + }<br> > > + ODP_ASSERT(i < hi);<br> > > + ts->pktin_poll_cnts++;<br> > > + ts->pktin_next = i;<br> > > +}<br> > > +<br> > > +static int _schedule(odp_queue_t *from, odp_event_t ev[], int num_evts)<br> > > +{<br> > > + sched_scalable_thread_state_t *ts;<br> > > + sched_elem_t *atomq;<br> > > + int num;<br> > > + uint32_t i;<br> > > +<br> > > + ts = sched_ts;<br> > > + atomq = ts->atomq;<br> > > +<br> > > + /* Once an atomic queue has been scheduled to a thread, it will stay<br> > > + * on that thread until empty or 'rotated' by WRR<br> > > + */<br> > > + if (atomq != NULL) {<br> > > + ODP_ASSERT(ts->ticket != TICKET_INVALID);<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + LOCK(&atomq->qschlock);<br> > > +#endif<br> > > +dequeue_atomic:<br> > > + ODP_ASSERT(ts->ticket == atomq->qschst.cur_ticket);<br> > > + ODP_ASSERT(ts->ticket != atomq->qschst.nxt_ticket);<br> > > + /* Atomic queues can be dequeued without lock since this thread<br> > > + * has the only reference to the atomic queue being processed.<br> > > + */<br> > > + if (ts->dequeued < atomq->qschst.wrr_budget) {<br> > > + num = _odp_queue_deq_sc(atomq, ev, num_evts);<br> > > + if (odp_likely(num != 0)) {<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + UNLOCK(&atomq->qschlock);<br> > > +#endif<br> > > + ts->dequeued += num;<br> > > + /* Allow this thread to continue to 'own' this<br> > > + * atomic queue until all events have been<br> > > + * processed and the thread re-invokes the<br> > > + * scheduler.<br> > > + */<br> > > + if (from)<br> > > + *from = queue_get_handle(<br> > > + (queue_entry_t *)atomq);<br> > > + return num;<br> > > + }<br> > > + }<br> > > + /* Atomic queue was empty or interrupted by WRR, release it. */<br> > > + _schedule_release_atomic(ts);<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + UNLOCK(&atomq->qschlock);<br> > > +#endif<br> > > + }<br> > > +<br> > > + /* Release any previous reorder context. */<br> > > + if (ts->rctx != NULL)<br> > > + _schedule_release_ordered(ts);<br> > > +<br> > > + /* Check for and perform any scheduler group updates. */<br> > > + if (odp_unlikely(__atomic_load_n(&ts->sg_sem, __ATOMIC_RELAXED) != 0)) {<br> > > + (void)__atomic_load_n(&ts->sg_sem, __ATOMIC_ACQUIRE);<br> > > + ts->sg_sem = 0;<br> > > + update_sg_membership(ts);<br> > > + }<br> > > +<br> > > + /* Scan our schedq list from beginning to end */<br> > > + for (i = 0; i < ts->num_schedq; i++) {<br> > > + sched_queue_t *schedq = ts->schedq_list[i];<br> > > + sched_elem_t *elem;<br> > > +restart_same:<br> > > + elem = schedq_peek(schedq);<br> > > + if (odp_unlikely(elem == NULL)) {<br> > > + /* Schedq empty, look at next one. */<br> > > + continue;<br> > > + }<br> > > +<br> > > + if (elem->cons_type == ODP_SCHED_SYNC_ATOMIC) {<br> > > + /* Dequeue element only if it is still at head<br> > > + * of schedq.<br> > > + */<br> > > + if (odp_unlikely(!schedq_cond_pop(schedq, elem))) {<br> > > + /* Queue not at head of schedq anymore, some<br> > > + * other thread popped it.<br> > > + */<br> > > + goto restart_same;<br> > > + }<br> > > + ts->atomq = atomq = elem;<br> > > + ts->dequeued = 0;<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + LOCK(&atomq->qschlock);<br> > > + ts->ticket = atomq->qschst.nxt_ticket++;<br> > > + ODP_ASSERT(atomq->qschst.cur_ticket == ts->ticket);<br> > > +#else<br> > > + /* Dequeued atomic queue from the schedq, only we<br> > > + * can process it and any qschst updates are our<br> > > + * responsibility.<br> > > + */<br> > > + /* The ticket taken below will signal producers */<br> > > + ts->ticket = __atomic_fetch_add(<br> > > + &atomq->qschst.nxt_ticket, 1, __ATOMIC_RELAXED);<br> > > + while (__atomic_load_n(<br> > > + &atomq->qschst.cur_ticket,<br> > > + __ATOMIC_ACQUIRE) != ts->ticket) {<br> > > + /* No need to use WFE, spinning here seems<br> > > + * very infrequent.<br> > > + */<br> > > + odp_cpu_pause();<br> > > + }<br> > > +#endif<br> > > + goto dequeue_atomic;<br> > > + } else if (elem->cons_type == ODP_SCHED_SYNC_PARALLEL) {<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + LOCK(&elem->qschlock);<br> > > + num = _odp_queue_deq_sc(elem, ev, num_evts);<br> > > + if (odp_likely(num != 0)) {<br> > > + sched_update_deq_sc(elem, num, false);<br> > > + UNLOCK(&elem->qschlock);<br> > > + if (from)<br> > > + *from =<br> > > + queue_get_handle((queue_entry_t *)elem);<br> > > + return num;<br> > > + }<br> > > + UNLOCK(&elem->qschlock);<br> > > +#else<br> > > + num = _odp_queue_deq(elem, ev, num_evts);<br> > > + if (odp_likely(num != 0)) {<br> > > + sched_update_deq(elem, num, false);<br> > > + if (from)<br> > > + *from =<br> > > + queue_get_handle((queue_entry_t *)elem);<br> > > + return num;<br> > > + }<br> > > +#endif<br> > > + } else if (elem->cons_type == ODP_SCHED_SYNC_ORDERED) {<br> > > + reorder_window_t *rwin;<br> > > + reorder_context_t *rctx;<br> > > + uint32_t sn;<br> > > + uint32_t idx;<br> > > + int ret;<br> > > +<br> > > + /* The ordered queue has a reorder window so requires<br> > > + * order restoration. We must use a reorder context to<br> > > + * collect all outgoing events. Ensure there is at least<br> > > + * one available reorder context.<br> > > + */<br> > > + if (odp_unlikely(bitset_is_null(ts->priv_rvec_free))) {<br> > > + ts->priv_rvec_free = atom_bitset_xchg(<br> > > + &ts->rvec_free, 0,<br> > > + __ATOMIC_RELAXED);<br> > > + if (odp_unlikely(bitset_is_null(<br> > > + ts->priv_rvec_free))) {<br> > > + /* No free reorder contexts for<br> > > + * this thread. Look at next schedq,<br> > > + * hope we find non-ordered queue.<br> > > + */<br> > > + continue;<br> > > + }<br> > > + }<br> > > + /* rwin_reserve and odp_queue_deq must be atomic or<br> > > + * there will be a potential race condition.<br> > > + * Allocate a slot in the reorder window.<br> > > + */<br> > > + rwin = queue_get_rwin((queue_entry_t *)elem);<br> > > + ODP_ASSERT(rwin != NULL);<br> > > + if (odp_unlikely(!rwin_reserve(rwin, &sn))) {<br> > > + /* Reorder window full */<br> > > + /* Look at next schedq, find other queue */<br> > > + continue;<br> > > + }<br> > > + /* Wait for our turn to dequeue */<br> > > + if (odp_unlikely(__atomic_load_n(&rwin->turn,<br> > > + __ATOMIC_ACQUIRE) != sn)) {<br> > > + SEVL();<br> > > + while (WFE() &&<br> > > + LDXR32(&rwin->turn, __ATOMIC_ACQUIRE)<br> > > + != sn)<br> > > + DOZE();<br> > > + }<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + LOCK(&elem->qschlock);<br> > > +#endif<br> > > + ret = _odp_queue_deq_sc(elem, ev, num_evts);<br> > > +#ifdef ODP_CONFIG_USE_DMB<br> > > + /* Wait for prod_read write in _odp_queue_dequeue_sc()<br> > > + * to complete.<br> > > + */<br> > > + __asm volatile("dmb ishst" ::: "memory");<br> > > + /* before we signal the next consumer */<br> > > + rwin->turn = sn + 1;<br> > > +#else<br> > > + __atomic_store_n(&rwin->turn, sn + 1, __ATOMIC_RELEASE);<br> > > +#endif<br> > > + /* Find and initialise an unused reorder context. */<br> > > + idx = bitset_ffs(ts->priv_rvec_free) - 1;<br> > > + ts->priv_rvec_free =<br> > > + bitset_clr(ts->priv_rvec_free, idx);<br> > > + rctx = &ts->rvec[idx];<br> > > + /* Need to initialise reorder context or we can't<br> > > + * release it later.<br> > > + */<br> > > + rctx_init(rctx, idx, rwin, sn);<br> > > +<br> > > + /* Was dequeue successful? */<br> > > + if (odp_likely(ret != 0)) {<br> > > + /* Perform scheduler related updates */<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + sched_update_deq_sc(elem, ret,<br> > > + /*atomic=*/false);<br> > > + UNLOCK(&elem->qschlock);<br> > > +#else<br> > > + sched_update_deq(elem, ret, /*atomic=*/false);<br> > > +#endif<br> > > +<br> > > + /* Are we in-order or out-of-order? */<br> > > + ts->out_of_order = sn != rwin->hc.head;<br> > > +<br> > > + ts->rctx = rctx;<br> > > + if (from)<br> > > + *from = queue_get_handle(<br> > > + (queue_entry_t *)elem);<br> > > + return ret;<br> > > + }<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + UNLOCK(&elem->qschlock);<br> > > +#endif<br> > > + /* Since a slot was reserved in the reorder window,<br> > > + * the reorder context needs to be released and<br> > > + * inserted into the reorder window.<br> > > + */<br> > > + rctx_release(rctx);<br> > > + ODP_ASSERT(ts->rctx == NULL);<br> > > + }<br> > > + /* Dequeue from parallel/ordered queue failed<br> > > + * Check if we have a queue at the head of the schedq that needs<br> > > + * to be popped<br> > > + */<br> > > + if (odp_unlikely(__atomic_load_n(&elem->pop_deficit,<br> > > + __ATOMIC_RELAXED) != 0)) {<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + LOCK(&elem->qschlock);<br> > > + sched_update_popd_sc(elem);<br> > > + UNLOCK(&elem->qschlock);<br> > > +#else<br> > > + sched_update_popd(elem);<br> > > +#endif<br> > > + }<br> > > + }<br> > > +<br> > > + pktin_poll(ts);<br> > > + return 0;<br> > > +}<br> > > +<br> > > +/******************************************************************************/<br> > > +<br> > > +static void schedule_order_lock(unsigned lock_index)<br> > > +{<br> > > + struct reorder_context *rctx = sched_ts->rctx;<br> > > +<br> > > + if (odp_unlikely(rctx == NULL ||<br> > > + rctx->rwin == NULL ||<br> > > + lock_index >= rctx->rwin->lock_count)) {<br> > > + ODP_ERR("Invalid call to odp_schedule_order_lock\n");<br> > > + return;<br> > > + }<br> > > + if (odp_unlikely(__atomic_load_n(&rctx->rwin->olock[lock_index],<br> > > + __ATOMIC_ACQUIRE) != rctx->sn)) {<br> > > + SEVL();<br> > > + while (WFE() &&<br> > > + LDXR32(&rctx->rwin->olock[lock_index],<br> > > + __ATOMIC_ACQUIRE) != rctx->sn)<br> > > + DOZE();<br> > > + }<br> > > +}<br> > > +<br> > > +static void schedule_order_unlock(unsigned lock_index)<br> > > +{<br> > > + struct reorder_context *rctx;<br> > > +<br> > > + rctx = sched_ts->rctx;<br> > > + if (odp_unlikely(rctx == NULL ||<br> > > + rctx->rwin == NULL ||<br> > > + lock_index >= rctx->rwin->lock_count ||<br> > > + rctx->rwin->olock[lock_index] != rctx->sn)) {<br> > > + ODP_ERR("Invalid call to odp_schedule_order_unlock\n");<br> > > + return;<br> > > + }<br> > > +#ifdef ODP_CONFIG_USE_DMB<br> > > + __atomic_thread_fence(__ATOMIC_RELEASE);<br> > > + __atomic_store_n(&rctx->rwin->olock[lock_index],<br> > > + rctx->sn + 1,<br> > > + __ATOMIC_RELAXED);<br> > > +#else<br> > > + __atomic_store_n(&rctx->rwin->olock[lock_index],<br> > > + rctx->sn + 1,<br> > > + __ATOMIC_RELEASE);<br> > > +#endif<br> > > + rctx->olock_flags |= 1U << lock_index;<br> > > +}<br> > > +<br> > > +static void schedule_release_atomic(void)<br> > > +{<br> > > + sched_scalable_thread_state_t *ts;<br> > > +<br> > > + ts = sched_ts;<br> > > + if (odp_likely(ts->atomq != NULL)) {<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + sched_elem_t *atomq;<br> > > +<br> > > + atomq = ts->atomq;<br> > > + LOCK(&atomq->qschlock);<br> > > +#endif<br> > > + _schedule_release_atomic(ts);<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + UNLOCK(&atomq->qschlock);<br> > > +#endif<br> > > + }<br> > > +}<br> > > +<br> > > +static void schedule_release_ordered(void)<br> > > +{<br> > > + sched_scalable_thread_state_t *ts;<br> > > +<br> > > + ts = sched_ts;<br> > > + if (ts->rctx != NULL)<br> > > + _schedule_release_ordered(ts);<br> > > +}<br> > > +<br> > > +static int schedule_multi(odp_queue_t *from, uint64_t wait, odp_event_t ev[],<br> > > + int num)<br> > > +{<br> > > + sched_scalable_thread_state_t *ts;<br> > > + int n;<br> > > + odp_time_t start;<br> > > + odp_time_t delta;<br> > > + odp_time_t deadline;<br> > > +<br> > > + ts = sched_ts;<br> > > + if (odp_unlikely(ts->pause)) {<br> > > + if (ts->atomq != NULL) {<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + sched_elem_t *atomq;<br> > > +<br> > > + atomq = ts->atomq;<br> > > + LOCK(&atomq->qschlock);<br> > > +#endif<br> > > + _schedule_release_atomic(ts);<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + UNLOCK(&atomq->qschlock);<br> > > +#endif<br> > > + } else if (ts->rctx != NULL) {<br> > > + _schedule_release_ordered(ts);<br> > > + }<br> > > + return 0;<br> > > + }<br> > > +<br> > > + if (wait == ODP_SCHED_NO_WAIT)<br> > > + return _schedule(from, ev, num);<br> > > +<br> > > + if (wait == ODP_SCHED_WAIT) {<br> > > + for (;;) {<br> > > + n = _schedule(from, ev, num);<br> > > + if (odp_likely(n > 0))<br> > > + return n;<br> > > + }<br> > > + }<br> > > +<br> > > + start = odp_time_local();<br> > > +<br> > > + n = _schedule(from, ev, num);<br> > > + if (odp_likely(n > 0))<br> > > + return n;<br> > > +<br> > > + delta = odp_time_local_from_ns(wait);<br> > > + deadline = odp_time_sum(start, delta);<br> > > +<br> > > + while (odp_time_cmp(deadline, odp_time_local()) > 0) {<br> > > + n = _schedule(from, ev, num);<br> > > + if (odp_likely(n > 0))<br> > > + return n;<br> > > + }<br> > > +<br> > > + return 0;<br> > > +}<br> > > +<br> > > +static odp_event_t schedule(odp_queue_t *from, uint64_t wait)<br> > > +{<br> > > + odp_event_t ev = ODP_EVENT_INVALID;<br> > > + const int num = 1;<br> > > + sched_scalable_thread_state_t *ts;<br> > > + int n;<br> > > + odp_time_t start;<br> > > + odp_time_t delta;<br> > > + odp_time_t deadline;<br> > > +<br> > > + ts = sched_ts;<br> > > + if (odp_unlikely(ts->pause)) {<br> > > + if (ts->atomq != NULL) {<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + sched_elem_t *atomq;<br> > > +<br> > > + atomq = ts->atomq;<br> > > + LOCK(&atomq->qschlock);<br> > > +#endif<br> > > + _schedule_release_atomic(ts);<br> > > +#ifdef CONFIG_QSCHST_LOCK<br> > > + UNLOCK(&atomq->qschlock);<br> > > +#endif<br> > > + } else if (ts->rctx != NULL) {<br> > > + _schedule_release_ordered(ts);<br> > > + }<br> > > + return ev;<br> > > + }<br> > > +<br> > > + if (wait == ODP_SCHED_NO_WAIT) {<br> > > + (void)_schedule(from, &ev, num);<br> > > + return ev;<br> > > + }<br> > > +<br> > > + if (wait == ODP_SCHED_WAIT) {<br> > > + for (;;) {<br> > > + n = _schedule(from, &ev, num);<br> > > + if (odp_likely(n > 0))<br> > > + return ev;<br> > > + }<br> > > + }<br> > > +<br> > > + start = odp_time_local();<br> > > +<br> > > + n = _schedule(from, &ev, num);<br> > > + if (odp_likely(n > 0))<br> > > + return ev;<br> > > +<br> > > + delta = odp_time_local_from_ns(wait);<br> > > + deadline = odp_time_sum(start, delta);<br> > > +<br> > > + while (odp_time_cmp(deadline, odp_time_local()) > 0) {<br> > > + n = _schedule(from, &ev, num);<br> > > + if (odp_likely(n > 0))<br> > > + return ev;<br> > > + }<br> > > +<br> > > + return ev;<br> > > +}<br> > > +<br> > > +static void schedule_pause(void)<br> > > +{<br> > > + sched_ts->pause = true;<br> > > +}<br> > > +<br> > > +static void schedule_resume(void)<br> > > +{<br> > > + sched_ts->pause = false;<br> > > +}<br> > > +<br> > > +static uint64_t schedule_wait_time(uint64_t ns)<br> > > +{<br> > > + return ns;<br> > > +}<br> > > +<br> > > +static int schedule_num_prio(void)<br> > > +{<br> > > + return ODP_SCHED_PRIO_NUM;<br> > > +}<br> > > +<br> > > +static int schedule_group_update(sched_group_t *sg,<br> > > + uint32_t sgi,<br> > > + const odp_thrmask_t *mask,<br> > > + int join_leave)<br> > > +{<br> > > + int thr;<br> > > + uint32_t p;<br> > > +<br> > > + /* Internal function, do not validate inputs */<br> > > +<br> > > + /* Notify relevant threads about the change */<br> > > + thr = odp_thrmask_first(mask);<br> > > + while (0 <= thr) {<br> > > + /* Add thread to scheduler group's wanted thread mask */<br> > > + if (join_leave == SCHED_GROUP_JOIN)<br> > > + atom_bitset_set(&sg->thr_wanted, thr, __ATOMIC_RELAXED);<br> > > + else<br> > > + atom_bitset_clr(&sg->thr_wanted, thr, __ATOMIC_RELAXED);<br> > > + for (p = 0; p < ODP_SCHED_PRIO_NUM; p++) {<br> > > + if (sg->xcount[p] != 0) {<br> > > + /* This priority level has ODP queues<br> > > + * Notify the thread about membership in<br> > > + * this group/priority<br> > > + */<br> > > + if (join_leave == SCHED_GROUP_JOIN)<br> > > + atom_bitset_set(<br> > > + &thread_state[thr].sg_wanted[p],<br> > > + sgi,<br> > > + __ATOMIC_RELEASE);<br> > > + else<br> > > + atom_bitset_clr(<br> > > + &thread_state[thr].sg_wanted[p],<br> > > + sgi,<br> > > + __ATOMIC_RELEASE);<br> > > + __atomic_store_n(&thread_state[thr].sg_sem,<br> > > + 1,<br> > > + __ATOMIC_RELEASE);<br> > > + }<br> > > + }<br> > > + thr = odp_thrmask_next(mask, thr);<br> > > + }<br> > > +<br> > > + return 0;<br> > > +}<br> > > +<br> > > +static int _schedule_group_thrmask(sched_group_t *sg, odp_thrmask_t *mask)<br> > > +{<br> > > + bitset_t bs;<br> > > + uint32_t bit;<br> > > +<br> > > + /* Internal function, do not validate inputs */<br> > > +<br> > > + odp_thrmask_zero(mask);<br> > > + bs = sg->thr_wanted;<br> > > + while (!bitset_is_null(bs)) {<br> > > + bit = bitset_ffs(bs) - 1;<br> > > + bs = bitset_clr(bs, bit);<br> > > + odp_thrmask_set(mask, bit);<br> > > + }<br> > > +<br> > > + return 0;<br> > > +}<br> > > +<br> > > +static odp_schedule_group_t schedule_group_create(const char *name,<br> > > + const odp_thrmask_t *mask)<br> > > +{<br> > > + odp_shm_t shm;<br> > > + uint32_t sgi;<br> > > + sched_group_mask_t free;<br> > > + uint32_t xfactor;<br> > > + sched_group_t *sg;<br> > > + uint32_t p;<br> > > + uint32_t x;<br> > > +<br> > > + /* Validate inputs */<br> > > + if (mask == NULL)<br> > > + ODP_ABORT("mask is NULL\n");<br> > > +<br> > > + odp_spinlock_lock(&sched_grp_lock);<br> > > +<br> > > + /* Allocate a scheduler group */<br> > > + free = __atomic_load_n(&sg_free, __ATOMIC_RELAXED);<br> > > + do {<br> > > + /* All sched_groups in use */<br> > > + if (bitset_is_null(free))<br> > > + goto no_free_sched_group;<br> > > +<br> > > + sgi = bitset_ffs(free) - 1;<br> > > + /* All sched_groups in use */<br> > > + if (sgi >= MAX_SCHED_GROUP)<br> > > + goto no_free_sched_group;<br> > > + } while (!__atomic_compare_exchange_n(&sg_free,<br> > > + &free,<br> > > + bitset_clr(free, sgi),<br> > > + true,<br> > > + __ATOMIC_ACQUIRE,<br> > > + __ATOMIC_ACQUIRE));<br> > > +<br> > > + /* Compute xfactor (spread factor) from the number of threads<br> > > + * present in the thread mask. Preferable this would be an<br> > > + * explicit parameter.<br> > > + */<br> > > + xfactor = odp_thrmask_count(mask);<br> > > + if (xfactor < 1)<br> > > + xfactor = 1;<br> > > +<br> > > + shm = odp_shm_reserve(name ? name : "",<br> > > + (sizeof(sched_group_t) +<br> > > + (ODP_SCHED_PRIO_NUM * xfactor - 1) *<br> > > + sizeof(sched_queue_t)),<br> > > + ODP_CACHE_LINE_SIZE,<br> > > + ODP_SHM_PROC);<br> > > + if (ODP_SHM_INVALID == shm)<br> > > + goto shm_reserve_failed;<br> > > +<br> > > + sg = (sched_group_t *) odp_shm_addr(shm);<br> > > + if (sg == NULL)<br> > > + goto shm_addr_failed;<br> > > +<br> > > + strncpy(sg->name, name ? name : "", ODP_SCHED_GROUP_NAME_LEN - 1);<br> > > + sg->shm = shm;<br> > > + sg_vec[sgi] = sg;<br> > > + memset(sg->thr_actual, 0, sizeof(sg->thr_actual));<br> > > + sg->thr_wanted = bitset_null();<br> > > + sg->xfactor = xfactor;<br> > > + for (p = 0; p < ODP_SCHED_PRIO_NUM; p++) {<br> > > + sg->xcount[p] = 0;<br> > > + for (x = 0; x < xfactor; x++)<br> > > + schedq_init(&sg->schedq[p * xfactor + x], p);<br> > > + }<br> > > + if (odp_thrmask_count(mask) != 0)<br> > > + schedule_group_update(sg, sgi, mask, SCHED_GROUP_JOIN);<br> > > +<br> > > + odp_spinlock_unlock(&sched_grp_lock);<br> > > +<br> > > + return (odp_schedule_group_t) (sgi);<br> > > +<br> > > +shm_addr_failed:<br> > > + odp_shm_free(shm);<br> > > +<br> > > +shm_reserve_failed:<br> > > + /* Free the allocated group index */<br> > > + atom_bitset_set(&sg_free, sgi, __ATOMIC_RELAXED);<br> > > +<br> > > +no_free_sched_group:<br> > > + odp_spinlock_unlock(&sched_grp_lock);<br> > > +<br> > > + return ODP_SCHED_GROUP_INVALID;<br> > > +}<br> > > +<br> > > +static int schedule_group_destroy(odp_schedule_group_t group)<br> > > +{<br> > > + uint32_t sgi;<br> > > + sched_group_t *sg;<br> > > + uint32_t p;<br> > > + int ret = 0;<br> > > +<br> > > + /* Validate inputs */<br> > > + if (group < 0 && group >= (odp_schedule_group_t)MAX_SCHED_GROUP) {<br> > > + ret = -1;<br> > > + goto invalid_group;<br> > > + }<br> > > +<br> > > + if (sched_ts &&<br> > > + odp_unlikely(__atomic_load_n(&sched_ts->sg_sem,<br> > > + __ATOMIC_RELAXED) != 0)) {<br> > > + (void)__atomic_load_n(&sched_ts->sg_sem,<br> > > + __ATOMIC_ACQUIRE);<br> > > + sched_ts->sg_sem = 0;<br> > > + update_sg_membership(sched_ts);<br> > > + }<br> > > + odp_spinlock_lock(&sched_grp_lock);<br> > > +<br> > > + sgi = (uint32_t)group;<br> > > + if (bitset_is_set(sg_free, sgi)) {<br> > > + ret = -1;<br> > > + goto group_not_found;<br> > > + }<br> > > +<br> > > + sg = sg_vec[sgi];<br> > > + /* First ensure all threads have processed group_join/group_leave<br> > > + * requests.<br> > > + */<br> > > + for (p = 0; p < ODP_SCHED_PRIO_NUM; p++) {<br> > > + if (sg->xcount[p] != 0) {<br> > > + bitset_t wanted = atom_bitset_load(<br> > > + &sg->thr_wanted, __ATOMIC_RELAXED);<br> > > +<br> > > + SEVL();<br> > > + while (WFE() &&<br> > > + !bitset_is_eql(wanted,<br> > > + bitset_ldex(&sg->thr_actual[p],<br> > > + __ATOMIC_RELAXED)))<br> > > + DOZE();<br> > > + }<br> > > + /* Else ignore because no ODP queues on this prio */<br> > > + }<br> > > +<br> > > + /* Check if all threads/queues have left the group */<br> > > + for (p = 0; p < ODP_SCHED_PRIO_NUM; p++) {<br> > > + if (!bitset_is_null(sg->thr_actual[p])) {<br> > > + ODP_ERR("Group has threads\n");<br> > > + ret = -1;<br> > > + goto thrd_q_present_in_group;<br> > > + }<br> > > + if (sg->xcount[p] != 0) {<br> > > + ODP_ERR("Group has queues\n");<br> > > + ret = -1;<br> > > + goto thrd_q_present_in_group;<br> > > + }<br> > > + }<br> > > +<br> > > + odp_shm_free(sg->shm);<br> > > + sg_vec[sgi] = NULL;<br> > > + atom_bitset_set(&sg_free, sgi, __ATOMIC_RELEASE);<br> > > +<br> > > + odp_spinlock_unlock(&sched_grp_lock);<br> > > +<br> > > + return ret;<br> > > +<br> > > +thrd_q_present_in_group:<br> > > +<br> > > +group_not_found:<br> > > + odp_spinlock_unlock(&sched_grp_lock);<br> > > +<br> > > +invalid_group:<br> > > +<br> > > + return ret;<br> > > +}<br> > > +<br> > > +static odp_schedule_group_t schedule_group_lookup(const char *name)<br> > > +{<br> > > + uint32_t sgi;<br> > > + odp_schedule_group_t group;<br> > > +<br> > > + /* Validate inputs */<br> > > + if (name == NULL)<br> > > + ODP_ABORT("name or mask is NULL\n");<br> > > +<br> > > + group = ODP_SCHED_GROUP_INVALID;<br> > > +<br> > > + odp_spinlock_lock(&sched_grp_lock);<br> > > +<br> > > + /* Scan through the schedule group array */<br> > > + for (sgi = 0; sgi < MAX_SCHED_GROUP; sgi++) {<br> > > + if ((sg_vec[sgi] != NULL) &&<br> > > + (strncmp(name, sg_vec[sgi]->name,<br> > > + ODP_SCHED_GROUP_NAME_LEN) == 0)) {<br> > > + group = (odp_schedule_group_t)sgi;<br> > > + break;<br> > > + }<br> > > + }<br> > > +<br> > > + odp_spinlock_unlock(&sched_grp_lock);<br> > > +<br> > > + return group;<br> > > +}<br> > > +<br> > > +static int schedule_group_join(odp_schedule_group_t group,<br> > > + const odp_thrmask_t *mask)<br> > > +{<br> > > + uint32_t sgi;<br> > > + sched_group_t *sg;<br> > > + int ret;<br> > > +<br> > > + /* Validate inputs */<br> > > + if (group < 0 && group >= ((odp_schedule_group_t)MAX_SCHED_GROUP))<br> > > + return -1;<br> > > +<br> > > + if (mask == NULL)<br> > > + ODP_ABORT("name or mask is NULL\n");<br> > > +<br> > > + odp_spinlock_lock(&sched_grp_lock);<br> > > +<br> > > + sgi = (uint32_t)group;<br> > > + if (bitset_is_set(sg_free, sgi)) {<br> > > + odp_spinlock_unlock(&sched_grp_lock);<br> > > + return -1;<br> > > + }<br> > > +<br> > > + sg = sg_vec[sgi];<br> > > + ret = schedule_group_update(sg, sgi, mask, SCHED_GROUP_JOIN);<br> > > +<br> > > + odp_spinlock_unlock(&sched_grp_lock);<br> > > +<br> > > + return ret;<br> > > +}<br> > > +<br> > > +static int schedule_group_leave(odp_schedule_group_t group,<br> > > + const odp_thrmask_t *mask)<br> > > +{<br> > > + uint32_t sgi;<br> > > + sched_group_t *sg;<br> > > + int ret = 0;<br> > > +<br> > > + /* Validate inputs */<br> > > + if (group < 0 && group >= (odp_schedule_group_t)MAX_SCHED_GROUP) {<br> > > + ret = -1;<br> > > + goto invalid_group;<br> > > + }<br> > > +<br> > > + if (mask == NULL)<br> > > + ODP_ABORT("name or mask is NULL\n");<br> > > +<br> > > + odp_spinlock_lock(&sched_grp_lock);<br> > > +<br> > > + sgi = (uint32_t)group;<br> > > + if (bitset_is_set(sg_free, sgi)) {<br> > > + ret = -1;<br> > > + goto group_not_found;<br> > > + }<br> > > +<br> > > + sg = sg_vec[sgi];<br> > > +<br> > > + ret = schedule_group_update(sg, sgi, mask, SCHED_GROUP_LEAVE);<br> > > +<br> > > + odp_spinlock_unlock(&sched_grp_lock);<br> > > +<br> > > + return ret;<br> > > +<br> > > +group_not_found:<br> > > + odp_spinlock_unlock(&sched_grp_lock);<br> > > +<br> > > +invalid_group:<br> > > + return ret;<br> > > +}<br> > > +<br> > > +static int schedule_group_thrmask(odp_schedule_group_t group,<br> > > + odp_thrmask_t *mask)<br> > > +{<br> > > + uint32_t sgi;<br> > > + sched_group_t *sg;<br> > > + int ret = 0;<br> > > +<br> > > + /* Validate inputs */<br> > > + if (group < 0 && group >= ((odp_schedule_group_t)MAX_SCHED_GROUP)) {<br> > > + ret = -1;<br> > > + goto invalid_group;<br> > > + }<br> > > +<br> > > + if (mask == NULL)<br> > > + ODP_ABORT("name or mask is NULL\n");<br> > > +<br> > > + odp_spinlock_lock(&sched_grp_lock);<br> > > +<br> > > + sgi = (uint32_t)group;<br> > > + if (bitset_is_set(sg_free, sgi)) {<br> > > + ret = -1;<br> > > + goto group_not_found;<br> > > + }<br> > > +<br> > > + sg = sg_vec[sgi];<br> > > + ret = _schedule_group_thrmask(sg, mask);<br> > > +<br> > > + odp_spinlock_unlock(&sched_grp_lock);<br> > > +<br> > > + return ret;<br> > > +<br> > > +group_not_found:<br> > > + odp_spinlock_unlock(&sched_grp_lock);<br> > > +<br> > > +invalid_group:<br> > > + return ret;<br> > > +}<br> > > +<br> > > +static int schedule_group_info(odp_schedule_group_t group,<br> > > + odp_schedule_group_info_t *info)<br> > > +{<br> > > + uint32_t sgi;<br> > > + sched_group_t *sg;<br> > > + int ret = 0;<br> > > +<br> > > + /* Validate inputs */<br> > > + if (group < 0 && group >= ((odp_schedule_group_t)MAX_SCHED_GROUP)) {<br> > > + ret = -1;<br> > > + goto invalid_group;<br> > > + }<br> > > +<br> > > + if (info == NULL)<br> > > + ODP_ABORT("name or mask is NULL\n");<br> > > +<br> > > + odp_spinlock_lock(&sched_grp_lock);<br> > > +<br> > > + sgi = (uint32_t)group;<br> > > + if (bitset_is_set(sg_free, sgi)) {<br> > > + ret = -1;<br> > > + goto group_not_found;<br> > > + }<br> > > +<br> > > + sg = sg_vec[sgi];<br> > > +<br> > > + ret = _schedule_group_thrmask(sg, &info->thrmask);<br> > > +<br> > > + info->name = sg->name;<br> > > +<br> > > + odp_spinlock_unlock(&sched_grp_lock);<br> > > +<br> > > + return ret;<br> > > +<br> > > +group_not_found:<br> > > + odp_spinlock_unlock(&sched_grp_lock);<br> > > +<br> > > +invalid_group:<br> > > + return ret;<br> > > +}<br> > > +<br> > > +static int schedule_init_global(void)<br> > > +{<br> > > + odp_thrmask_t mask;<br> > > + odp_schedule_group_t tmp_all;<br> > > + odp_schedule_group_t tmp_wrkr;<br> > > + odp_schedule_group_t tmp_ctrl;<br> > > + uint32_t bits;<br> > > +<br> > > + odp_spinlock_init(&sched_grp_lock);<br> > > +<br> > > + bits = MAX_SCHED_GROUP;<br> > > + if (MAX_SCHED_GROUP == sizeof(sg_free) * CHAR_BIT)<br> > > + sg_free = ~0ULL;<br> > > + else<br> > > + sg_free = (1ULL << bits) - 1;<br> > > +<br> > > + for (uint32_t i = 0; i < MAX_SCHED_GROUP; i++)<br> > > + sg_vec[i] = NULL;<br> > > + for (uint32_t i = 0; i < MAXTHREADS; i++) {<br> > > + thread_state[i].sg_sem = 0;<br> > > + for (uint32_t j = 0; j < ODP_SCHED_PRIO_NUM; j++)<br> > > + thread_state[i].sg_wanted[j] = bitset_null();<br> > > + }<br> > > +<br> > > + /* Create sched groups for default GROUP_ALL, GROUP_WORKER and<br> > > + * GROUP_CONTROL groups.<br> > > + */<br> > > + odp_thrmask_zero(&mask);<br> > > + tmp_all = odp_schedule_group_create("__group_all", &mask);<br> > > + if (tmp_all != ODP_SCHED_GROUP_ALL) {<br> > > + ODP_ERR("Could not create ODP_SCHED_GROUP_ALL()\n");<br> > > + goto failed_create_group_all;<br> > > + }<br> > > +<br> > > + tmp_wrkr = odp_schedule_group_create("__group_worker", &mask);<br> > > + if (tmp_wrkr != ODP_SCHED_GROUP_WORKER) {<br> > > + ODP_ERR("Could not create ODP_SCHED_GROUP_WORKER()\n");<br> > > + goto failed_create_group_worker;<br> > > + }<br> > > +<br> > > + tmp_ctrl = odp_schedule_group_create("__group_control", &mask);<br> > > + if (tmp_ctrl != ODP_SCHED_GROUP_CONTROL) {<br> > > + ODP_ERR("Could not create ODP_SCHED_GROUP_CONTROL()\n");<br> > > + goto failed_create_group_control;<br> > > + }<br> > > +<br> > > + return 0;<br> > > +<br> > > +failed_create_group_control:<br> > > + if (tmp_ctrl != ODP_SCHED_GROUP_INVALID)<br> > > + odp_schedule_group_destroy(ODP_SCHED_GROUP_CONTROL);<br> > > +<br> > > +failed_create_group_worker:<br> > > + if (tmp_wrkr != ODP_SCHED_GROUP_INVALID)<br> > > + odp_schedule_group_destroy(ODP_SCHED_GROUP_WORKER);<br> > > +<br> > > +failed_create_group_all:<br> > > + if (tmp_all != ODP_SCHED_GROUP_INVALID)<br> > > + odp_schedule_group_destroy(ODP_SCHED_GROUP_ALL);<br> > > +<br> > > + return -1;<br> > > +}<br> > > +<br> > > +static int schedule_term_global(void)<br> > > +{<br> > > + /* Destroy sched groups for default GROUP_ALL, GROUP_WORKER and<br> > > + * GROUP_CONTROL groups.<br> > > + */<br> > > + if (odp_schedule_group_destroy(ODP_SCHED_GROUP_ALL) != 0)<br> > > + ODP_ERR("Failed to destroy ODP_SCHED_GROUP_ALL\n");<br> > > + if (odp_schedule_group_destroy(ODP_SCHED_GROUP_WORKER) != 0)<br> > > + ODP_ERR("Failed to destroy ODP_SCHED_GROUP_WORKER\n");<br> > > + if (odp_schedule_group_destroy(ODP_SCHED_GROUP_CONTROL) != 0)<br> > > + ODP_ERR("Failed to destroy ODP_SCHED_GROUP_CONTROL\n");<br> > > +<br> > > + return 0;<br> > > +}<br> > > +<br> > > +static int schedule_init_local(void)<br> > > +{<br> > > + int thr_id;<br> > > + odp_thread_type_t thr_type;<br> > > + odp_thrmask_t mask;<br> > > +<br> > > + thr_id = odp_thread_id();<br> > > + if (thread_state_init(thr_id))<br> > > + goto failed_to_init_ts;<br> > > +<br> > > + /* Add this thread to default schedule groups */<br> > > + thr_type = odp_thread_type();<br> > > + odp_thrmask_zero(&mask);<br> > > + odp_thrmask_set(&mask, thr_id);<br> > > +<br> > > + if (odp_schedule_group_join(ODP_SCHED_GROUP_ALL, &mask) != 0) {<br> > > + ODP_ERR("Failed to join ODP_SCHED_GROUP_ALL\n");<br> > > + goto failed_to_join_grp_all;<br> > > + }<br> > > + if (thr_type == ODP_THREAD_CONTROL) {<br> > > + if (odp_schedule_group_join(ODP_SCHED_GROUP_CONTROL,<br> > > + &mask) != 0) {<br> > > + ODP_ERR("Failed to join ODP_SCHED_GROUP_CONTROL\n");<br> > > + goto failed_to_join_grp_ctrl;<br> > > + }<br> > > + } else {<br> > > + if (odp_schedule_group_join(ODP_SCHED_GROUP_WORKER,<br> > > + &mask) != 0) {<br> > > + ODP_ERR("Failed to join ODP_SCHED_GROUP_WORKER\n");<br> > > + goto failed_to_join_grp_wrkr;<br> > > + }<br> > > + }<br> > > +<br> > > + return 0;<br> > > +<br> > > +failed_to_join_grp_wrkr:<br> > > +<br> > > +failed_to_join_grp_ctrl:<br> > > + odp_schedule_group_leave(ODP_SCHED_GROUP_ALL, &mask);<br> > > +<br> > > +failed_to_join_grp_all:<br> > > +failed_to_init_ts:<br> > > +<br> > > + return -1;<br> > > +}<br> > > +<br> > > +static int schedule_term_local(void)<br> > > +{<br> > > + int thr_id;<br> > > + odp_thread_type_t thr_type;<br> > > + odp_thrmask_t mask;<br> > > + int rc = 0;<br> > > +<br> > > + /* Remove this thread from default schedule groups */<br> > > + thr_id = odp_thread_id();<br> > > + thr_type = odp_thread_type();<br> > > + odp_thrmask_zero(&mask);<br> > > + odp_thrmask_set(&mask, thr_id);<br> > > +<br> > > + if (odp_schedule_group_leave(ODP_SCHED_GROUP_ALL, &mask) != 0)<br> > > + ODP_ERR("Failed to leave ODP_SCHED_GROUP_ALL\n");<br> > > + if (thr_type == ODP_THREAD_CONTROL) {<br> > > + if (odp_schedule_group_leave(ODP_SCHED_GROUP_CONTROL,<br> > > + &mask) != 0)<br> > > + ODP_ERR("Failed to leave ODP_SCHED_GROUP_CONTROL\n");<br> > > + } else {<br> > > + if (odp_schedule_group_leave(ODP_SCHED_GROUP_WORKER,<br> > > + &mask) != 0)<br> > > + ODP_ERR("Failed to leave ODP_SCHED_GROUP_WORKER\n");<br> > > + }<br> > > +<br> > > + update_sg_membership(sched_ts);<br> > > +<br> > > + /* Check if the thread is still part of any groups */<br> > > + if (sched_ts->num_schedq != 0) {<br> > > + ODP_ERR("Thread %d still part of scheduler group(s)\n",<br> > > + sched_ts->tidx);<br> > > + rc = -1;<br> > > + }<br> > > +<br> > > + return rc;<br> > > +}<br> > > +<br> > > +int queue_tm_reorder(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr)<br> > > +{<br> > > + (void)queue;<br> > > + (void)buf_hdr;<br> > > + return 0;<br> > > +}<br> > > +<br> > > +static void pktio_start(int pktio_index, int num_in_queue, int in_queue_idx[])<br> > > +{<br> > > + int i;<br> > > + uint32_t old, tag, j;<br> > > +<br> > > + for (i = 0; i < num_in_queue; i++) {<br> > > + /* Try to reserve a slot */<br> > > + if (__atomic_fetch_add(&pktin_num,<br> > > + 1, __ATOMIC_RELAXED) >= PKTIN_MAX) {<br> > > + __atomic_fetch_sub(&pktin_num, 1, __ATOMIC_RELAXED);<br> > > + ODP_ABORT("Too many pktio in queues for scheduler\n");<br> > > + }<br> > > + /* A slot has been reserved, now we need to find an empty one */<br> > > + for (j = 0; ; j = (j + 1) % PKTIN_MAX) {<br> > > + if (__atomic_load_n(&pktin_tags[j],<br> > > + __ATOMIC_RELAXED) != TAG_EMPTY)<br> > > + /* Slot used, continue with next */<br> > > + continue;<br> > > + /* Empty slot found */<br> > > + old = TAG_EMPTY;<br> > > + tag = PKTIO_QUEUE_2_TAG(pktio_index, in_queue_idx[i]);<br> > > + if (__atomic_compare_exchange_n(&pktin_tags[j],<br> > > + &old,<br> > > + tag,<br> > > + true,<br> > > + __ATOMIC_RELEASE,<br> > > + __ATOMIC_RELAXED)) {<br> > > + /* Success grabbing slot,update high<br> > > + * watermark<br> > > + */<br> > > + __atomic_fetch_max(&pktin_hi,<br> > > + j + 1, __ATOMIC_RELAXED);<br> > > + /* One more tag (queue) for this pktio<br> > > + * instance<br> > > + */<br> > > + __atomic_fetch_add(&pktin_count[pktio_index],<br> > > + 1, __ATOMIC_RELAXED);<br> > > + /* Continue with next RX queue */<br> > > + break;<br> > > + }<br> > > + /* Failed to grab slot */<br> > > + }<br> > > + }<br> > > +}<br> > > +<br> > > +static int num_grps(void)<br> > > +{<br> > > + return MAX_SCHED_GROUP;<br> > > +}<br> > > +<br> > > +/*<br> > > + * Stubs for internal scheduler abstraction layer due to absence of NULL<br> > > + * checking before calling the function pointer.<br> > > + */<br> > > +<br> > > +static int thr_add(odp_schedule_group_t group, int thr)<br> > > +{<br> > > + /* This function is a schedule_init_local duplicate. */<br> > > + (void)group;<br> > > + (void)thr;<br> > > + return 0;<br> > > +}<br> > > +static int thr_rem(odp_schedule_group_t group, int thr)<br> > > +{<br> > > + /* This function is a schedule_term_local duplicate. */<br> > > + (void)group;<br> > > + (void)thr;<br> > > + return 0;<br> > > +}<br> > > +static int init_queue(uint32_t queue_index,<br> > > + const odp_schedule_param_t *sched_param)<br> > > +{<br> > > + /* Not used in scalable scheduler. */<br> > > + (void)queue_index;<br> > > + (void)sched_param;<br> > > + return 0;<br> > > +}<br> > > +static void destroy_queue(uint32_t queue_index)<br> > > +{<br> > > + /* Not used in scalable scheduler. */<br> > > + (void)queue_index;<br> > > +}<br> > > +static int sched_queue(uint32_t queue_index)<br> > > +{<br> > > + /* Not used in scalable scheduler. */<br> > > + (void)queue_index;<br> > > + return 0;<br> > > +}<br> > > +static int ord_enq_multi(uint32_t queue_index, void *p_buf_hdr[],<br> > > + int num, int *ret)<br> > > +{<br> > > + (void)queue_index;<br> > > + (void)p_buf_hdr;<br> > > + (void)num;<br> > > + (void)ret;<br> > > + return 0;<br> > > +}<br> > > +<br> > > +static void schedule_prefetch(int num)<br> > > +{<br> > > + (void)num;<br> > > +}<br> > > +<br> > > +static void order_lock(void)<br> > > +{<br> > > +}<br> > > +<br> > > +static void order_unlock(void)<br> > > +{<br> > > +}<br> > > +<br> > > +const schedule_fn_t schedule_scalable_fn = {<br> > > + .pktio_start = pktio_start,<br> > > + .thr_add = thr_add,<br> > > + .thr_rem = thr_rem,<br> > > + .num_grps = num_grps,<br> > > + .init_queue = init_queue,<br> > > + .destroy_queue = destroy_queue,<br> > > + .sched_queue = sched_queue,<br> > > + .ord_enq_multi = ord_enq_multi,<br> > > + .init_global = schedule_init_global,<br> > > + .term_global = schedule_term_global,<br> > > + .init_local = schedule_init_local,<br> > > + .term_local = schedule_term_local,<br> > > + .order_lock = order_lock,<br> > > + .order_unlock = order_unlock,<br> > > +};<br> > > +<br> > > +const schedule_api_t schedule_scalable_api = {<br> > > + .schedule_wait_time = schedule_wait_time,<br> > > + .schedule = schedule,<br> > > + .schedule_multi = schedule_multi,<br> > > + .schedule_pause = schedule_pause,<br> > > + .schedule_resume = schedule_resume,<br> > > + .schedule_release_atomic = schedule_release_atomic,<br> > > + .schedule_release_ordered = schedule_release_ordered,<br> > > + .schedule_prefetch = schedule_prefetch,<br> > > + .schedule_num_prio = schedule_num_prio,<br> > > + .schedule_group_create = schedule_group_create,<br> > > + .schedule_group_destroy = schedule_group_destroy,<br> > > + .schedule_group_lookup = schedule_group_lookup,<br> > > + .schedule_group_join = schedule_group_join,<br> > > + .schedule_group_leave = schedule_group_leave,<br> > > + .schedule_group_thrmask = schedule_group_thrmask,<br> > > + .schedule_group_info = schedule_group_info,<br> > > + .schedule_order_lock = schedule_order_lock,<br> > > + .schedule_order_unlock = schedule_order_unlock,<br> > > +};<br> > > diff --git a/platform/linux-generic/odp_schedule_scalable_ordered.c b/platform/linux-generic/odp_schedule_scalable_ordered.c<br> > > new file mode 100644<br> > > index 00000000..ff4431b5<br> > > --- /dev/null<br> > > +++ b/platform/linux-generic/odp_schedule_scalable_ordered.c<br> > > @@ -0,0 +1,298 @@<br> > > +/* Copyright (c) 2017, ARM Limited<br> > > + * All rights reserved.<br> > > + *<br> > > + * SPDX-License-Identifier: BSD-3-Clause<br> > > + */<br> > > +<br> > > +#include <odp/api/shared_memory.h><br> > > +#include <odp_queue_internal.h><br> > > +#include <odp_schedule_if.h><br> > > +#include <odp_schedule_ordered_internal.h><br> > > +#include <odp_llsc.h><br> > > +#include <odp_bitset.h><br> > > +<br> > > +#include <string.h><br> > > +<br> > > +extern __thread sched_scalable_thread_state_t *sched_ts;<br> > > +<br> > > +#define RWIN_NAME_SIZE 32<br> > > +<br> > > +reorder_window_t *rwin_alloc(int rwin_id, odp_shm_t *shm, unsigned lock_count)<br> > > +{<br> > > + char rwin_name[RWIN_NAME_SIZE];<br> > > + reorder_window_t *rwin;<br> > > + uint32_t i;<br> > > +<br> > > + strncpy(rwin_name, "rwin", RWIN_NAME_SIZE);<br> > > + i = strlen(rwin_name);<br> > > + snprintf(rwin_name + i,<br> > > + (RWIN_NAME_SIZE - i), "%d", rwin_id);<br> > > +<br> > > + *shm = odp_shm_reserve(rwin_name,<br> > > + sizeof(reorder_window_t),<br> > > + ODP_CACHE_LINE_SIZE,<br> > > + ODP_SHM_PROC);<br> > > + if (ODP_SHM_INVALID == *shm)<br> > > + goto shm_reserve_failed;<br> > > +<br> > > + rwin = (reorder_window_t *) odp_shm_addr(*shm);<br> > > + if (rwin == NULL)<br> > > + goto shm_addr_failed;<br> > > +<br> > > + rwin->hc.head = 0;<br> > > + rwin->hc.chgi = 0;<br> > > + rwin->winmask = RWIN_SIZE - 1;<br> > > + rwin->tail = 0;<br> > > + rwin->turn = 0;<br> > > + rwin->lock_count = (uint16_t)lock_count;<br> > > + memset(rwin->olock, 0, sizeof(rwin->olock));<br> > > + for (i = 0; i < RWIN_SIZE; i++)<br> > > + rwin->ring[i] = NULL;<br> > > +<br> > > + return rwin;<br> > > +<br> > > +shm_addr_failed:<br> > > + odp_shm_free(*shm);<br> > > +<br> > > +shm_reserve_failed:<br> > > + return NULL;<br> > > +}<br> > > +<br> > > +bool rwin_reserve(reorder_window_t *rwin, uint32_t *sn)<br> > > +{<br> > > + uint32_t head;<br> > > + uint32_t oldt;<br> > > + uint32_t newt;<br> > > + uint32_t winmask;<br> > > +<br> > > + /* Read head and tail separately */<br> > > +#ifndef ODP_CONFIG_USE_LLSC<br> > > + oldt = rwin->tail;<br> > > +#endif<br> > > + __builtin_prefetch(&rwin->tail, 1, 0);<br> > > + winmask = rwin->winmask;<br> > > + do {<br> > > + /* Need __atomic_load to avoid compiler reordering */<br> > > + head = __atomic_load_n(&rwin->hc.head, __ATOMIC_RELAXED);<br> > > +#ifdef ODP_CONFIG_USE_LLSC<br> > > + oldt = ll32(&rwin->tail, __ATOMIC_RELAXED);<br> > > +#endif<br> > > + if (odp_unlikely(oldt - head >= winmask))<br> > > + return false;<br> > > +<br> > > + newt = oldt + 1;<br> > > +#ifdef ODP_CONFIG_USE_LLSC<br> > > + } while (odp_unlikely(sc32(&rwin->tail, newt, __ATOMIC_RELAXED)));<br> > > +#else<br> > > + } while (!__atomic_compare_exchange(&rwin->tail,<br> > > + &oldt,<br> > > + &newt,<br> > > + true,<br> > > + __ATOMIC_RELAXED,<br> > > + __ATOMIC_RELAXED));<br> > > +#endif<br> > > + *sn = oldt;<br> > > +<br> > > + return true;<br> > > +}<br> > > +<br> > > +void rwin_insert(reorder_window_t *rwin,<br> > > + reorder_context_t *rctx,<br> > > + uint32_t sn,<br> > > + void (*callback)(reorder_context_t *))<br> > > +{<br> > > + /* Initialise to silence scan-build */<br> > > + hc_t old = {0, 0};<br> > > + hc_t new;<br> > > + uint32_t winmask;<br> > > +<br> > > + __atomic_load(&rwin->hc, &old, __ATOMIC_ACQUIRE);<br> > > + winmask = rwin->winmask;<br> > > + if (old.head != sn) {<br> > > + /* We are out-of-order. Store context in reorder window,<br> > > + * releasing its content.<br> > > + */<br> > > + ODP_ASSERT(rwin->ring[sn & winmask] == NULL);<br> > > +#ifdef ODP_CONFIG_USE_DMB<br> > > + __atomic_thread_fence(__ATOMIC_RELEASE);<br> > > + __atomic_store_n(&rwin->ring[sn & winmask], rctx,<br> > > + __ATOMIC_RELAXED);<br> > > +#else<br> > > + __atomic_store_n(&rwin->ring[sn & winmask], rctx,<br> > > + __ATOMIC_RELEASE);<br> > > +#endif<br> > > + rctx = NULL;<br> > > + do {<br> > > + hc_t new;<br> > > +<br> > > + new.head = old.head;<br> > > + new.chgi = old.chgi + 1; /* Changed value */<br> > > + /* Update head & chgi, fail if any has changed */<br> > > + if (__atomic_compare_exchange(&rwin->hc,<br> > > + &old, /* Updated on failure */<br> > > + &new,<br> > > + true,<br> > > + __ATOMIC_RELEASE, /* Release our ring update */<br> > > + __ATOMIC_ACQUIRE))<br> > > + /* CAS succeeded => head same (we are not<br> > > + * in-order), chgi updated.<br> > > + */<br> > > + return;<br> > > + /* CAS failed => head and/or chgi changed.<br> > > + * We might not be out-of-order anymore.<br> > > + */<br> > > + } while (old.head != sn);<br> > > + }<br> > > +<br> > > + /* old.head == sn => we are now in-order! */<br> > > + ODP_ASSERT(old.head == sn);<br> > > + /* We are in-order so our responsibility to retire contexts */<br> > > + new.head = old.head;<br> > > + new.chgi = old.chgi + 1;<br> > > +<br> > > + /* Retire our in-order context (if we still have it) */<br> > > + if (rctx != NULL) {<br> > > + callback(rctx);<br> > > + new.head++;<br> > > + }<br> > > +<br> > > + /* Retire in-order contexts in the ring<br> > > + * The first context might actually be ours (if we were originally<br> > > + * out-of-order)<br> > > + */<br> > > + do {<br> > > + for (;;) {<br> > > + rctx = __atomic_load_n(&rwin->ring[new.head & winmask],<br> > > + __ATOMIC_ACQUIRE);<br> > > + if (rctx == NULL)<br> > > + break;<br> > > + /* We are the only thread that are in-order<br> > > + * (until head updated) so don't have to use<br> > > + * atomic load-and-clear (exchange)<br> > > + */<br> > > + rwin->ring[new.head & winmask] = NULL;<br> > > + callback(rctx);<br> > > + new.head++;<br> > > + }<br> > > + /* Update head&chgi, fail if chgi has changed (head cannot change) */<br> > > + } while (!__atomic_compare_exchange(&rwin->hc,<br> > > + &old, /* Updated on failure */<br> > > + &new,<br> > > + false, /* weak */<br> > > + __ATOMIC_RELEASE, /* Release our ring updates */<br> > > + __ATOMIC_ACQUIRE));<br> > > +}<br> > > +<br> > > +void rctx_init(reorder_context_t *rctx, uint16_t idx,<br> > > + reorder_window_t *rwin, uint32_t sn)<br> > > +{<br> > > + /* rctx->rvec_free and rctx->idx already initialised in<br> > > + * thread_state_init function.<br> > > + */<br> > > + ODP_ASSERT(rctx->idx == idx);<br> > > + rctx->rwin = rwin;<br> > > + rctx->sn = sn;<br> > > + rctx->olock_flags = 0;<br> > > + /* First => no next reorder context */<br> > > + rctx->next_idx = idx;<br> > > + /* Where to store next event */<br> > > + rctx->cur_idx = idx;<br> > > + rctx->numevts = 0;<br> > > +}<br> > > +<br> > > +inline void rctx_free(const reorder_context_t *rctx)<br> > > +{<br> > > + const reorder_context_t *const base = &rctx[-(int)rctx->idx];<br> > > + const uint32_t first = rctx->idx;<br> > > + uint32_t next_idx;<br> > > +<br> > > + next_idx = rctx->next_idx;<br> > > +<br> > > + ODP_ASSERT(rctx->rwin != NULL);<br> > > + /* Set free bit */<br> > > + if (rctx->rvec_free == &sched_ts->rvec_free)<br> > > + /* Since it is our own reorder context, we can instead<br> > > + * perform a non-atomic and relaxed update on our private<br> > > + * rvec_free.<br> > > + */<br> > > + sched_ts->priv_rvec_free =<br> > > + bitset_set(sched_ts->priv_rvec_free, rctx->idx);<br> > > + else<br> > > + atom_bitset_set(rctx->rvec_free, rctx->idx, __ATOMIC_RELEASE);<br> > > +<br> > > + /* Can't dereference rctx after the corresponding free bit is set */<br> > > + while (next_idx != first) {<br> > > + rctx = &base[next_idx];<br> > > + next_idx = rctx->next_idx;<br> > > + /* Set free bit */<br> > > + if (rctx->rvec_free == &sched_ts->rvec_free)<br> > > + sched_ts->priv_rvec_free =<br> > > + bitset_set(sched_ts->priv_rvec_free, rctx->idx);<br> > > + else<br> > > + atom_bitset_set(rctx->rvec_free, rctx->idx,<br> > > + __ATOMIC_RELEASE);<br> > > + }<br> > > +}<br> > > +<br> > > +inline void olock_unlock(const reorder_context_t *rctx, reorder_window_t *rwin,<br> > > + uint32_t lock_index)<br> > > +{<br> > > + if ((rctx->olock_flags & (1U << lock_index)) == 0) {<br> > > + /* Use relaxed ordering, we are not releasing any updates */<br> > > + rwin->olock[lock_index] = rctx->sn + 1;<br> > > + }<br> > > +}<br> > > +<br> > > +void olock_release(const reorder_context_t *rctx)<br> > > +{<br> > > + reorder_window_t *rwin;<br> > > +<br> > > + rwin = rctx->rwin;<br> > > +<br> > > + switch (rwin->lock_count) {<br> > > + case 2:<br> > > + olock_unlock(rctx, rwin, 1);<br> > > + case 1:<br> > > + olock_unlock(rctx, rwin, 0);<br> > > + }<br> > > + ODP_STATIC_ASSERT(NUM_OLOCKS == 2, "Number of ordered locks != 2");<br> > > +}<br> > > +<br> > > +void rctx_retire(reorder_context_t *first)<br> > > +{<br> > > + reorder_context_t *rctx;<br> > > + queue_entry_t *q;<br> > > + uint32_t i;<br> > > + uint32_t j;<br> > > + uint32_t num;<br> > > + int rc;<br> > > +<br> > > + rctx = first;<br> > > + do {<br> > > + /* Process all events in this reorder context */<br> > > + for (i = 0; i < rctx->numevts;) {<br> > > + q = rctx->destq[i];<br> > > + /* Find index of next different destq */<br> > > + j = i + 1;<br> > > + while (j < rctx->numevts && rctx->destq[j] == q)<br> > > + j++;<br> > > + num = j - i;<br> > > + rc = q->s.enqueue_multi(q, &rctx->events[i], num);<br> > > + if (odp_unlikely(rc != (int)num))<br> > > + ODP_ERR("Failed to enqueue deferred events\n");<br> > > + i += num;<br> > > + }<br> > > + /* Update rctx pointer to point to 'next_idx' element */<br> > > + rctx += (int)rctx->next_idx - (int)rctx->idx;<br> > > + } while (rctx != first);<br> > > + olock_release(first);<br> > > + rctx_free(first);<br> > > +}<br> > > +<br> > > +void rctx_release(reorder_context_t *rctx)<br> > > +{<br> > > + /* Insert reorder context into reorder window, potentially calling the<br> > > + * rctx_retire function for all pending reorder_contexts.<br> > > + */<br> > > + rwin_insert(rctx->rwin, rctx, rctx->sn, rctx_retire);<br> > > +}<br> > > diff --git a/platform/linux-generic/odp_traffic_mngr.c b/platform/linux-generic/odp_traffic_mngr.c<br> > > index 4e9358b9..3244dfe3 100644<br> > > --- a/platform/linux-generic/odp_traffic_mngr.c<br> > > +++ b/platform/linux-generic/odp_traffic_mngr.c<br> > > @@ -3918,9 +3918,10 @@ odp_tm_queue_t odp_tm_queue_create(odp_tm_t odp_tm,<br> > > tm_queue_obj->pkt = ODP_PACKET_INVALID;<br> > > odp_ticketlock_init(&tm_wred_node->tm_wred_node_lock);<br> > ><br> > > - tm_queue_obj->tm_qentry.s.type = QUEUE_TYPE_TM;<br> > > - tm_queue_obj->tm_qentry.s.enqueue = queue_tm_reenq;<br> > > - tm_queue_obj->tm_qentry.s.enqueue_multi = queue_tm_reenq_multi;<br> > > + queue_set_type(&tm_queue_obj->tm_qentry, QUEUE_TYPE_TM);<br> > > + queue_set_enq_func(&tm_queue_obj->tm_qentry, queue_tm_reenq);<br> > > + queue_set_enq_multi_func(&tm_queue_obj->tm_qentry,<br> > > + queue_tm_reenq_multi);<br> > ><br> > > tm_system->queue_num_tbl[tm_queue_obj->queue_num - 1] = tm_queue_obj;<br> > > odp_ticketlock_lock(&tm_system->tm_system_lock);<br> > > diff --git a/platform/linux-generic/pktio/loop.c b/platform/linux-generic/pktio/loop.c<br> > > index 70962839..1f2f16c1 100644<br> > > --- a/platform/linux-generic/pktio/loop.c<br> > > +++ b/platform/linux-generic/pktio/loop.c<br> > > @@ -80,11 +80,13 @@ static int loopback_recv(pktio_entry_t *pktio_entry, int index ODP_UNUSED,<br> > ><br> > > for (i = 0; i < nbr; i++) {<br> > > uint32_t pkt_len;<br> > > -<br> > > +#ifdef ODP_SCHEDULE_SCALABLE<br> > > + pkt = _odp_packet_from_buffer((odp_buffer_t)(hdr_tbl[i]));<br> > > +#else<br> > > pkt = _odp_packet_from_buffer(odp_hdr_to_buf(hdr_tbl[i]));<br> > > +#endif<br> > > pkt_len = odp_packet_len(pkt);<br> > ><br> > > -<br> > > if (pktio_cls_enabled(pktio_entry)) {<br> > > odp_packet_t new_pkt;<br> > > odp_pool_t new_pool;<br> > > @@ -162,7 +164,11 @@ static int loopback_send(pktio_entry_t *pktio_entry, int index ODP_UNUSED,<br> > > len = QUEUE_MULTI_MAX;<br> > ><br> > > for (i = 0; i < len; ++i) {<br> > > +#ifdef ODP_SCHEDULE_SCALABLE<br> > > + hdr_tbl[i] = _odp_packet_to_buf_hdr_ptr(pkt_tbl[i]);<br> > > +#else<br> > > hdr_tbl[i] = buf_hdl_to_hdr(_odp_packet_to_buffer(pkt_tbl[i]));<br> > > +#endif<br> > > bytes += odp_packet_len(pkt_tbl[i]);<br> > > }<br> > ><br> > > @@ -176,6 +182,7 @@ static int loopback_send(pktio_entry_t *pktio_entry, int index ODP_UNUSED,<br> > > pktio_entry->s.stats.out_octets += bytes;<br> > > } else {<br> > > ODP_DBG("queue enqueue failed %i\n", ret);<br> > > + odp_ticketlock_unlock(&pktio_entry->s.txl);<br> > > return -1;<br> > > }<br> > ><br> > > diff --git a/test/common_plat/performance/odp_sched_latency.c b/test/common_plat/performance/odp_sched_latency.c<br> > > index 2b28cd7b..5e11a4f3 100644<br> > > --- a/test/common_plat/performance/odp_sched_latency.c<br> > > +++ b/test/common_plat/performance/odp_sched_latency.c<br> > > @@ -28,7 +28,13 @@<br> > > #define MAX_WORKERS 64 /**< Maximum number of worker threads */<br> > > #define MAX_QUEUES 4096 /**< Maximum number of queues */<br> > > #define EVENT_POOL_SIZE (1024 * 1024) /**< Event pool size */<br> > > +<br> > > +#ifdef ODP_SCHEDULE_SP<br> > > #define TEST_ROUNDS (4 * 1024 * 1024) /**< Test rounds for each thread */<br> > > +#else<br> > > +#define TEST_ROUNDS (32 * 1024 * 1024) /**< Test rounds for each thread */<br> > > +#endif<br> > > +<br> > > #define MAIN_THREAD 1 /**< Thread ID performing maintenance tasks */<br> > ><br> > > /* Default values for command line arguments */<br> > > @@ -104,6 +110,9 @@ typedef union {<br> > > typedef struct {<br> > > core_stat_t core_stat[MAX_WORKERS]; /**< Core specific stats */<br> > > odp_barrier_t barrier; /**< Barrier for thread synchronization */<br> > > +#ifdef ODP_SCHEDULE_SCALABLE<br> > > + odp_schedule_group_t schedule_group;<br> > > +#endif<br> > > odp_pool_t pool; /**< Pool for allocating test events */<br> > > test_args_t args; /**< Parsed command line arguments */<br> > > odp_queue_t queue[NUM_PRIOS][MAX_QUEUES]; /**< Scheduled queues */<br> > > @@ -119,7 +128,11 @@ static void clear_sched_queues(void)<br> > > odp_event_t ev;<br> > ><br> > > while (1) {<br> > > - ev = odp_schedule(NULL, ODP_SCHED_NO_WAIT);<br> > > + /* Need a non-zero timeout to ensure we can observe<br> > > + * any non-empty queue made eligible for scheduling<br> > > + * by some other thread.<br> > > + */<br> > > + ev = odp_schedule(NULL, 1000);<br> > ><br> > > if (ev == ODP_EVENT_INVALID)<br> > > break;<br> > > @@ -428,6 +441,20 @@ static int run_thread(void *arg ODP_UNUSED)<br> > > return -1;<br> > > }<br> > ><br> > > +#ifdef ODP_SCHEDULE_SCALABLE<br> > > + int err;<br> > > + odp_thrmask_t thrmask;<br> > > +<br> > > + odp_thrmask_zero(&thrmask);<br> > > + odp_thrmask_set(&thrmask, thr);<br> > > +<br> > > + err = odp_schedule_group_join(globals->schedule_group, &thrmask);<br> > > + if (err != 0) {<br> > > + LOG_ERR("odp_schedule_group_join failed\n");<br> > > + return -1;<br> > > + }<br> > > +#endif<br> > > +<br> > > if (thr == MAIN_THREAD) {<br> > > args = &globals->args;<br> > ><br> > > @@ -452,6 +479,13 @@ static int run_thread(void *arg ODP_UNUSED)<br> > > if (test_schedule(thr, globals))<br> > > return -1;<br> > ><br> > > +#ifdef ODP_SCHEDULE_SCALABLE<br> > > + err = odp_schedule_group_leave(globals->schedule_group, &thrmask);<br> > > + if (err != 0) {<br> > > + LOG_ERR("odp_schedule_group_leave failed\n");<br> > > + return -1;<br> > > + }<br> > > +#endif<br> > > return 0;<br> > > }<br> > ><br> > > @@ -692,6 +726,31 @@ int main(int argc, char *argv[])<br> > > }<br> > > globals->pool = pool;<br> > ><br> > > +#ifdef ODP_SCHEDULE_SCALABLE<br> > > + /*<br> > > + * Create scheduler group<br> > > + */<br> > > + odp_thrmask_t expected_thrmask;<br> > > + int cpu;<br> > > + int thr;<br> > > +<br> > > + odp_thrmask_zero(&expected_thrmask);<br> > > + /* This is a odp_thrmask_from_cpumask() */<br> > > + cpu = odp_cpumask_first(&cpumask);<br> > > + thr = 1;<br> > > + while (0 <= cpu) {<br> > > + odp_thrmask_set(&expected_thrmask, thr++);<br> > > + cpu = odp_cpumask_next(&cpumask, cpu);<br> > > + }<br> > > +<br> > > + globals->schedule_group =<br> > > + odp_schedule_group_create("sg0", &expected_thrmask);<br> > > + if (globals->schedule_group == ODP_SCHED_GROUP_INVALID) {<br> > > + LOG_ERR("odp_schedule_group_create failed\n");<br> > > + return -1;<br> > > + }<br> > > +#endif<br> > > +<br> > > /*<br> > > * Create queues for schedule test<br> > > */<br> > > @@ -713,7 +772,11 @@ int main(int argc, char *argv[])<br> > > param.type = ODP_QUEUE_TYPE_SCHED;<br> > > param.sched.prio = prio;<br> > > param.sched.sync = args.sync_type;<br> > > +#ifdef ODP_SCHEDULE_SCALABLE<br> > > + param.sched.group = globals->schedule_group;<br> > > +#else<br> > > param.sched.group = ODP_SCHED_GROUP_ALL;<br> > > +#endif<br> > ><br> > > for (j = 0; j < args.prio[i].queues; j++) {<br> > > name[9] = '0' + j / 10;<br> > > @@ -758,6 +821,10 @@ int main(int argc, char *argv[])<br> > > }<br> > > }<br> > ><br> > > +<br> > > +#ifdef ODP_SCHEDULE_SCALABLE<br> > > + ret += odp_schedule_group_destroy(globals->schedule_group);<br> > > +#endif<br> > > ret += odp_shm_free(shm);<br> > > ret += odp_pool_destroy(pool);<br> > > ret += odp_term_local();<br> > > diff --git a/test/common_plat/performance/odp_scheduling.c b/test/common_plat/performance/odp_scheduling.c<br> > > index c74a0713..38e76257 100644<br> > > --- a/test/common_plat/performance/odp_scheduling.c<br> > > +++ b/test/common_plat/performance/odp_scheduling.c<br> > > @@ -273,7 +273,7 @@ static int test_plain_queue(int thr, test_globals_t *globals)<br> > > test_message_t *t_msg;<br> > > odp_queue_t queue;<br> > > uint64_t c1, c2, cycles;<br> > > - int i;<br> > > + int i, j;<br> > ><br> > > /* Alloc test message */<br> > > buf = odp_buffer_alloc(globals->pool);<br> > > @@ -307,7 +307,15 @@ static int test_plain_queue(int thr, test_globals_t *globals)<br> > > return -1;<br> > > }<br> > ><br> > > - ev = odp_queue_deq(queue);<br> > > + /* When enqueue and dequeue are decoupled (e.g. not using a<br> > > + * common lock), an enqueued event may not be immediately<br> > > + * visible to dequeue. So we just try again for a while. */<br> > > + for (j = 0; j < 100; j++) {<br> > > + ev = odp_queue_deq(queue);<br> > > + if (ev != ODP_EVENT_INVALID)<br> > > + break;<br> > > + odp_cpu_pause();<br> > > + }<br> > ><br> > > buf = odp_buffer_from_event(ev);<br> > ><br> > > diff --git a/test/common_plat/validation/api/classification/odp_classification_basic.c b/test/common_plat/validation/api/classification/odp_classification_basic.c<br> > > index 9817287e..b5b92851 100644<br> > > --- a/test/common_plat/validation/api/classification/odp_classification_basic.c<br> > > +++ b/test/common_plat/validation/api/classification/odp_classification_basic.c<br> > > @@ -93,10 +93,10 @@ void classification_test_create_pmr_match(void)<br> > > configure_default_cos(pktio, &default_cos,<br> > > &default_queue, &default_pool);<br> > ><br> > > - queue = queue_create("pmr_match", true);<br> > > + queue = queue_create("pmr_match_queue", true);<br> > > CU_ASSERT(queue != ODP_QUEUE_INVALID);<br> > ><br> > > - pool = pool_create("pmr_match");<br> > > + pool = pool_create("pmr_match_pool");<br> > > CU_ASSERT_FATAL(pool != ODP_POOL_INVALID);<br> > ><br> > > odp_cls_cos_param_init(&cls_param);<br> > > @@ -277,10 +277,10 @@ void classification_test_pmr_composite_create(void)<br> > > configure_default_cos(pktio, &default_cos,<br> > > &default_queue, &default_pool);<br> > ><br> > > - queue = queue_create("pmr_match", true);<br> > > + queue = queue_create("pmr_match_queue", true);<br> > > CU_ASSERT(queue != ODP_QUEUE_INVALID);<br> > ><br> > > - pool = pool_create("pmr_match");<br> > > + pool = pool_create("pmr_match_pool");<br> > > CU_ASSERT_FATAL(pool != ODP_POOL_INVALID);<br> > ><br> > > odp_cls_cos_param_init(&cls_param);<br> > > diff --git a/test/common_plat/validation/api/classification/odp_classification_test_pmr.c b/test/common_plat/validation/api/classification/odp_classification_test_pmr.c<br> > > index d9524205..c1da5159 100644<br> > > --- a/test/common_plat/validation/api/classification/odp_classification_test_pmr.c<br> > > +++ b/test/common_plat/validation/api/classification/odp_classification_test_pmr.c<br> > > @@ -121,10 +121,10 @@ void classification_test_pmr_term_tcp_dport(void)<br> > > configure_default_cos(pktio, &default_cos,<br> > > &default_queue, &default_pool);<br> > ><br> > > - queue = queue_create("tcp_dport1", true);<br> > > + queue = queue_create("tcp_dport1_queue", true);<br> > > CU_ASSERT(queue != ODP_QUEUE_INVALID);<br> > ><br> > > - pool = pool_create("tcp_dport1");<br> > > + pool = pool_create("tcp_dport1_pool");<br> > > CU_ASSERT_FATAL(pool != ODP_POOL_INVALID);<br> > ><br> > > sprintf(cosname, "tcp_dport");<br> > > @@ -235,10 +235,10 @@ void classification_test_pmr_term_tcp_sport(void)<br> > > configure_default_cos(pktio, &default_cos,<br> > > &default_queue, &default_pool);<br> > ><br> > > - queue = queue_create("tcp_sport", true);<br> > > + queue = queue_create("tcp_sport_queue", true);<br> > > CU_ASSERT_FATAL(queue != ODP_QUEUE_INVALID);<br> > ><br> > > - pool = pool_create("tcp_sport");<br> > > + pool = pool_create("tcp_sport_pool");<br> > > CU_ASSERT_FATAL(pool != ODP_POOL_INVALID);<br> > ><br> > > sprintf(cosname, "tcp_sport");<br> > > @@ -348,10 +348,10 @@ void classification_test_pmr_term_udp_dport(void)<br> > > configure_default_cos(pktio, &default_cos,<br> > > &default_queue, &default_pool);<br> > ><br> > > - queue = queue_create("udp_dport", true);<br> > > + queue = queue_create("udp_dport_queue", true);<br> > > CU_ASSERT_FATAL(queue != ODP_QUEUE_INVALID);<br> > ><br> > > - pool = pool_create("udp_dport");<br> > > + pool = pool_create("udp_dport_pool");<br> > > CU_ASSERT_FATAL(pool != ODP_POOL_INVALID);<br> > ><br> > > sprintf(cosname, "udp_dport");<br> > > @@ -464,10 +464,10 @@ void classification_test_pmr_term_udp_sport(void)<br> > > configure_default_cos(pktio, &default_cos,<br> > > &default_queue, &default_pool);<br> > ><br> > > - queue = queue_create("udp_sport", true);<br> > > + queue = queue_create("udp_sport_queue", true);<br> > > CU_ASSERT_FATAL(queue != ODP_QUEUE_INVALID);<br> > ><br> > > - pool = pool_create("udp_sport");<br> > > + pool = pool_create("udp_sport_pool");<br> > > CU_ASSERT_FATAL(pool != ODP_POOL_INVALID);<br> > ><br> > > sprintf(cosname, "udp_sport");<br> > > @@ -578,10 +578,10 @@ void classification_test_pmr_term_ipproto(void)<br> > > configure_default_cos(pktio, &default_cos,<br> > > &default_queue, &default_pool);<br> > ><br> > > - queue = queue_create("ipproto", true);<br> > > + queue = queue_create("ipproto_queue", true);<br> > > CU_ASSERT_FATAL(queue != ODP_QUEUE_INVALID);<br> > ><br> > > - pool = pool_create("ipproto");<br> > > + pool = pool_create("ipproto_pool");<br> > > CU_ASSERT_FATAL(pool != ODP_POOL_INVALID);<br> > ><br> > > sprintf(cosname, "ipproto");<br> > > @@ -687,10 +687,10 @@ void classification_test_pmr_term_dmac(void)<br> > > configure_default_cos(pktio, &default_cos,<br> > > &default_queue, &default_pool);<br> > ><br> > > - queue = queue_create("dmac", true);<br> > > + queue = queue_create("dmac_queue", true);<br> > > CU_ASSERT_FATAL(queue != ODP_QUEUE_INVALID);<br> > ><br> > > - pool = pool_create("dmac");<br> > > + pool = pool_create("dmac_pool");<br> > > CU_ASSERT_FATAL(pool != ODP_POOL_INVALID);<br> > ><br> > > sprintf(cosname, "dmac");<br> > > @@ -794,10 +794,10 @@ void classification_test_pmr_term_packet_len(void)<br> > > configure_default_cos(pktio, &default_cos,<br> > > &default_queue, &default_pool);<br> > ><br> > > - queue = queue_create("packet_len", true);<br> > > + queue = queue_create("packet_len_queue", true);<br> > > CU_ASSERT_FATAL(queue != ODP_QUEUE_INVALID);<br> > ><br> > > - pool = pool_create("packet_len");<br> > > + pool = pool_create("packet_len_pool");<br> > > CU_ASSERT_FATAL(pool != ODP_POOL_INVALID);<br> > ><br> > > sprintf(cosname, "packet_len");<br> > > @@ -1355,10 +1355,10 @@ static void classification_test_pmr_pool_set(void)<br> > > configure_default_cos(pktio, &default_cos,<br> > > &default_queue, &default_pool);<br> > ><br> > > - queue = queue_create("ipproto1", true);<br> > > + queue = queue_create("ipproto1_queue", true);<br> > > CU_ASSERT_FATAL(queue != ODP_QUEUE_INVALID);<br> > ><br> > > - pool = pool_create("ipproto1");<br> > > + pool = pool_create("ipproto1_pool");<br> > > CU_ASSERT_FATAL(pool != ODP_POOL_INVALID);<br> > ><br> > > sprintf(cosname, "ipproto1");<br> > > @@ -1454,10 +1454,10 @@ static void classification_test_pmr_queue_set(void)<br> > > configure_default_cos(pktio, &default_cos,<br> > > &default_queue, &default_pool);<br> > ><br> > > - queue = queue_create("ipproto1", true);<br> > > + queue = queue_create("ipproto1_queue", true);<br> > > CU_ASSERT_FATAL(queue != ODP_QUEUE_INVALID);<br> > ><br> > > - pool = pool_create("ipproto1");<br> > > + pool = pool_create("ipproto1_pool");<br> > > CU_ASSERT_FATAL(pool != ODP_POOL_INVALID);<br> > ><br> > > sprintf(cosname, "ipproto1");<br> > > @@ -1469,7 +1469,7 @@ static void classification_test_pmr_queue_set(void)<br> > > cos = odp_cls_cos_create(cosname, &cls_param);<br> > > CU_ASSERT_FATAL(cos != ODP_COS_INVALID);<br> > ><br> > > - queue_new = queue_create("ipproto2", true);<br> > > + queue_new = queue_create("ipproto2_queue", true);<br> > > CU_ASSERT_FATAL(queue_new != ODP_QUEUE_INVALID);<br> > ><br> > > /* new queue is set on CoS */<br> > > @@ -1546,10 +1546,10 @@ static void classification_test_pmr_term_daddr(void)<br> > > configure_default_cos(pktio, &default_cos,<br> > > &default_queue, &default_pool);<br> > ><br> > > - queue = queue_create("daddr", true);<br> > > + queue = queue_create("daddr_queue", true);<br> > > CU_ASSERT_FATAL(queue != ODP_QUEUE_INVALID);<br> > ><br> > > - pool = pool_create("daddr");<br> > > + pool = pool_create("daddr_pool");<br> > > CU_ASSERT_FATAL(pool != ODP_POOL_INVALID);<br> > ><br> > > sprintf(cosname, "daddr");<br> > > diff --git a/test/common_plat/validation/api/scheduler/scheduler.c b/test/common_plat/validation/api/scheduler/scheduler.c<br> > > index 952561cd..bc486192 100644<br> > > --- a/test/common_plat/validation/api/scheduler/scheduler.c<br> > > +++ b/test/common_plat/validation/api/scheduler/scheduler.c<br> > > @@ -251,7 +251,10 @@ void scheduler_test_queue_destroy(void)<br> > > CU_ASSERT_FATAL(u32[0] == MAGIC);<br> > ><br> > > odp_buffer_free(buf);<br> > > - odp_schedule_release_ordered();<br> > > + if (qp.sched.sync == ODP_SCHED_SYNC_ATOMIC)<br> > > + odp_schedule_release_atomic();<br> > > + else if (qp.sched.sync == ODP_SCHED_SYNC_ORDERED)<br> > > + odp_schedule_release_ordered();<br> > ><br> > > CU_ASSERT_FATAL(odp_queue_destroy(queue) == 0);<br> > > }<br> > > @@ -478,6 +481,11 @@ void scheduler_test_groups(void)<br> > > odp_schedule_group_leave(mygrp1, &mymask);<br> > > odp_schedule_group_leave(mygrp2, &mymask);<br> > ><br> > > + if (qp.sched.sync == ODP_SCHED_SYNC_ATOMIC)<br> > > + odp_schedule_release_atomic();<br> > > + else if (qp.sched.sync == ODP_SCHED_SYNC_ORDERED)<br> > > + odp_schedule_release_ordered();<br> > > +<br> > > /* Done with queues for this round */<br> > > CU_ASSERT_FATAL(odp_queue_destroy(queue_grp1) == 0);<br> > > CU_ASSERT_FATAL(odp_queue_destroy(queue_grp2) == 0);<br> > > @@ -959,7 +967,6 @@ static void fill_queues(thread_args_t *args)<br> > > }<br> > ><br> > > ret = odp_queue_enq(queue, ev);<br> > > - CU_ASSERT_FATAL(ret == 0);<br> > ><br> > > if (ret)<br> > > odp_buffer_free(buf);<br> > > diff --git a/test/common_plat/validation/api/timer/timer.c b/test/common_plat/validation/api/timer/timer.c<br> > > index b7d84c64..362f33c8 100644<br> > > --- a/test/common_plat/validation/api/timer/timer.c<br> > > +++ b/test/common_plat/validation/api/timer/timer.c<br> > > @@ -297,8 +297,11 @@ static int worker_entrypoint(void *arg TEST_UNUSED)<br> > > struct timespec ts;<br> > > uint32_t nstale;<br> > > odp_timer_set_t timer_rc;<br> > > + char name[32];<br> > ><br> > > - queue = odp_queue_create("timer_queue", NULL);<br> > > + snprintf(name, sizeof(name), "timer_queue_%d", thr);<br> > > +<br> > > + queue = odp_queue_create(name, NULL);<br> > > if (queue == ODP_QUEUE_INVALID)<br> > > CU_FAIL_FATAL("Queue create failed");<br> > ><br> > > --<br> > > 2.12.1<br> > ><br> > </div></span></font> > </body> > </html>
First off, I'm not sure why you're seeing HTML since Gmail is set to plain text mode for me. On Tue, Mar 28, 2017 at 9:18 PM, Brian Brooks <brian.brooks@arm.com> wrote: > On 03/28 18:50:32, Bill Fischofer wrote: > > <html><head><meta http-equiv="Content-Type" content="text/html; > charset=utf-8"> > > <meta name="Generator" content="Microsoft Exchange Server"> > > <!-- converted from text --> > > <style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; > border-left: #800000 2px solid; } --></style></head> > > <body> > > <font size="2"><span style="font-size:10pt;"><div class="PlainText">This > part generates numerous checkpatch warnings and errors. Please<br> > > run checkpatch and correct for v2.<br> > > We ran checkpatch.pl and corrected the issues that made sense. We all > know that > checkpatch.pl is not perfect. Please point out the checkpatch.pl issues > that > are generated from this patch which are not false positives or not > destructive > to programmer readability. > Checkpatch does issue a number of spurious warnings relating to the use of volatile, etc. but the following seem legit: WARNING: Possible unwrapped commit description (prefer a maximum 75 chars per line) #11: respected, and (W)RR scheduling may be used within queues of the same priority. WARNING: 'absense' may be misspelled - perhaps 'absence'? #16: concurrent queues. LL/SC and CAS variants exist in cases where absense of CHECK: No space is necessary after a cast #171: FILE: platform/linux-generic/include/odp/api/plat/schedule_types.h:65: +#define ODP_SCHED_GROUP_INVALID ((odp_schedule_group_t) -1) CHECK: Alignment should match open parenthesis #225: FILE: platform/linux-generic/include/odp_atomic16.h:45: +static inline bool __atomic_compare_exchange_16(register __int128 *var, + __int128 *exp, register __int128 neu, CHECK: Alignment should match open parenthesis #271: FILE: platform/linux-generic/include/odp_atomic16.h:91: +static inline bool __atomic_compare_exchange_16_frail(register __int128 *var, + __int128 *exp, register __int128 neu, bool weak, CHECK: Alignment should match open parenthesis #393: FILE: platform/linux-generic/include/odp_atomic16.h:213: +static inline __int128 __atomic_fetch_or_16(__int128 *var, + __int128 mask, CHECK: multiple assignments should be avoided #856: FILE: platform/linux-generic/include/odp_llqueue.h:123: + llq->u.st.head = llq->u.st.tail = node; CHECK: multiple assignments should be avoided #965: FILE: platform/linux-generic/include/odp_llqueue.h:232: + llq->u.st.head = llq->u.st.tail = NULL; CHECK: multiple assignments should be avoided #1064: FILE: platform/linux-generic/include/odp_llqueue.h:331: + llq->u.st.head = llq->u.st.tail = NULL; CHECK: Please use a blank line after function/struct/union/enum declarations #1189: FILE: platform/linux-generic/include/odp_llsc.h:64: +} +#define ll32(a, b) ll((a), (b)) CHECK: Please use a blank line after function/struct/union/enum declarations #1205: FILE: platform/linux-generic/include/odp_llsc.h:80: +} +#define sc32(a, b, c) sc((a), (b), (c)) CHECK: Please use a blank line after function/struct/union/enum declarations #1220: FILE: platform/linux-generic/include/odp_llsc.h:95: +} +#define ll64(a, b) lld((a), (b)) CHECK: Please use a blank line after function/struct/union/enum declarations #1236: FILE: platform/linux-generic/include/odp_llsc.h:111: +} +#define sc64(a, b, c) scd((a), (b), (c)) CHECK: Please use a blank line after function/struct/union/enum declarations #1336: FILE: platform/linux-generic/include/odp_llsc.h:211: +} +#define ll64(a, b) ll((a), (b)) CHECK: Please use a blank line after function/struct/union/enum declarations #1357: FILE: platform/linux-generic/include/odp_llsc.h:232: +} +#define sc64(a, b, c) sc((a), (b), (c)) CHECK: Alignment should match open parenthesis #1981: FILE: platform/linux-generic/include/odp_schedule_ordered_internal.h:131: +void olock_unlock(const reorder_context_t *rctx, reorder_window_t *rwin, + uint32_t lock_index); WARNING: 'noone' may be misspelled - perhaps 'no one'? #2402: FILE: platform/linux-generic/odp_queue_scalable.c:95: + /* All remaining elements claimed, noone else can enqueue */ CHECK: spaces preferred around that '<<' (ctx:VxV) #2417: FILE: platform/linux-generic/odp_queue_scalable.c:110: + return 1<<i; ^ CHECK: Alignment should match open parenthesis #2438: FILE: platform/linux-generic/odp_queue_scalable.c:131: + shm = odp_shm_reserve(name, + ring_size * sizeof(odp_buffer_hdr_t *), CHECK: Alignment should match open parenthesis #2698: FILE: platform/linux-generic/odp_queue_scalable.c:391: + while (__atomic_load_n(&q->qschst.numevts, __ATOMIC_RELAXED) != 0 || + __atomic_load_n(&q->qschst.cur_ticket, __ATOMIC_RELAXED) != WARNING: macros should not use a trailing semicolon #3366: FILE: platform/linux-generic/odp_schedule_scalable.c:55: +#define __atomic_fetch_max(var, v, mo) do { \ + /* Evalulate 'v' once */ \ + __typeof__(v) tmp_v = (v); \ + __typeof__(*var) old_var = \ + __atomic_load_n((var), __ATOMIC_RELAXED); \ + while (tmp_v > old_var) { \ + /* Attempt to store 'v' in '*var' */ \ + if (__atomic_compare_exchange_n((var), &old_var, \ + tmp_v, true, (mo), \ + (mo))) \ + break; \ + /* Else failure, try again (with updated value of \ WARNING: Avoid unnecessary line continuations #3379: FILE: platform/linux-generic/odp_schedule_scalable.c:68: + */ \ CHECK: Alignment should match open parenthesis #3581: FILE: platform/linux-generic/odp_schedule_scalable.c:270: + LDXR8(&q->qschst.cur_ticket, + __ATOMIC_ACQUIRE) != ticket) CHECK: Alignment should match open parenthesis #3659: FILE: platform/linux-generic/odp_schedule_scalable.c:348: +static void sched_update_deq(sched_elem_t *q, + uint32_t actual, bool atomic) __attribute__((always_inline)); CHECK: Alignment should match open parenthesis #3661: FILE: platform/linux-generic/odp_schedule_scalable.c:350: +static inline void sched_update_deq(sched_elem_t *q, + uint32_t actual, bool atomic) CHECK: Alignment should match open parenthesis #3745: FILE: platform/linux-generic/odp_schedule_scalable.c:434: + LDXR8(&q->qschst.cur_ticket, + __ATOMIC_ACQUIRE) != ticket) CHECK: Alignment should match open parenthesis #3791: FILE: platform/linux-generic/odp_schedule_scalable.c:480: +static void sched_update_deq_sc(sched_elem_t *q, + uint32_t actual, bool atomic) __attribute__((always_inline)); CHECK: Alignment should match open parenthesis #3793: FILE: platform/linux-generic/odp_schedule_scalable.c:482: +static inline void sched_update_deq_sc(sched_elem_t *q, + uint32_t actual, bool atomic) CHECK: Alignment should match open parenthesis #3977: FILE: platform/linux-generic/odp_schedule_scalable.c:666: + sg_wanted = __atomic_load_n(&ts->sg_wanted[p], + __ATOMIC_ACQUIRE); CHECK: Alignment should match open parenthesis #3989: FILE: platform/linux-generic/odp_schedule_scalable.c:678: + insert_schedq_in_list(ts, + &sg->schedq[p * sg->xfactor + CHECK: Alignment should match open parenthesis #4002: FILE: platform/linux-generic/odp_schedule_scalable.c:691: + remove_schedq_from_list(ts, + &sg->schedq[p * sg->xfactor + x]); CHECK: multiple assignments should be avoided #4171: FILE: platform/linux-generic/odp_schedule_scalable.c:860: + ts->atomq = atomq = elem; CHECK: Alignment should match open parenthesis #4256: FILE: platform/linux-generic/odp_schedule_scalable.c:945: + if (odp_unlikely(__atomic_load_n(&rwin->turn, + __ATOMIC_ACQUIRE) != sn)) { CHECK: Alignment should match open parenthesis #4584: FILE: platform/linux-generic/odp_schedule_scalable.c:1273: + __atomic_store_n(&thread_state[thr].sg_sem, + 1, CHECK: Alignment should match open parenthesis #4656: FILE: platform/linux-generic/odp_schedule_scalable.c:1345: + shm = odp_shm_reserve(name ? name : "", + (sizeof(sched_group_t) + CHECK: No space is necessary after a cast #4664: FILE: platform/linux-generic/odp_schedule_scalable.c:1353: + sg = (sched_group_t *) odp_shm_addr(shm); CHECK: No space is necessary after a cast #4684: FILE: platform/linux-generic/odp_schedule_scalable.c:1373: + return (odp_schedule_group_t) (sgi); CHECK: Alignment should match open parenthesis #5054: FILE: platform/linux-generic/odp_schedule_scalable.c:1743: + if (odp_schedule_group_join(ODP_SCHED_GROUP_CONTROL, + &mask) != 0) { CHECK: Alignment should match open parenthesis #5060: FILE: platform/linux-generic/odp_schedule_scalable.c:1749: + if (odp_schedule_group_join(ODP_SCHED_GROUP_WORKER, + &mask) != 0) { CHECK: Alignment should match open parenthesis #5096: FILE: platform/linux-generic/odp_schedule_scalable.c:1785: + if (odp_schedule_group_leave(ODP_SCHED_GROUP_CONTROL, + &mask) != 0) CHECK: Alignment should match open parenthesis #5100: FILE: platform/linux-generic/odp_schedule_scalable.c:1789: + if (odp_schedule_group_leave(ODP_SCHED_GROUP_WORKER, + &mask) != 0) CHECK: Please use a blank line after function/struct/union/enum declarations #5185: FILE: platform/linux-generic/odp_schedule_scalable.c:1874: +} +static int thr_rem(odp_schedule_group_t group, int thr) CHECK: Please use a blank line after function/struct/union/enum declarations #5192: FILE: platform/linux-generic/odp_schedule_scalable.c:1881: +} +static int init_queue(uint32_t queue_index, CHECK: Please use a blank line after function/struct/union/enum declarations #5200: FILE: platform/linux-generic/odp_schedule_scalable.c:1889: +} +static void destroy_queue(uint32_t queue_index) CHECK: Please use a blank line after function/struct/union/enum declarations #5205: FILE: platform/linux-generic/odp_schedule_scalable.c:1894: +} +static int sched_queue(uint32_t queue_index) CHECK: Please use a blank line after function/struct/union/enum declarations #5211: FILE: platform/linux-generic/odp_schedule_scalable.c:1900: +} +static int ord_enq_multi(uint32_t queue_index, void *p_buf_hdr[], CHECK: No space is necessary after a cast #5314: FILE: platform/linux-generic/odp_schedule_scalable_ordered.c:38: + rwin = (reorder_window_t *) odp_shm_addr(*shm); CHECK: Alignment should match open parenthesis #5408: FILE: platform/linux-generic/odp_schedule_scalable_ordered.c:132: + if (__atomic_compare_exchange(&rwin->hc, + &old, /* Updated on failure */ CHECK: Please don't use multiple blank lines #5757: FILE: test/common_plat/performance/odp_sched_latency.c:824: + total: 0 errors, 54 warnings, 44 checks, 5742 lines checked NOTE: Ignored message types: BIT_MACRO COMPARISON_TO_NULL DEPRECATED_VARIABLE NEW_TYPEDEFS SPLIT_STRING SSCANF_TO_KSTRTO 0004-A-scalable-software-scheduler.patch has style problems, please review. If any of these errors are false positives, please report them to the maintainer, see CHECKPATCH in MAINTAINERS. > > <br> > > Also, this part introduces a number of errors that result in failure<br> > > to compile using clang. > > This is likely true. Is compilation with Clang a blocker? > Yes, it is. We require ODP modules to be compilable with either GCC or clang. > >
I had run the checkpatch.pl, but I did not see these issues. May be it was to do with the way I generated the patch. I will address the ones that make sense. Do we have to compile for 32b with gcc? From: Bill Fischofer [mailto:bill.fischofer@linaro.org] Sent: Tuesday, March 28, 2017 10:01 PM To: Brian Brooks Cc: lng-odp-forward; Ola Liljedahl; Kevin Wang; Honnappa Nagarahalli Subject: Re: [lng-odp] [API-NEXT 4/4] A scalable software scheduler First off, I'm not sure why you're seeing HTML since Gmail is set to plain text mode for me. On Tue, Mar 28, 2017 at 9:18 PM, Brian Brooks <brian.brooks@arm.com> wrote: On 03/28 18:50:32, Bill Fischofer wrote: > <html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"> > <meta name="Generator" content="Microsoft Exchange Server"> > <!-- converted from text --> > <style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style></head> > <body> > <font size="2"><span style="font-size:10pt;"><div class="PlainText">This part generates numerous checkpatch warnings and errors. Please<br> > run checkpatch and correct for v2.<br> We ran checkpatch.pl and corrected the issues that made sense. We all know that checkpatch.pl is not perfect. Please point out the checkpatch.pl issues that are generated from this patch which are not false positives or not destructive to programmer readability. Checkpatch does issue a number of spurious warnings relating to the use of volatile, etc. but the following seem legit: WARNING: Possible unwrapped commit description (prefer a maximum 75 chars per line) #11: respected, and (W)RR scheduling may be used within queues of the same priority. WARNING: 'absense' may be misspelled - perhaps 'absence'? #16: concurrent queues. LL/SC and CAS variants exist in cases where absense of CHECK: No space is necessary after a cast #171: FILE: platform/linux-generic/include/odp/api/plat/schedule_types.h:65: +#define ODP_SCHED_GROUP_INVALID ((odp_schedule_group_t) -1) CHECK: Alignment should match open parenthesis #225: FILE: platform/linux-generic/include/odp_atomic16.h:45: +static inline bool __atomic_compare_exchange_16(register __int128 *var, +__int128 *exp, register __int128 neu, CHECK: Alignment should match open parenthesis #271: FILE: platform/linux-generic/include/odp_atomic16.h:91: +static inline bool __atomic_compare_exchange_16_frail(register __int128 *var, +__int128 *exp, register __int128 neu, bool weak, CHECK: Alignment should match open parenthesis #393: FILE: platform/linux-generic/include/odp_atomic16.h:213: +static inline __int128 __atomic_fetch_or_16(__int128 *var, +__int128 mask, CHECK: multiple assignments should be avoided #856: FILE: platform/linux-generic/include/odp_llqueue.h:123: +llq->u.st.head = llq->u.st.tail = node; CHECK: multiple assignments should be avoided #965: FILE: platform/linux-generic/include/odp_llqueue.h:232: +llq->u.st.head = llq->u.st.tail = NULL; CHECK: multiple assignments should be avoided #1064: FILE: platform/linux-generic/include/odp_llqueue.h:331: +llq->u.st.head = llq->u.st.tail = NULL; CHECK: Please use a blank line after function/struct/union/enum declarations #1189: FILE: platform/linux-generic/include/odp_llsc.h:64: +} +#define ll32(a, b) ll((a), (b)) CHECK: Please use a blank line after function/struct/union/enum declarations #1205: FILE: platform/linux-generic/include/odp_llsc.h:80: +} +#define sc32(a, b, c) sc((a), (b), (c)) CHECK: Please use a blank line after function/struct/union/enum declarations #1220: FILE: platform/linux-generic/include/odp_llsc.h:95: +} +#define ll64(a, b) lld((a), (b)) CHECK: Please use a blank line after function/struct/union/enum declarations #1236: FILE: platform/linux-generic/include/odp_llsc.h:111: +} +#define sc64(a, b, c) scd((a), (b), (c)) CHECK: Please use a blank line after function/struct/union/enum declarations #1336: FILE: platform/linux-generic/include/odp_llsc.h:211: +} +#define ll64(a, b) ll((a), (b)) CHECK: Please use a blank line after function/struct/union/enum declarations #1357: FILE: platform/linux-generic/include/odp_llsc.h:232: +} +#define sc64(a, b, c) sc((a), (b), (c)) CHECK: Alignment should match open parenthesis #1981: FILE: platform/linux-generic/include/odp_schedule_ordered_internal.h:131: +void olock_unlock(const reorder_context_t *rctx, reorder_window_t *rwin, + uint32_t lock_index); WARNING: 'noone' may be misspelled - perhaps 'no one'? #2402: FILE: platform/linux-generic/odp_queue_scalable.c:95: +/* All remaining elements claimed, noone else can enqueue */ CHECK: spaces preferred around that '<<' (ctx:VxV) #2417: FILE: platform/linux-generic/odp_queue_scalable.c:110: +return 1<<i; ^ CHECK: Alignment should match open parenthesis #2438: FILE: platform/linux-generic/odp_queue_scalable.c:131: +shm = odp_shm_reserve(name, +ring_size * sizeof(odp_buffer_hdr_t *), CHECK: Alignment should match open parenthesis #2698: FILE: platform/linux-generic/odp_queue_scalable.c:391: +while (__atomic_load_n(&q->qschst.numevts, __ATOMIC_RELAXED) != 0 || +__atomic_load_n(&q->qschst.cur_ticket, __ATOMIC_RELAXED) != WARNING: macros should not use a trailing semicolon #3366: FILE: platform/linux-generic/odp_schedule_scalable.c:55: +#define __atomic_fetch_max(var, v, mo) do { \ +/* Evalulate 'v' once */ \ +__typeof__(v) tmp_v = (v); \ +__typeof__(*var) old_var = \ +__atomic_load_n((var), __ATOMIC_RELAXED); \ +while (tmp_v > old_var) { \ +/* Attempt to store 'v' in '*var' */ \ +if (__atomic_compare_exchange_n((var), &old_var, \ +tmp_v, true, (mo), \ +(mo))) \ +break; \ +/* Else failure, try again (with updated value of \ WARNING: Avoid unnecessary line continuations #3379: FILE: platform/linux-generic/odp_schedule_scalable.c:68: + */ \ CHECK: Alignment should match open parenthesis #3581: FILE: platform/linux-generic/odp_schedule_scalable.c:270: + LDXR8(&q->qschst.cur_ticket, + __ATOMIC_ACQUIRE) != ticket) CHECK: Alignment should match open parenthesis #3659: FILE: platform/linux-generic/odp_schedule_scalable.c:348: +static void sched_update_deq(sched_elem_t *q, +uint32_t actual, bool atomic) __attribute__((always_inline)); CHECK: Alignment should match open parenthesis #3661: FILE: platform/linux-generic/odp_schedule_scalable.c:350: +static inline void sched_update_deq(sched_elem_t *q, +uint32_t actual, bool atomic) CHECK: Alignment should match open parenthesis #3745: FILE: platform/linux-generic/odp_schedule_scalable.c:434: + LDXR8(&q->qschst.cur_ticket, + __ATOMIC_ACQUIRE) != ticket) CHECK: Alignment should match open parenthesis #3791: FILE: platform/linux-generic/odp_schedule_scalable.c:480: +static void sched_update_deq_sc(sched_elem_t *q, +uint32_t actual, bool atomic) __attribute__((always_inline)); CHECK: Alignment should match open parenthesis #3793: FILE: platform/linux-generic/odp_schedule_scalable.c:482: +static inline void sched_update_deq_sc(sched_elem_t *q, +uint32_t actual, bool atomic) CHECK: Alignment should match open parenthesis #3977: FILE: platform/linux-generic/odp_schedule_scalable.c:666: +sg_wanted = __atomic_load_n(&ts->sg_wanted[p], +__ATOMIC_ACQUIRE); CHECK: Alignment should match open parenthesis #3989: FILE: platform/linux-generic/odp_schedule_scalable.c:678: +insert_schedq_in_list(ts, +&sg->schedq[p * sg->xfactor + CHECK: Alignment should match open parenthesis #4002: FILE: platform/linux-generic/odp_schedule_scalable.c:691: +remove_schedq_from_list(ts, + &sg->schedq[p * sg->xfactor + x]); CHECK: multiple assignments should be avoided #4171: FILE: platform/linux-generic/odp_schedule_scalable.c:860: +ts->atomq = atomq = elem; CHECK: Alignment should match open parenthesis #4256: FILE: platform/linux-generic/odp_schedule_scalable.c:945: +if (odp_unlikely(__atomic_load_n(&rwin->turn, + __ATOMIC_ACQUIRE) != sn)) { CHECK: Alignment should match open parenthesis #4584: FILE: platform/linux-generic/odp_schedule_scalable.c:1273: +__atomic_store_n(&thread_state[thr].sg_sem, +1, CHECK: Alignment should match open parenthesis #4656: FILE: platform/linux-generic/odp_schedule_scalable.c:1345: +shm = odp_shm_reserve(name ? name : "", + (sizeof(sched_group_t) + CHECK: No space is necessary after a cast #4664: FILE: platform/linux-generic/odp_schedule_scalable.c:1353: +sg = (sched_group_t *) odp_shm_addr(shm); CHECK: No space is necessary after a cast #4684: FILE: platform/linux-generic/odp_schedule_scalable.c:1373: +return (odp_schedule_group_t) (sgi); CHECK: Alignment should match open parenthesis #5054: FILE: platform/linux-generic/odp_schedule_scalable.c:1743: +if (odp_schedule_group_join(ODP_SCHED_GROUP_CONTROL, +&mask) != 0) { CHECK: Alignment should match open parenthesis #5060: FILE: platform/linux-generic/odp_schedule_scalable.c:1749: +if (odp_schedule_group_join(ODP_SCHED_GROUP_WORKER, +&mask) != 0) { CHECK: Alignment should match open parenthesis #5096: FILE: platform/linux-generic/odp_schedule_scalable.c:1785: +if (odp_schedule_group_leave(ODP_SCHED_GROUP_CONTROL, +&mask) != 0) CHECK: Alignment should match open parenthesis #5100: FILE: platform/linux-generic/odp_schedule_scalable.c:1789: +if (odp_schedule_group_leave(ODP_SCHED_GROUP_WORKER, +&mask) != 0) CHECK: Please use a blank line after function/struct/union/enum declarations #5185: FILE: platform/linux-generic/odp_schedule_scalable.c:1874: +} +static int thr_rem(odp_schedule_group_t group, int thr) CHECK: Please use a blank line after function/struct/union/enum declarations #5192: FILE: platform/linux-generic/odp_schedule_scalable.c:1881: +} +static int init_queue(uint32_t queue_index, CHECK: Please use a blank line after function/struct/union/enum declarations #5200: FILE: platform/linux-generic/odp_schedule_scalable.c:1889: +} +static void destroy_queue(uint32_t queue_index) CHECK: Please use a blank line after function/struct/union/enum declarations #5205: FILE: platform/linux-generic/odp_schedule_scalable.c:1894: +} +static int sched_queue(uint32_t queue_index) CHECK: Please use a blank line after function/struct/union/enum declarations #5211: FILE: platform/linux-generic/odp_schedule_scalable.c:1900: +} +static int ord_enq_multi(uint32_t queue_index, void *p_buf_hdr[], CHECK: No space is necessary after a cast #5314: FILE: platform/linux-generic/odp_schedule_scalable_ordered.c:38: +rwin = (reorder_window_t *) odp_shm_addr(*shm); CHECK: Alignment should match open parenthesis #5408: FILE: platform/linux-generic/odp_schedule_scalable_ordered.c:132: +if (__atomic_compare_exchange(&rwin->hc, +&old, /* Updated on failure */ CHECK: Please don't use multiple blank lines #5757: FILE: test/common_plat/performance/odp_sched_latency.c:824: + total: 0 errors, 54 warnings, 44 checks, 5742 lines checked NOTE: Ignored message types: BIT_MACRO COMPARISON_TO_NULL DEPRECATED_VARIABLE NEW_TYPEDEFS SPLIT_STRING SSCANF_TO_KSTRTO 0004-A-scalable-software-scheduler.patch has style problems, please review. If any of these errors are false positives, please report them to the maintainer, see CHECKPATCH in MAINTAINERS. > <br> > Also, this part introduces a number of errors that result in failure<br> > to compile using clang. This is likely true. Is compilation with Clang a blocker? Yes, it is. We require ODP modules to be compilable with either GCC or clang. IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
On Tue, Mar 28, 2017 at 10:28 PM, Honnappa Nagarahalli < Honnappa.Nagarahalli@arm.com> wrote: > I had run the checkpatch.pl, but I did not see these issues. May be it > was to do with the way I generated the patch. I will address the ones that > make sense. > I did a git am for your patches and then regenerated them for checkpatch testing via: git format-patch origin/api-next --subject-prefix="API-NEXT PATCH" > > Do we have to compile for 32b with gcc? > Yes, we require that ODP code compile and run in both 32-bit and 64-bit mode. > > From: Bill Fischofer [mailto:bill.fischofer@linaro.org] > Sent: Tuesday, March 28, 2017 10:01 PM > To: Brian Brooks > Cc: lng-odp-forward; Ola Liljedahl; Kevin Wang; Honnappa Nagarahalli > Subject: Re: [lng-odp] [API-NEXT 4/4] A scalable software scheduler > > First off, I'm not sure why you're seeing HTML since Gmail is set to plain > text mode for me. > > On Tue, Mar 28, 2017 at 9:18 PM, Brian Brooks <brian.brooks@arm.com> > wrote: > On 03/28 18:50:32, Bill Fischofer wrote: > > <html><head><meta http-equiv="Content-Type" content="text/html; > charset=utf-8"> > > <meta name="Generator" content="Microsoft Exchange Server"> > > <!-- converted from text --> > > <style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; > border-left: #800000 2px solid; } --></style></head> > > <body> > > <font size="2"><span style="font-size:10pt;"><div class="PlainText">This > part generates numerous checkpatch warnings and errors. Please<br> > > run checkpatch and correct for v2.<br> > > We ran checkpatch.pl and corrected the issues that made sense. We all > know that > checkpatch.pl is not perfect. Please point out the checkpatch.pl issues > that > are generated from this patch which are not false positives or not > destructive > to programmer readability. > > Checkpatch does issue a number of spurious warnings relating to the use of > volatile, etc. but the following seem legit: > > WARNING: Possible unwrapped commit description (prefer a maximum 75 chars > per line) > #11: > respected, and (W)RR scheduling may be used within queues of the same > priority. > > WARNING: 'absense' may be misspelled - perhaps 'absence'? > #16: > concurrent queues. LL/SC and CAS variants exist in cases where absense of > > CHECK: No space is necessary after a cast > #171: FILE: platform/linux-generic/include/odp/api/plat/schedule_ > types.h:65: > +#define ODP_SCHED_GROUP_INVALID ((odp_schedule_group_t) -1) > > CHECK: Alignment should match open parenthesis > #225: FILE: platform/linux-generic/include/odp_atomic16.h:45: > +static inline bool __atomic_compare_exchange_16(register __int128 *var, > +__int128 *exp, register __int128 neu, > > CHECK: Alignment should match open parenthesis > #271: FILE: platform/linux-generic/include/odp_atomic16.h:91: > +static inline bool __atomic_compare_exchange_16_frail(register __int128 > *var, > +__int128 *exp, register __int128 neu, bool weak, > > CHECK: Alignment should match open parenthesis > #393: FILE: platform/linux-generic/include/odp_atomic16.h:213: > +static inline __int128 __atomic_fetch_or_16(__int128 *var, > +__int128 mask, > > CHECK: multiple assignments should be avoided > #856: FILE: platform/linux-generic/include/odp_llqueue.h:123: > +llq->u.st.head = llq->u.st.tail = node; > > CHECK: multiple assignments should be avoided > #965: FILE: platform/linux-generic/include/odp_llqueue.h:232: > +llq->u.st.head = llq->u.st.tail = NULL; > > CHECK: multiple assignments should be avoided > #1064: FILE: platform/linux-generic/include/odp_llqueue.h:331: > +llq->u.st.head = llq->u.st.tail = NULL; > > > CHECK: Please use a blank line after function/struct/union/enum > declarations > #1189: FILE: platform/linux-generic/include/odp_llsc.h:64: > +} > +#define ll32(a, b) ll((a), (b)) > > > CHECK: Please use a blank line after function/struct/union/enum > declarations > #1205: FILE: platform/linux-generic/include/odp_llsc.h:80: > +} > +#define sc32(a, b, c) sc((a), (b), (c)) > > > CHECK: Please use a blank line after function/struct/union/enum > declarations > #1220: FILE: platform/linux-generic/include/odp_llsc.h:95: > +} > +#define ll64(a, b) lld((a), (b)) > > > CHECK: Please use a blank line after function/struct/union/enum > declarations > #1236: FILE: platform/linux-generic/include/odp_llsc.h:111: > +} > +#define sc64(a, b, c) scd((a), (b), (c)) > > > CHECK: Please use a blank line after function/struct/union/enum > declarations > #1336: FILE: platform/linux-generic/include/odp_llsc.h:211: > +} > +#define ll64(a, b) ll((a), (b)) > > > CHECK: Please use a blank line after function/struct/union/enum > declarations > #1357: FILE: platform/linux-generic/include/odp_llsc.h:232: > +} > +#define sc64(a, b, c) sc((a), (b), (c)) > > > CHECK: Alignment should match open parenthesis > #1981: FILE: platform/linux-generic/include/odp_schedule_ordered_ > internal.h:131: > +void olock_unlock(const reorder_context_t *rctx, reorder_window_t *rwin, > + uint32_t lock_index); > > > WARNING: 'noone' may be misspelled - perhaps 'no one'? > #2402: FILE: platform/linux-generic/odp_queue_scalable.c:95: > +/* All remaining elements claimed, noone else can enqueue */ > > CHECK: spaces preferred around that '<<' (ctx:VxV) > #2417: FILE: platform/linux-generic/odp_queue_scalable.c:110: > +return 1<<i; > ^ > > CHECK: Alignment should match open parenthesis > #2438: FILE: platform/linux-generic/odp_queue_scalable.c:131: > +shm = odp_shm_reserve(name, > +ring_size * sizeof(odp_buffer_hdr_t *), > > CHECK: Alignment should match open parenthesis > #2698: FILE: platform/linux-generic/odp_queue_scalable.c:391: > +while (__atomic_load_n(&q->qschst.numevts, __ATOMIC_RELAXED) != 0 || > +__atomic_load_n(&q->qschst.cur_ticket, __ATOMIC_RELAXED) != > > > WARNING: macros should not use a trailing semicolon > #3366: FILE: platform/linux-generic/odp_schedule_scalable.c:55: > +#define __atomic_fetch_max(var, v, mo) do { \ > +/* Evalulate 'v' once */ \ > +__typeof__(v) tmp_v = (v); \ > +__typeof__(*var) old_var = \ > +__atomic_load_n((var), __ATOMIC_RELAXED); \ > +while (tmp_v > old_var) { \ > +/* Attempt to store 'v' in '*var' */ \ > +if (__atomic_compare_exchange_n((var), &old_var, \ > +tmp_v, true, (mo), \ > +(mo))) \ > +break; \ > +/* Else failure, try again (with updated value of \ > > WARNING: Avoid unnecessary line continuations > #3379: FILE: platform/linux-generic/odp_schedule_scalable.c:68: > + */ \ > > CHECK: Alignment should match open parenthesis > #3581: FILE: platform/linux-generic/odp_schedule_scalable.c:270: > + LDXR8(&q->qschst.cur_ticket, > + __ATOMIC_ACQUIRE) != ticket) > > CHECK: Alignment should match open parenthesis > #3659: FILE: platform/linux-generic/odp_schedule_scalable.c:348: > +static void sched_update_deq(sched_elem_t *q, > +uint32_t actual, bool atomic) __attribute__((always_inline)); > > CHECK: Alignment should match open parenthesis > #3661: FILE: platform/linux-generic/odp_schedule_scalable.c:350: > +static inline void sched_update_deq(sched_elem_t *q, > +uint32_t actual, bool atomic) > > CHECK: Alignment should match open parenthesis > #3745: FILE: platform/linux-generic/odp_schedule_scalable.c:434: > + LDXR8(&q->qschst.cur_ticket, > + __ATOMIC_ACQUIRE) != ticket) > > CHECK: Alignment should match open parenthesis > #3791: FILE: platform/linux-generic/odp_schedule_scalable.c:480: > +static void sched_update_deq_sc(sched_elem_t *q, > +uint32_t actual, bool atomic) __attribute__((always_inline)); > > CHECK: Alignment should match open parenthesis > #3793: FILE: platform/linux-generic/odp_schedule_scalable.c:482: > +static inline void sched_update_deq_sc(sched_elem_t *q, > +uint32_t actual, bool atomic) > > CHECK: Alignment should match open parenthesis > #3977: FILE: platform/linux-generic/odp_schedule_scalable.c:666: > +sg_wanted = __atomic_load_n(&ts->sg_wanted[p], > +__ATOMIC_ACQUIRE); > > CHECK: Alignment should match open parenthesis > #3989: FILE: platform/linux-generic/odp_schedule_scalable.c:678: > +insert_schedq_in_list(ts, > +&sg->schedq[p * sg->xfactor + > > CHECK: Alignment should match open parenthesis > #4002: FILE: platform/linux-generic/odp_schedule_scalable.c:691: > +remove_schedq_from_list(ts, > + &sg->schedq[p * sg->xfactor + x]); > > CHECK: multiple assignments should be avoided > #4171: FILE: platform/linux-generic/odp_schedule_scalable.c:860: > +ts->atomq = atomq = elem; > > CHECK: Alignment should match open parenthesis > #4256: FILE: platform/linux-generic/odp_schedule_scalable.c:945: > +if (odp_unlikely(__atomic_load_n(&rwin->turn, > + __ATOMIC_ACQUIRE) != sn)) { > > > CHECK: Alignment should match open parenthesis > #4584: FILE: platform/linux-generic/odp_schedule_scalable.c:1273: > +__atomic_store_n(&thread_state[thr].sg_sem, > +1, > > CHECK: Alignment should match open parenthesis > #4656: FILE: platform/linux-generic/odp_schedule_scalable.c:1345: > +shm = odp_shm_reserve(name ? name : "", > + (sizeof(sched_group_t) + > > CHECK: No space is necessary after a cast > #4664: FILE: platform/linux-generic/odp_schedule_scalable.c:1353: > +sg = (sched_group_t *) odp_shm_addr(shm); > > CHECK: No space is necessary after a cast > #4684: FILE: platform/linux-generic/odp_schedule_scalable.c:1373: > +return (odp_schedule_group_t) (sgi); > > CHECK: Alignment should match open parenthesis > #5054: FILE: platform/linux-generic/odp_schedule_scalable.c:1743: > +if (odp_schedule_group_join(ODP_SCHED_GROUP_CONTROL, > +&mask) != 0) { > > CHECK: Alignment should match open parenthesis > #5060: FILE: platform/linux-generic/odp_schedule_scalable.c:1749: > +if (odp_schedule_group_join(ODP_SCHED_GROUP_WORKER, > +&mask) != 0) { > > CHECK: Alignment should match open parenthesis > #5096: FILE: platform/linux-generic/odp_schedule_scalable.c:1785: > +if (odp_schedule_group_leave(ODP_SCHED_GROUP_CONTROL, > +&mask) != 0) > > CHECK: Alignment should match open parenthesis > #5100: FILE: platform/linux-generic/odp_schedule_scalable.c:1789: > +if (odp_schedule_group_leave(ODP_SCHED_GROUP_WORKER, > +&mask) != 0) > > CHECK: Please use a blank line after function/struct/union/enum > declarations > #5185: FILE: platform/linux-generic/odp_schedule_scalable.c:1874: > +} > +static int thr_rem(odp_schedule_group_t group, int thr) > > CHECK: Please use a blank line after function/struct/union/enum > declarations > #5192: FILE: platform/linux-generic/odp_schedule_scalable.c:1881: > +} > +static int init_queue(uint32_t queue_index, > > CHECK: Please use a blank line after function/struct/union/enum > declarations > #5200: FILE: platform/linux-generic/odp_schedule_scalable.c:1889: > +} > +static void destroy_queue(uint32_t queue_index) > > CHECK: Please use a blank line after function/struct/union/enum > declarations > #5205: FILE: platform/linux-generic/odp_schedule_scalable.c:1894: > +} > +static int sched_queue(uint32_t queue_index) > > CHECK: Please use a blank line after function/struct/union/enum > declarations > #5211: FILE: platform/linux-generic/odp_schedule_scalable.c:1900: > +} > +static int ord_enq_multi(uint32_t queue_index, void *p_buf_hdr[], > > > CHECK: No space is necessary after a cast > #5314: FILE: platform/linux-generic/odp_schedule_scalable_ordered.c:38: > +rwin = (reorder_window_t *) odp_shm_addr(*shm); > > CHECK: Alignment should match open parenthesis > #5408: FILE: platform/linux-generic/odp_schedule_scalable_ordered.c:132: > +if (__atomic_compare_exchange(&rwin->hc, > +&old, /* Updated on failure */ > > CHECK: Please don't use multiple blank lines > #5757: FILE: test/common_plat/performance/odp_sched_latency.c:824: > > + > > total: 0 errors, 54 warnings, 44 checks, 5742 lines checked > > NOTE: Ignored message types: BIT_MACRO COMPARISON_TO_NULL > DEPRECATED_VARIABLE NEW_TYPEDEFS SPLIT_STRING SSCANF_TO_KSTRTO > > 0004-A-scalable-software-scheduler.patch has style problems, please > review. > > If any of these errors are false positives, please report > them to the maintainer, see CHECKPATCH in MAINTAINERS. > > > > <br> > > Also, this part introduces a number of errors that result in failure<br> > > to compile using clang. > > This is likely true. Is compilation with Clang a blocker? > > Yes, it is. We require ODP modules to be compilable with either GCC or > clang. > > > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. >
isn't queue size connected to number elements in the pool? Maxim. On 29 March 2017 at 04:55, Brian Brooks <brian.brooks@arm.com> wrote: > On 03/28 19:18:37, Bill Fischofer wrote: > > <html><head><meta http-equiv="Content-Type" content="text/html; > charset=utf-8"> > > It is infinitely better to do patch review in plain text rather > than HTML. I thought this was a plain text mailing list? > > > <meta name="Generator" content="Microsoft Exchange Server"> > > <!-- converted from text --> > > <style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; > border-left: #800000 2px solid; } --></style></head> > > <body> > > <font size="2"><span style="font-size:10pt;"><div class="PlainText">On > Tue, Mar 28, 2017 at 2:23 PM, Brian Brooks <brian.brooks@arm.com> > wrote:<br> > > > Signed-off-by: Brian Brooks <brian.brooks@arm.com><br> > > > ---<br> > > > include/odp/api/spec/queue.h | 5 +++++<br> > > > 1 file changed, 5 insertions(+)<br> > > ><br> > > > diff --git a/include/odp/api/spec/queue.h > b/include/odp/api/spec/queue.h<br> > > > index 7972feac..1cec4773 100644<br> > > > --- a/include/odp/api/spec/queue.h<br> > > > +++ b/include/odp/api/spec/queue.h<br> > > > @@ -124,6 +124,11 @@ typedef struct odp_queue_param_t {<br> > > > * the > queue type. */<br> > > > odp_queue_type_t > type;<br> > > ><br> > > > + /** Queue size<br> > > > + *<br> > > > + * Indicates > the max ring size of the ring buffer. */<br> > > > + uint32_t ring_size;<br> > > <br> > > ODP queues have historically been of unspecified size. If we're going<br> > > to introduce the notion of explicitly limited sized queues this has<br> > > additional implications.<br> > > > > First, ring_size is an inappropriate choice of name here since a ring<br> > > is an implementation model, not a specification. The documentation<br> > > says "Queue size", so<br> > > <br> > > uint32_t size;<br> > > <br> > > is sufficient here. > > Agree, will change 'ring_size' to a better name. > > > We should document that size = 0 requests a queue<br> > > of default size (which may be unbounded).<br> > > Unbounded size is not practical or possible. Can we agree that 0 means > that the default aka ODP_CONFIG_QUEUE_SIZE is used? Should we allow for > greater than ODP_CONFIG_QUEUE_SIZE? E.g. 10,000 queue depth is also not > practical or possible. Perhaps we need a max which also acts as the > default. > > > <br> > > Second, if we're going to allow a queue size to be specified then > this<br> > > needs to be added as an output to odp_queue_capability() > > OK, will look into that. > > > so the<br> > > application knows the max_size supported (again 0 = unbounded).<br> > > <br> > > A larger question, however, is why is this being introduced at all<br> > > since this field is only used in the modified<br> > > odph_cuckoo_table_create() helper routine and this, in turn, is only<br> > > used within the cuckootable test module? This seems an extraneous and<br> > > unnecessary change and has no relationship to the rest of this patch<br> > > series.<br> > > AFAIK, the cuckoo unit test enqueues too many events (millions) to a queue. > That sounds like it makes no sense, but is an example of how anything is > possible. > > > <br> > > So Parts 1 and 3 of this series don't seem to have anything to do > with<br> > > the scalable scheduler. As a minor point, the order of these needs to<br> > > be reversed to preserve bisectability since Part 1 can't reference > the<br> > > new field before Part 3 defines it.<br> > > Agree that there are 2 independent sets of patches, but the order is needed > in order to get a sane `make check' on ARM-based chips. Without these fixes > going in first, we have no way of knowing whether the scalable scheduler > patches > caused the issue or not. I can break these into 2 separate sets of patches. > > > <br> > > > +<br> > > > /** Enqueue > mode<br> > > > *<br> > > > * > Default value for both queue types is ODP_QUEUE_OP_MT. Application<br> > > > --<br> > > > 2.12.1<br> > > ><br> > > </div></span></font> > > </body> > > </html> >
1. Prefix has to be "PATCH API-NEXT" not "API-NEXT" 2. Please remove ifdefs. And compile all variants of schedules into the library. Code needs to be compile each time. Maxim. On 29 March 2017 at 06:30, Bill Fischofer <bill.fischofer@linaro.org> wrote: > On Tue, Mar 28, 2017 at 10:28 PM, Honnappa Nagarahalli < > Honnappa.Nagarahalli@arm.com> wrote: > > > I had run the checkpatch.pl, but I did not see these issues. May be it > > was to do with the way I generated the patch. I will address the ones > that > > make sense. > > > > I did a git am for your patches and then regenerated them for checkpatch > testing via: > git format-patch origin/api-next --subject-prefix="API-NEXT PATCH" > > > > > > Do we have to compile for 32b with gcc? > > > > Yes, we require that ODP code compile and run in both 32-bit and 64-bit > mode. > > > > > > From: Bill Fischofer [mailto:bill.fischofer@linaro.org] > > Sent: Tuesday, March 28, 2017 10:01 PM > > To: Brian Brooks > > Cc: lng-odp-forward; Ola Liljedahl; Kevin Wang; Honnappa Nagarahalli > > Subject: Re: [lng-odp] [API-NEXT 4/4] A scalable software scheduler > > > > First off, I'm not sure why you're seeing HTML since Gmail is set to > plain > > text mode for me. > > > > On Tue, Mar 28, 2017 at 9:18 PM, Brian Brooks <brian.brooks@arm.com> > > wrote: > > On 03/28 18:50:32, Bill Fischofer wrote: > > > <html><head><meta http-equiv="Content-Type" content="text/html; > > charset=utf-8"> > > > <meta name="Generator" content="Microsoft Exchange Server"> > > > <!-- converted from text --> > > > <style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; > > border-left: #800000 2px solid; } --></style></head> > > > <body> > > > <font size="2"><span style="font-size:10pt;"><div > class="PlainText">This > > part generates numerous checkpatch warnings and errors. Please<br> > > > run checkpatch and correct for v2.<br> > > > > We ran checkpatch.pl and corrected the issues that made sense. We all > > know that > > checkpatch.pl is not perfect. Please point out the checkpatch.pl issues > > that > > are generated from this patch which are not false positives or not > > destructive > > to programmer readability. > > > > Checkpatch does issue a number of spurious warnings relating to the use > of > > volatile, etc. but the following seem legit: > > > > WARNING: Possible unwrapped commit description (prefer a maximum 75 chars > > per line) > > #11: > > respected, and (W)RR scheduling may be used within queues of the same > > priority. > > > > WARNING: 'absense' may be misspelled - perhaps 'absence'? > > #16: > > concurrent queues. LL/SC and CAS variants exist in cases where absense of > > > > CHECK: No space is necessary after a cast > > #171: FILE: platform/linux-generic/include/odp/api/plat/schedule_ > > types.h:65: > > +#define ODP_SCHED_GROUP_INVALID ((odp_schedule_group_t) -1) > > > > CHECK: Alignment should match open parenthesis > > #225: FILE: platform/linux-generic/include/odp_atomic16.h:45: > > +static inline bool __atomic_compare_exchange_16(register __int128 *var, > > +__int128 *exp, register __int128 neu, > > > > CHECK: Alignment should match open parenthesis > > #271: FILE: platform/linux-generic/include/odp_atomic16.h:91: > > +static inline bool __atomic_compare_exchange_16_frail(register __int128 > > *var, > > +__int128 *exp, register __int128 neu, bool weak, > > > > CHECK: Alignment should match open parenthesis > > #393: FILE: platform/linux-generic/include/odp_atomic16.h:213: > > +static inline __int128 __atomic_fetch_or_16(__int128 *var, > > +__int128 mask, > > > > CHECK: multiple assignments should be avoided > > #856: FILE: platform/linux-generic/include/odp_llqueue.h:123: > > +llq->u.st.head = llq->u.st.tail = node; > > > > CHECK: multiple assignments should be avoided > > #965: FILE: platform/linux-generic/include/odp_llqueue.h:232: > > +llq->u.st.head = llq->u.st.tail = NULL; > > > > CHECK: multiple assignments should be avoided > > #1064: FILE: platform/linux-generic/include/odp_llqueue.h:331: > > +llq->u.st.head = llq->u.st.tail = NULL; > > > > > > CHECK: Please use a blank line after function/struct/union/enum > > declarations > > #1189: FILE: platform/linux-generic/include/odp_llsc.h:64: > > +} > > +#define ll32(a, b) ll((a), (b)) > > > > > > CHECK: Please use a blank line after function/struct/union/enum > > declarations > > #1205: FILE: platform/linux-generic/include/odp_llsc.h:80: > > +} > > +#define sc32(a, b, c) sc((a), (b), (c)) > > > > > > CHECK: Please use a blank line after function/struct/union/enum > > declarations > > #1220: FILE: platform/linux-generic/include/odp_llsc.h:95: > > +} > > +#define ll64(a, b) lld((a), (b)) > > > > > > CHECK: Please use a blank line after function/struct/union/enum > > declarations > > #1236: FILE: platform/linux-generic/include/odp_llsc.h:111: > > +} > > +#define sc64(a, b, c) scd((a), (b), (c)) > > > > > > CHECK: Please use a blank line after function/struct/union/enum > > declarations > > #1336: FILE: platform/linux-generic/include/odp_llsc.h:211: > > +} > > +#define ll64(a, b) ll((a), (b)) > > > > > > CHECK: Please use a blank line after function/struct/union/enum > > declarations > > #1357: FILE: platform/linux-generic/include/odp_llsc.h:232: > > +} > > +#define sc64(a, b, c) sc((a), (b), (c)) > > > > > > CHECK: Alignment should match open parenthesis > > #1981: FILE: platform/linux-generic/include/odp_schedule_ordered_ > > internal.h:131: > > +void olock_unlock(const reorder_context_t *rctx, reorder_window_t *rwin, > > + uint32_t lock_index); > > > > > > WARNING: 'noone' may be misspelled - perhaps 'no one'? > > #2402: FILE: platform/linux-generic/odp_queue_scalable.c:95: > > +/* All remaining elements claimed, noone else can enqueue */ > > > > CHECK: spaces preferred around that '<<' (ctx:VxV) > > #2417: FILE: platform/linux-generic/odp_queue_scalable.c:110: > > +return 1<<i; > > ^ > > > > CHECK: Alignment should match open parenthesis > > #2438: FILE: platform/linux-generic/odp_queue_scalable.c:131: > > +shm = odp_shm_reserve(name, > > +ring_size * sizeof(odp_buffer_hdr_t *), > > > > CHECK: Alignment should match open parenthesis > > #2698: FILE: platform/linux-generic/odp_queue_scalable.c:391: > > +while (__atomic_load_n(&q->qschst.numevts, __ATOMIC_RELAXED) != 0 || > > +__atomic_load_n(&q->qschst.cur_ticket, __ATOMIC_RELAXED) != > > > > > > WARNING: macros should not use a trailing semicolon > > #3366: FILE: platform/linux-generic/odp_schedule_scalable.c:55: > > +#define __atomic_fetch_max(var, v, mo) do { \ > > +/* Evalulate 'v' once */ \ > > +__typeof__(v) tmp_v = (v); \ > > +__typeof__(*var) old_var = \ > > +__atomic_load_n((var), __ATOMIC_RELAXED); \ > > +while (tmp_v > old_var) { \ > > +/* Attempt to store 'v' in '*var' */ \ > > +if (__atomic_compare_exchange_n((var), &old_var, \ > > +tmp_v, true, (mo), \ > > +(mo))) \ > > +break; \ > > +/* Else failure, try again (with updated value of \ > > > > WARNING: Avoid unnecessary line continuations > > #3379: FILE: platform/linux-generic/odp_schedule_scalable.c:68: > > + */ \ > > > > CHECK: Alignment should match open parenthesis > > #3581: FILE: platform/linux-generic/odp_schedule_scalable.c:270: > > + LDXR8(&q->qschst.cur_ticket, > > + __ATOMIC_ACQUIRE) != ticket) > > > > CHECK: Alignment should match open parenthesis > > #3659: FILE: platform/linux-generic/odp_schedule_scalable.c:348: > > +static void sched_update_deq(sched_elem_t *q, > > +uint32_t actual, bool atomic) __attribute__((always_inline)); > > > > CHECK: Alignment should match open parenthesis > > #3661: FILE: platform/linux-generic/odp_schedule_scalable.c:350: > > +static inline void sched_update_deq(sched_elem_t *q, > > +uint32_t actual, bool atomic) > > > > CHECK: Alignment should match open parenthesis > > #3745: FILE: platform/linux-generic/odp_schedule_scalable.c:434: > > + LDXR8(&q->qschst.cur_ticket, > > + __ATOMIC_ACQUIRE) != ticket) > > > > CHECK: Alignment should match open parenthesis > > #3791: FILE: platform/linux-generic/odp_schedule_scalable.c:480: > > +static void sched_update_deq_sc(sched_elem_t *q, > > +uint32_t actual, bool atomic) __attribute__((always_inline)); > > > > CHECK: Alignment should match open parenthesis > > #3793: FILE: platform/linux-generic/odp_schedule_scalable.c:482: > > +static inline void sched_update_deq_sc(sched_elem_t *q, > > +uint32_t actual, bool atomic) > > > > CHECK: Alignment should match open parenthesis > > #3977: FILE: platform/linux-generic/odp_schedule_scalable.c:666: > > +sg_wanted = __atomic_load_n(&ts->sg_wanted[p], > > +__ATOMIC_ACQUIRE); > > > > CHECK: Alignment should match open parenthesis > > #3989: FILE: platform/linux-generic/odp_schedule_scalable.c:678: > > +insert_schedq_in_list(ts, > > +&sg->schedq[p * sg->xfactor + > > > > CHECK: Alignment should match open parenthesis > > #4002: FILE: platform/linux-generic/odp_schedule_scalable.c:691: > > +remove_schedq_from_list(ts, > > + &sg->schedq[p * sg->xfactor + x]); > > > > CHECK: multiple assignments should be avoided > > #4171: FILE: platform/linux-generic/odp_schedule_scalable.c:860: > > +ts->atomq = atomq = elem; > > > > CHECK: Alignment should match open parenthesis > > #4256: FILE: platform/linux-generic/odp_schedule_scalable.c:945: > > +if (odp_unlikely(__atomic_load_n(&rwin->turn, > > + __ATOMIC_ACQUIRE) != sn)) { > > > > > > CHECK: Alignment should match open parenthesis > > #4584: FILE: platform/linux-generic/odp_schedule_scalable.c:1273: > > +__atomic_store_n(&thread_state[thr].sg_sem, > > +1, > > > > CHECK: Alignment should match open parenthesis > > #4656: FILE: platform/linux-generic/odp_schedule_scalable.c:1345: > > +shm = odp_shm_reserve(name ? name : "", > > + (sizeof(sched_group_t) + > > > > CHECK: No space is necessary after a cast > > #4664: FILE: platform/linux-generic/odp_schedule_scalable.c:1353: > > +sg = (sched_group_t *) odp_shm_addr(shm); > > > > CHECK: No space is necessary after a cast > > #4684: FILE: platform/linux-generic/odp_schedule_scalable.c:1373: > > +return (odp_schedule_group_t) (sgi); > > > > CHECK: Alignment should match open parenthesis > > #5054: FILE: platform/linux-generic/odp_schedule_scalable.c:1743: > > +if (odp_schedule_group_join(ODP_SCHED_GROUP_CONTROL, > > +&mask) != 0) { > > > > CHECK: Alignment should match open parenthesis > > #5060: FILE: platform/linux-generic/odp_schedule_scalable.c:1749: > > +if (odp_schedule_group_join(ODP_SCHED_GROUP_WORKER, > > +&mask) != 0) { > > > > CHECK: Alignment should match open parenthesis > > #5096: FILE: platform/linux-generic/odp_schedule_scalable.c:1785: > > +if (odp_schedule_group_leave(ODP_SCHED_GROUP_CONTROL, > > +&mask) != 0) > > > > CHECK: Alignment should match open parenthesis > > #5100: FILE: platform/linux-generic/odp_schedule_scalable.c:1789: > > +if (odp_schedule_group_leave(ODP_SCHED_GROUP_WORKER, > > +&mask) != 0) > > > > CHECK: Please use a blank line after function/struct/union/enum > > declarations > > #5185: FILE: platform/linux-generic/odp_schedule_scalable.c:1874: > > +} > > +static int thr_rem(odp_schedule_group_t group, int thr) > > > > CHECK: Please use a blank line after function/struct/union/enum > > declarations > > #5192: FILE: platform/linux-generic/odp_schedule_scalable.c:1881: > > +} > > +static int init_queue(uint32_t queue_index, > > > > CHECK: Please use a blank line after function/struct/union/enum > > declarations > > #5200: FILE: platform/linux-generic/odp_schedule_scalable.c:1889: > > +} > > +static void destroy_queue(uint32_t queue_index) > > > > CHECK: Please use a blank line after function/struct/union/enum > > declarations > > #5205: FILE: platform/linux-generic/odp_schedule_scalable.c:1894: > > +} > > +static int sched_queue(uint32_t queue_index) > > > > CHECK: Please use a blank line after function/struct/union/enum > > declarations > > #5211: FILE: platform/linux-generic/odp_schedule_scalable.c:1900: > > +} > > +static int ord_enq_multi(uint32_t queue_index, void *p_buf_hdr[], > > > > > > CHECK: No space is necessary after a cast > > #5314: FILE: platform/linux-generic/odp_schedule_scalable_ordered.c:38: > > +rwin = (reorder_window_t *) odp_shm_addr(*shm); > > > > CHECK: Alignment should match open parenthesis > > #5408: FILE: platform/linux-generic/odp_schedule_scalable_ordered.c:132: > > +if (__atomic_compare_exchange(&rwin->hc, > > +&old, /* Updated on failure */ > > > > CHECK: Please don't use multiple blank lines > > #5757: FILE: test/common_plat/performance/odp_sched_latency.c:824: > > > > + > > > > total: 0 errors, 54 warnings, 44 checks, 5742 lines checked > > > > NOTE: Ignored message types: BIT_MACRO COMPARISON_TO_NULL > > DEPRECATED_VARIABLE NEW_TYPEDEFS SPLIT_STRING SSCANF_TO_KSTRTO > > > > 0004-A-scalable-software-scheduler.patch has style problems, please > > review. > > > > If any of these errors are false positives, please report > > them to the maintainer, see CHECKPATCH in MAINTAINERS. > > > > > > > <br> > > > Also, this part introduces a number of errors that result in > failure<br> > > > to compile using clang. > > > > This is likely true. Is compilation with Clang a blocker? > > > > Yes, it is. We require ODP modules to be compilable with either GCC or > > clang. > > > > > > IMPORTANT NOTICE: The contents of this email and any attachments are > > confidential and may also be privileged. If you are not the intended > > recipient, please notify the sender immediately and do not disclose the > > contents to any other person, use it for any purpose, or store or copy > the > > information in any medium. Thank you. > > >
On 29 March 2017 at 03:55, Brian Brooks <brian.brooks@arm.com> wrote: > On 03/28 19:18:37, Bill Fischofer wrote: >> <html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"> > > It is infinitely better to do patch review in plain text rather > than HTML. I thought this was a plain text mailing list? > >> <meta name="Generator" content="Microsoft Exchange Server"> >> <!-- converted from text --> >> <style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style></head> >> <body> >> <font size="2"><span style="font-size:10pt;"><div class="PlainText">On Tue, Mar 28, 2017 at 2:23 PM, Brian Brooks <brian.brooks@arm.com> wrote:<br> >> > Signed-off-by: Brian Brooks <brian.brooks@arm.com><br> >> > ---<br> >> > include/odp/api/spec/queue.h | 5 +++++<br> >> > 1 file changed, 5 insertions(+)<br> >> ><br> >> > diff --git a/include/odp/api/spec/queue.h b/include/odp/api/spec/queue.h<br> >> > index 7972feac..1cec4773 100644<br> >> > --- a/include/odp/api/spec/queue.h<br> >> > +++ b/include/odp/api/spec/queue.h<br> >> > @@ -124,6 +124,11 @@ typedef struct odp_queue_param_t {<br> >> > * the queue type. */<br> >> > odp_queue_type_t type;<br> >> ><br> >> > + /** Queue size<br> >> > + *<br> >> > + * Indicates the max ring size of the ring buffer. */<br> >> > + uint32_t ring_size;<br> >> <br> >> ODP queues have historically been of unspecified size. If we're going<br> >> to introduce the notion of explicitly limited sized queues this has<br> >> additional implications.<br> Implementations have likely applied some internal limitation, unknown to the application. linux-generic's use of linked lists can't be seen as *the* way to do it. >> >> First, ring_size is an inappropriate choice of name here since a ring<br> >> is an implementation model, not a specification. The documentation<br> >> says "Queue size", so<br> >> <br> >> uint32_t size;<br> >> <br> >> is sufficient here. > > Agree, will change 'ring_size' to a better name. My suggestion is 'min_capacity' (or 'min_size'). The application specifies the minimum capacity requires/desires. > >> We should document that size = 0 requests a queue<br> >> of default size (which may be unbounded).<br> > > Unbounded size is not practical or possible. Can we agree that 0 means > that the default aka ODP_CONFIG_QUEUE_SIZE is used? Should we allow for > greater than ODP_CONFIG_QUEUE_SIZE? E.g. 10,000 queue depth is also not > practical or possible. Perhaps we need a max which also acts as the default. IMO, a 'min_capacity' of 0 should mean use default size (per the ODP configuration file). Which is likely what implementations already do when the application cannot specify the size. > >> <br> >> Second, if we're going to allow a queue size to be specified then this<br> >> needs to be added as an output to odp_queue_capability() > > OK, will look into that. > >> so the<br> >> application knows the max_size supported (again 0 = unbounded).<br> >> <br> >> A larger question, however, is why is this being introduced at all<br> >> since this field is only used in the modified<br> >> odph_cuckoo_table_create() helper routine and this, in turn, is only<br> >> used within the cuckootable test module? This seems an extraneous and<br> >> unnecessary change and has no relationship to the rest of this patch<br> >> series.<br> > > AFAIK, the cuckoo unit test enqueues too many events (millions) to a queue. > That sounds like it makes no sense, but is an example of how anything is > possible. The cuckoo hash table uses a queue to store unused "slots". The hash table could be very large so have a very large number of unused slots that need to be saved on a queue. I think it is unreasonable of the cuckoo hash table to expect infinite queue sizes. Better to specify the required capacity and then fail in odp_queue_create(). > >> <br> >> So Parts 1 and 3 of this series don't seem to have anything to do with<br> >> the scalable scheduler. As a minor point, the order of these needs to<br> >> be reversed to preserve bisectability since Part 1 can't reference the<br> >> new field before Part 3 defines it.<br> > > Agree that there are 2 independent sets of patches, but the order is needed > in order to get a sane `make check' on ARM-based chips. Without these fixes > going in first, we have no way of knowing whether the scalable scheduler patches > caused the issue or not. I can break these into 2 separate sets of patches. > >> <br> >> > +<br> >> > /** Enqueue mode<br> >> > *<br> >> > * Default value for both queue types is ODP_QUEUE_OP_MT. Application<br> >> > --<br> >> > 2.12.1<br> >> ><br> >> </div></span></font> >> </body> >> </html>
> -----Original Message----- > From: lng-odp [mailto:lng-odp-bounces@lists.linaro.org] On Behalf Of > Honnappa Nagarahalli > Sent: Wednesday, March 29, 2017 6:28 AM > To: Bill Fischofer <bill.fischofer@linaro.org>; Brian Brooks > <Brian.Brooks@arm.com> > Cc: Ola Liljedahl <Ola.Liljedahl@arm.com>; Kevin Wang > <Kevin.Wang@arm.com>; lng-odp-forward <lng-odp@lists.linaro.org> > Subject: Re: [lng-odp] [API-NEXT 4/4] A scalable software scheduler > > I had run the checkpatch.pl, but I did not see these issues. May be it was > to do with the way I generated the patch. I will address the ones that > make sense. > Checkpatch should pass cleanly for every patch. If you want an exception for some line, then you should give rationale for that e.g. on the cover letter. So, the expectation is that you will correct all and only few, well justified exceptions could be acceptable. An example would be a call to an external library (e.g. openssl) which uses camelCase function names - checkpatch warns but you cannot do anything about it. -Petri
> -----Original Message----- > From: lng-odp [mailto:lng-odp-bounces@lists.linaro.org] On Behalf Of Maxim > Uvarov > Sent: Wednesday, March 29, 2017 10:28 AM > To: Brian Brooks <brian.brooks@arm.com> > Cc: lng-odp-forward <lng-odp@lists.linaro.org> > Subject: Re: [lng-odp] [API-NEXT 3/4] api: queue: Add ring_size > > isn't queue size connected to number elements in the pool? > > Maxim. In theory, max queue size == sum of all pool sizes. But that's not a practical specification. I'll send a patch for the queue size capa/param API. -Petri
Couple of high level comments:
1) Too many #ifdefs
* Code is hard to read and maintain when every 10th line is #ifdef or #else
* Are all #ifdef combinations tested? Or meant to be supported?
* It seems that some design trade-offs would be needed for better code maintainability
2) Too many arch dependent #ifdefs
* ARM_ARCH ifdefs and assembly in main .c / .h files should be replaced with arch specific functions defined under platform/linux-generic/arch/xxx. That's the only directory which should contain inline assembly.
3) Keep and improve modularity
* #ifdef ODP_SCHEDULE_SCALABLE should not show up common pktio/queue/etc files. Only in makefiles and interface selection.
* Main build time config options (config_internal.h) should be high level (== max number of XXX), not algorithmic details of one scheduler. Move those into local .c/.h files.
* Keep definitions local when used only by one file. Check common headers before adding new definitions into common headers. E.g. CHECK_IS_POWER2 is already defined in odp_align_internal.h.
* Improve e.g. queue interface modularity first with current code base. After that's done, hook the new scheduler to the interface.
4) Split 4/4 into couple of patches
* First prepare the current code base with couple of patches
* Proper interfaces / modifications for modularity (see previous bullet)
* This enables us to see and test how current code base is impacted
* Also git history is easier to work with when single patch does not touch many files / features
* Bring in the new scheduler code as the last patch of the series
-Petri
On 03/29 10:39:35, Maxim Uvarov wrote: > 1. Prefix has to be "PATCH API-NEXT" not "API-NEXT" Okay. > 2. Please remove ifdefs. And compile all variants of schedules into the > library. Code needs to be compile each time. There are multiple ways to interpret this. Can you please describe the change(s) you would like to see in extremely verbose detail so that I can be sure I understand you? > Maxim.
On 03/29 11:20:37, Savolainen, Petri (Nokia - FI/Espoo) wrote: > Couple of high level comments: > > 1) Too many #ifdefs > * Code is hard to read and maintain when every 10th line is #ifdef or #else > * Are all #ifdef combinations tested? Or meant to be supported? > * It seems that some design trade-offs would be needed for better code maintainability > > 2) Too many arch dependent #ifdefs > * ARM_ARCH ifdefs and assembly in main .c / .h files should be replaced with arch specific functions defined under platform/linux-generic/arch/xxx. That's the only directory which should contain inline assembly. This is the current status quo and there is opportunity to do better. The first angle is general organization and code duplication. This picture may help: ~/api-next/platform/linux-generic/arch (api-next)$ tree . ├── arm │ ├── odp │ │ └── api │ │ └── cpu_arch.h │ ├── odp_cpu_arch.c │ └── odp_sysinfo_parse.c ├── default │ ├── odp │ │ └── api │ │ └── cpu_arch.h │ ├── odp_cpu_arch.c │ └── odp_sysinfo_parse.c ├── mips64 │ ├── odp │ │ └── api │ │ └── cpu_arch.h │ ├── odp_cpu_arch.c │ └── odp_sysinfo_parse.c ├── powerpc │ ├── odp │ │ └── api │ │ └── cpu_arch.h │ ├── odp_cpu_arch.c │ └── odp_sysinfo_parse.c └── x86 ├── odp │ └── api │ └── cpu_arch.h ├── odp_cpu_arch.c └── odp_sysinfo_parse.c There is not enough actual code to warrant the current situation, and I prefer to see a single file containing code for multiple architectures, e.g.: cpu.h static inline void odp_cpu_pause(void) { #if defined(__ARCH_ARM) asm volatile("isb" ::: "memory"); #elif defined(__mips__) asm volatile("nop"); asm volatile("nop"); asm volatile("nop"); asm volatile("nop"); #elif defined(__x86_64__) || defined(__i386__) asm volatile("pause"); #else #error Please add support for your CPU in cpu.h #endif } If architecture specific code must be split into architecture specific files, an alternative is: cpu.h #if defined(__ARCH_ARM) #include "cpu_arm.h" #elif defined(__mips__) #include "cpu_mips.h" .. cpu_arm.h static inline void odp_cpu_pause(void) { asm volatile("isb" ::: "memory"); } .. Notice how this eliminates the dependency on Autotools to setup proper include paths and to do architecture detection. The compiler's system-specific macros are a superior way to conditionally compile code. Some of the code isn't even CPU or architecture specific and does not even belong here, e.g. parsing sysfs. The second angle is that not all inline assembly applies to all architectures. So, the current way of doing things will continue to add more duplication and technical debt. I don't think anyone would feel good about seeing LL/SC related code in an x86 file even if it's just a bunch of empty functions. We can do better. I am curious to know your opinion here. > 3) Keep and improve modularity > * #ifdef ODP_SCHEDULE_SCALABLE should not show up common pktio/queue/etc files. Only in makefiles and interface selection. Will see what can be done. > * Main build time config options (config_internal.h) should be high level (== max number of XXX), not algorithmic details of one scheduler. Move those into local .c/.h files. This is a pre-existing condition. There are some component-specific build time config in this file (e.g. burst size, Christophe's work). It is also easy to locate build time config because it's in one file. I don't know what the long term approach should be, but I don't think this is a blocker. > * Keep definitions local when used only by one file. Check common headers before adding new definitions into common headers. E.g. CHECK_IS_POWER2 is already defined in odp_align_internal.h. Sure > * Improve e.g. queue interface modularity first with current code base. After that's done, hook the new scheduler to the interface. Will see what can be done, but may take a few iterations. > 4) Split 4/4 into couple of patches > * First prepare the current code base with couple of patches > * Proper interfaces / modifications for modularity (see previous bullet) > * This enables us to see and test how current code base is impacted > * Also git history is easier to work with when single patch does not touch many files / features > * Bring in the new scheduler code as the last patch of the series Will see what can be done, but may take a few iterations. > > > -Petri > >
On 03/28 18:50:32, Bill Fischofer wrote: > <html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"> > <meta name="Generator" content="Microsoft Exchange Server"> > <!-- converted from text --> > <style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style></head> > <body> > <font size="2"><span style="font-size:10pt;"><div class="PlainText">This part generates numerous checkpatch warnings and errors. Please<br> > run checkpatch and correct for v2.<br> > <br> > Also, this part introduces a number of errors that result in failure<br> > to compile using clang. Please test with both gcc and clang to ensure<br> > that it compiles cleanly for both (gcc looks fine)<br> > <br> > Specific clang issues:<br> Does upstream CI build with Clang on ARMv8? I don't think so because the build is broken when I try it locally. I have the patch and will send shortly.
On Thu, Mar 30, 2017 at 8:56 AM, Brian Brooks <brian.brooks@arm.com> wrote: > On 03/28 18:50:32, Bill Fischofer wrote: >> <html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"> >> <meta name="Generator" content="Microsoft Exchange Server"> >> <!-- converted from text --> >> <style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style></head> >> <body> >> <font size="2"><span style="font-size:10pt;"><div class="PlainText">This part generates numerous checkpatch warnings and errors. Please<br> >> run checkpatch and correct for v2.<br> >> <br> >> Also, this part introduces a number of errors that result in failure<br> >> to compile using clang. Please test with both gcc and clang to ensure<br> >> that it compiles cleanly for both (gcc looks fine)<br> >> <br> >> Specific clang issues:<br> > > Does upstream CI build with Clang on ARMv8? I don't think so because > the build is broken when I try it locally. I have the patch and will > send shortly. CI is supposed to be regression testing on ARM using clang as well as gcc. Maxim may have insight here.
I think for now we do not have build for arm v8 & clang. At least it did not capture build error. Maxim. On 30 March 2017 at 17:45, Bill Fischofer <bill.fischofer@linaro.org> wrote: > On Thu, Mar 30, 2017 at 8:56 AM, Brian Brooks <brian.brooks@arm.com> > wrote: > > On 03/28 18:50:32, Bill Fischofer wrote: > >> <html><head><meta http-equiv="Content-Type" content="text/html; > charset=utf-8"> > >> <meta name="Generator" content="Microsoft Exchange Server"> > >> <!-- converted from text --> > >> <style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; > border-left: #800000 2px solid; } --></style></head> > >> <body> > >> <font size="2"><span style="font-size:10pt;"><div > class="PlainText">This part generates numerous checkpatch warnings and > errors. Please<br> > >> run checkpatch and correct for v2.<br> > >> <br> > >> Also, this part introduces a number of errors that result in failure<br> > >> to compile using clang. Please test with both gcc and clang to > ensure<br> > >> that it compiles cleanly for both (gcc looks fine)<br> > >> <br> > >> Specific clang issues:<br> > > > > Does upstream CI build with Clang on ARMv8? I don't think so because > > the build is broken when I try it locally. I have the patch and will > > send shortly. > > CI is supposed to be regression testing on ARM using clang as well as > gcc. Maxim may have insight here. >
No build, no run. We found several problems (memory ordering related so visible on ARM but not on x86) in the upstream code when running on ARM systems. Aren’t there any ARM systems to run on in the LNG lab? -- Ola Ola Liljedahl, Networking System Architect, ARM Phone: +46 706 866 373 Skype: ola.liljedahl From: Maxim Uvarov <maxim.uvarov@linaro.org<mailto:maxim.uvarov@linaro.org>> Date: Thursday, 30 March 2017 at 16:56 To: Bill Fischofer <bill.fischofer@linaro.org<mailto:bill.fischofer@linaro.org>> Cc: Brian Brooks <Brian.Brooks@arm.com<mailto:Brian.Brooks@arm.com>>, lng-odp-forward <lng-odp@lists.linaro.org<mailto:lng-odp@lists.linaro.org>>, Ola Liljedahl <ola.liljedahl@arm.com<mailto:ola.liljedahl@arm.com>>, Kevin Wang <Kevin.Wang@arm.com<mailto:Kevin.Wang@arm.com>>, Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com<mailto:Honnappa.Nagarahalli@arm.com>> Subject: Re: [lng-odp] [API-NEXT 4/4] A scalable software scheduler I think for now we do not have build for arm v8 & clang. At least it did not capture build error. Maxim. On 30 March 2017 at 17:45, Bill Fischofer <bill.fischofer@linaro.org<mailto:bill.fischofer@linaro.org>> wrote: On Thu, Mar 30, 2017 at 8:56 AM, Brian Brooks <brian.brooks@arm.com<mailto:brian.brooks@arm.com>> wrote: > On 03/28 18:50:32, Bill Fischofer wrote: >> <html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"> >> <meta name="Generator" content="Microsoft Exchange Server"> >> <!-- converted from text --> >> <style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style></head> >> <body> >> <font size="2"><span style="font-size:10pt;"><div class="PlainText">This part generates numerous checkpatch warnings and errors. Please<br> >> run checkpatch and correct for v2.<br> >> <br> >> Also, this part introduces a number of errors that result in failure<br> >> to compile using clang. Please test with both gcc and clang to ensure<br> >> that it compiles cleanly for both (gcc looks fine)<br> >> <br> >> Specific clang issues:<br> > > Does upstream CI build with Clang on ARMv8? I don't think so because > the build is broken when I try it locally. I have the patch and will > send shortly. CI is supposed to be regression testing on ARM using clang as well as gcc. Maxim may have insight here. IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
diff --git a/helper/cuckootable.c b/helper/cuckootable.c index 80ff4989..d3d1563c 100644 --- a/helper/cuckootable.c +++ b/helper/cuckootable.c @@ -256,6 +256,7 @@ odph_cuckoo_table_create( /* initialize free_slots queue */ odp_queue_param_init(&qparam); qparam.type = ODP_QUEUE_TYPE_PLAIN; + qparam.ring_size = capacity; snprintf(queue_name, sizeof(queue_name), "fs_%s", name); queue = odp_queue_create(queue_name, &qparam);