[v5.15,3/4] mm/memcg: Add a local_lock_t for IRQ and TASK object.

From: Sebastian Andrzej Siewior <bigeasy@linutronix.de>

From: Sebastian Andrzej Siewior <bigeasy@linutronix.de>

The members of the per-CPU structure memcg_stock_pcp are protected
either by disabling interrupts or by disabling preemption if the
invocation occurred in process context.
Disabling interrupts protects most of the structure excluding task_obj
while disabling preemption protects only task_obj.
This schema is incompatible with PREEMPT_RT because it creates atomic
context in which actions are performed which require preemptible
context. One example is obj_cgroup_release().

The IRQ-disable and preempt-disable sections can be replaced with
local_lock_t which preserves the explicit disabling of interrupts while
keeps the code preemptible on PREEMPT_RT.

The task_obj has been added for performance reason on non-preemptible
kernels where preempt_disable() is a NOP. On the PREEMPT_RT preemption
model preempt_disable() is always implemented. Also there are no memory
allocations in_irq() context and softirqs are processed in (preemptible)
process context. Therefore it makes sense to avoid using task_obj.

Don't use task_obj on PREEMPT_RT and replace manual disabling of
interrupts with a local_lock_t. This change requires some factoring:

- drain_obj_stock() drops a reference on obj_cgroup which leads to an
invocation of obj_cgroup_release() if it is the last object. This in
turn leads to recursive locking of the local_lock_t. To avoid this,
obj_cgroup_release() is invoked outside of the locked section.

- drain_obj_stock() gets a memcg_stock_pcp passed if the stock_lock has been
acquired (instead of the task_obj_lock) to avoid recursive locking later
in refill_stock().

- drain_all_stock() disables preemption via get_cpu() and then invokes
drain_local_stock() if it is the local CPU to avoid scheduling a worker
(which invokes the same function). Disabling preemption here is
problematic due to the sleeping locks in drain_local_stock().
This can be avoided by always scheduling a worker, even for the local
CPU. Using cpus_read_lock() stabilizes cpu_online_mask which ensures
that no worker is scheduled for an offline CPU. Since there is no
flush_work(), it is still possible that a worker is invoked on the wrong
CPU but it is okay since it operates always on the local-CPU data.

- drain_local_stock() is always invoked as a worker so it can be optimized
by removing in_task() (it is always true) and avoiding the "irq_save"
variant because interrupts are always enabled here. Operating on
task_obj first allows to acquire the lock_lock_t without lockdep
complains.

Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
[rw: backported to v5.15]
Signed-off-by: Richard Weinberger <richard@nod.at>
---
 mm/memcontrol.c | 173 +++++++++++++++++++++++++++++++-----------------
 1 file changed, 113 insertions(+), 60 deletions(-)

Message ID	20220622110506.10534-4-richard@nod.at
State	New
Headers	show Return-Path: <linux-rt-users-owner@kernel.org> From: Richard Weinberger <richard@nod.at> To: linux-rt-users@vger.kernel.org Cc: williams@redhat.com, bigeasy@linutronix.de, Richard Weinberger <richard@nod.at> Subject: [v5.15 3/4] mm/memcg: Add a local_lock_t for IRQ and TASK object. Date: Wed, 22 Jun 2022 13:05:05 +0200 Message-Id: <20220622110506.10534-4-richard@nod.at> In-Reply-To: <20220622110506.10534-1-richard@nod.at> References: <20220622110506.10534-1-richard@nod.at> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk
Series	Backport MEMCG changes from v5.16 \| expand [v5.15,0/4] Backport MEMCG changes from v5.16 [v5.15,1/4] mm/memcg: Disable threshold event handlers on PREEMPT_RT [v5.15,2/4] mm/memcg: Protect per-CPU counter by disabling preemption on PREEMPT_RT where needed. [v5.15,3/4] mm/memcg: Add a local_lock_t for IRQ and TASK object. [v5.15,4/4] Allow MEMCG on PREEMPT_RT

[v5.15,3/4] mm/memcg: Add a local_lock_t for IRQ and TASK object.

Commit Message

Patch