From patchwork Tue Jun 20 21:28:55 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Lezcano X-Patchwork-Id: 106029 Delivered-To: patch@linaro.org Received: by 10.140.91.2 with SMTP id y2csp1607792qgd; Tue, 20 Jun 2017 14:30:23 -0700 (PDT) X-Received: by 10.98.63.138 with SMTP id z10mr3856724pfj.203.1497994223793; Tue, 20 Jun 2017 14:30:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1497994223; cv=none; d=google.com; s=arc-20160816; b=glih3uQlKsj+04CUzHH+rdc8oCAB/9FtasbC6I/7S8HzCktAqHNK2g/jrhvrqL5cPj VIB+1Lc9MdgF72ydXCwQ2p2feY6dcqUuqCKTWcCXMi/47E7ZtaG+x8ZH9ziXqdf1OX/n y804WYd0rY36Mf6IphW81//9JhbvQ3eQOE6JuJiGM4EG1owuCl42UlCHZZf86IdKdsIo vQXE8voQXgiIf4IXlN/VMGbd6fbscD1wLr9gpM1xrz7jHG5x65aJB83Y2qVDCNhaUPem COk3oYuvILcH8a3jWp40O0l27vtzzd9P4EvgDCl5AqYjk9gE5R1JKVDpRw/XLfu4x+nJ qg7w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=iIMOoa2LzPYwv1IOW4YPVJ8A7ooVUYcH8Q+Cnr8W3NM=; b=oYWR07pRGJzhy9KV3Ie14EkMlBa4qJHp7GHa7j25a7dGPK6map/0pdoU4zio0wjOYa HLzkt1HWnx8Mvjf/7Ky8EXOlxM1LrmSZBpscnWFI9ANCRV34c6Anr3tIRxgFJXm8bAiT OE5aJgAlTc2v8pEbdoRzzxqrOwabY3T8r4+GvFkuHP5LAuUR3kz2u7ffxiwPIpcuExff jmEn5/6D3inQHyHFFCrwYqt7UgiI9QyX2l5ORCbKZ2bCvE9fEgp2PsHZJoYRyI7nrpZE yP2qQnZHPiEc0kKzvVyGlxb+ilYR0g/KyiNFZhOHR3LBIEclrdhs2f/hiSzPxKCpCPcD nwpQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.b=WfgU66nK; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 31si12395690plk.261.2017.06.20.14.30.23; Tue, 20 Jun 2017 14:30:23 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.b=WfgU66nK; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752619AbdFTVaV (ORCPT + 25 others); Tue, 20 Jun 2017 17:30:21 -0400 Received: from mail-wr0-f173.google.com ([209.85.128.173]:35216 "EHLO mail-wr0-f173.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751090AbdFTVaT (ORCPT ); Tue, 20 Jun 2017 17:30:19 -0400 Received: by mail-wr0-f173.google.com with SMTP id y25so69187102wrd.2 for ; Tue, 20 Jun 2017 14:30:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=iIMOoa2LzPYwv1IOW4YPVJ8A7ooVUYcH8Q+Cnr8W3NM=; b=WfgU66nKfSSGwEl8K7jEUKwo9M2SE/O+oxiwEaThHddmO6t76D/dgKrBlBEjdmRu1m 4GKJ94XbEB+clMGba1zWuygguXTKjnnYLEL09x6onedMFe/H4i82WbSh8L/MaP7STHvf LBDjtcs+1tvBvN4vuHk8tmqMtabkjQ2mpu6YQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=iIMOoa2LzPYwv1IOW4YPVJ8A7ooVUYcH8Q+Cnr8W3NM=; b=NwrDaP2jDOWaYrxpSNFXY13zbeFZoW32XOCgvo+wNy8aiR29n+bdpiZdpuJ5toFvNU P2WYhArFBQu8Hq+/ts2ItEVkoj2DheyJERkl7OxnIrxUvwdgloBZslDYyqBxTBIROs3d taITASZKRQB9U/ijsoWSZA/Hv2J7r8k3iD9LS6uE0zo6hrtUKeEX0Xxl2/j86gkp+SIz mZWAclOfLqaAsiYenUVSTYmPMJaYhKYclGijBX4yH4yBk6Ck3cHSw9NkiGlB+LQKFZrx lsfg4NInMiD+HV+SrACecxtVOwyon67ukHjfoql6jNWdoBpOV31z1FtzCAoqVud+5ZwS gouA== X-Gm-Message-State: AKS2vOz3VNCs1ejlq6R5T/rcZawyJLtlcYLxIz41cmsf7GKyFgmcV4mw cgqaIJ3GwkkLMDhfXhwF9g== X-Received: by 10.28.210.194 with SMTP id j185mr4246031wmg.36.1497994218296; Tue, 20 Jun 2017 14:30:18 -0700 (PDT) Received: from localhost.localdomain ([2a01:e35:879a:6cd0:ccbe:9633:e651:d298]) by smtp.gmail.com with ESMTPSA id o7sm13044234wmd.0.2017.06.20.14.30.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 20 Jun 2017 14:30:17 -0700 (PDT) From: Daniel Lezcano To: tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, Mark Rutland , Jens Axboe , Hannes Reinecke , Bjorn Helgaas Subject: [PATCH 1/3] irq: Allow to pass the IRQF_TIMER flag with percpu irq request Date: Tue, 20 Jun 2017 23:28:55 +0200 Message-Id: <1497994139-19816-1-git-send-email-daniel.lezcano@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <20170620212657.GB1812@mai> References: <20170620212657.GB1812@mai> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In the next changes, we track when the interrupts occur in order to statistically compute when is supposed to happen the next interrupt. In all the interruptions, it does not make sense to store the timer interrupt occurences and try to predict the next interrupt as we know the expiration time. The request_irq() has a irq flags parameter and the timer drivers use it to pass the IRQF_TIMER flag, letting us know the interrupt is coming from a timer. Based on this flag, we can discard these interrupts when tracking them. But, the API request_percpu_irq does not allow to pass a flag, hence specifying if the interrupt type is a timer. Add a function __request_percpu_irq() where we can specify the flags. The request_percpu_irq() function is changed to be a wrapper to __request_percpu_irq() passing a zero flag parameter. For now, in order to prevent a misusage of this parameter, only the IRQF_TIMER flag (or zero) is a valid parameter to be passed to the __request_percpu_irq() function. Signed-off-by: Daniel Lezcano Cc: Mark Rutland --- Changelog: V11: - Drop the changes in the timer drivers and rename the function request_percpu_irq_flags to __request_percpu_irq. V10: - Drop the change for arch arm virtual timer https://lkml.org/lkml/2017/4/25/184 V9: - Clarified the patch description - Fixed EXPORT_SYMBOL_GPL(request_percpu_irq_flags) Signed-off-by: Daniel Lezcano --- include/linux/interrupt.h | 11 ++++++++++- kernel/irq/manage.c | 15 ++++++++++----- 2 files changed, 20 insertions(+), 6 deletions(-) -- 2.7.4 Acked-by: Mark Rutland diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h index a6fba48..d4ae50e 100644 --- a/include/linux/interrupt.h +++ b/include/linux/interrupt.h @@ -152,8 +152,17 @@ request_any_context_irq(unsigned int irq, irq_handler_t handler, unsigned long flags, const char *name, void *dev_id); extern int __must_check +__request_percpu_irq(unsigned int irq, irq_handler_t handler, + unsigned long flags, const char *devname, + void __percpu *percpu_dev_id); + +static inline int __must_check request_percpu_irq(unsigned int irq, irq_handler_t handler, - const char *devname, void __percpu *percpu_dev_id); + const char *devname, void __percpu *percpu_dev_id) +{ + return __request_percpu_irq(irq, handler, 0, + devname, percpu_dev_id); +} extern const void *free_irq(unsigned int, void *); extern void free_percpu_irq(unsigned int, void __percpu *); diff --git a/kernel/irq/manage.c b/kernel/irq/manage.c index 4c34696..5424cfb 100644 --- a/kernel/irq/manage.c +++ b/kernel/irq/manage.c @@ -1974,9 +1974,10 @@ int setup_percpu_irq(unsigned int irq, struct irqaction *act) } /** - * request_percpu_irq - allocate a percpu interrupt line + * __request_percpu_irq - allocate a percpu interrupt line * @irq: Interrupt line to allocate * @handler: Function to be called when the IRQ occurs. + * @flags: Interrupt type flags (IRQF_TIMER only) * @devname: An ascii name for the claiming device * @dev_id: A percpu cookie passed back to the handler function * @@ -1989,8 +1990,9 @@ int setup_percpu_irq(unsigned int irq, struct irqaction *act) * the handler gets called with the interrupted CPU's instance of * that variable. */ -int request_percpu_irq(unsigned int irq, irq_handler_t handler, - const char *devname, void __percpu *dev_id) +int __request_percpu_irq(unsigned int irq, irq_handler_t handler, + unsigned long flags, const char *devname, + void __percpu *dev_id) { struct irqaction *action; struct irq_desc *desc; @@ -2004,12 +2006,15 @@ int request_percpu_irq(unsigned int irq, irq_handler_t handler, !irq_settings_is_per_cpu_devid(desc)) return -EINVAL; + if (flags && flags != IRQF_TIMER) + return -EINVAL; + action = kzalloc(sizeof(struct irqaction), GFP_KERNEL); if (!action) return -ENOMEM; action->handler = handler; - action->flags = IRQF_PERCPU | IRQF_NO_SUSPEND; + action->flags = flags | IRQF_PERCPU | IRQF_NO_SUSPEND; action->name = devname; action->percpu_dev_id = dev_id; @@ -2030,7 +2035,7 @@ int request_percpu_irq(unsigned int irq, irq_handler_t handler, return retval; } -EXPORT_SYMBOL_GPL(request_percpu_irq); +EXPORT_SYMBOL_GPL(__request_percpu_irq); /** * irq_get_irqchip_state - returns the irqchip state of a interrupt. From patchwork Tue Jun 20 21:28:56 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Lezcano X-Patchwork-Id: 106030 Delivered-To: patch@linaro.org Received: by 10.140.91.2 with SMTP id y2csp1607871qgd; Tue, 20 Jun 2017 14:30:37 -0700 (PDT) X-Received: by 10.84.131.6 with SMTP id 6mr6384391pld.33.1497994237525; Tue, 20 Jun 2017 14:30:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1497994237; cv=none; d=google.com; s=arc-20160816; b=A57nCztLcPkK8xGFDMgu0IyARGSGGpDpth+Kd0JxJOKGPKaVnYnhljzUxJnGIPf7LP ZQpJ9fp45vCIp3UH/isUTIu0n9DzI3ytRI7sDt6wyeymKvL+1pp8PZfPe4rp2r5tHfOf l09+SVp9vE+YGuPpmvUEOMqD6NHW8IeAwpRfKB+ORIdGQY+Z49KBOhWK9mDzwzLaYVpW rfrCIn9dRcfvEPcL853k44NHQ493doT+J/bgMBx+v0DSvJVQRub6TQpQt/FNyz14VDAG RRpaLHlr5UgWAbaMa2mk/+QbWrs4FeE9HtWv7hTjuaHrCGsQcaTSa3/OMwMmftrpei85 +TQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=eKGBgjggCrlURhqBYo6yiaYRZ1zt68EJ4TEm757MmjE=; b=0A1hUGR8kCKtB7cBLWGCqdTfr3R4b3XGk/9J9TmWWTqaQBF6VH7eBGMulMB++6IgKw th5Javb302WBdKi/sl2j2ztnkBS4rViDtPL2xgsBmAAkypuAnCLKhqQZLaVCnGqrwy8X UWpMo0/wIKR2mbZbkZh75WGTj9UX5xJAOPF4dRXBVeMt1DiJ5wNCv0Pl92UC/En8tj8w h4vWtdCDjsQeea9BXkwwiLljjBUc4T071sbrzoatVkYYX9z16ImYuSyDBrFUOhW0O9OI LSXNtWPrsQQjaKh46JJ9CTFmIL5Z1/R/CtbszojS0jLMrubt741Wh4Z3+p4dpr55ryiK +HrQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.b=bmsGI2Ap; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 31si12395690plk.261.2017.06.20.14.30.37; Tue, 20 Jun 2017 14:30:37 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.b=bmsGI2Ap; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752697AbdFTVae (ORCPT + 25 others); Tue, 20 Jun 2017 17:30:34 -0400 Received: from mail-wr0-f181.google.com ([209.85.128.181]:34701 "EHLO mail-wr0-f181.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752479AbdFTVac (ORCPT ); Tue, 20 Jun 2017 17:30:32 -0400 Received: by mail-wr0-f181.google.com with SMTP id 77so108477074wrb.1 for ; Tue, 20 Jun 2017 14:30:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=eKGBgjggCrlURhqBYo6yiaYRZ1zt68EJ4TEm757MmjE=; b=bmsGI2ApxkPIjnRML+FmHk29cdASQT6XMBl+Ky0LHTMWniR4F0lv3jGzKbKvdETf2Y G5U9A23UKWnd7bh+f/t3dwb+FpXj0WU/Vkpr1lQ1+7gpj8UUhsmStYKhWY0LbxvMPhfw fiPeeV90YbhxHt63kAFkte7cfrRsyeq8/1mjI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=eKGBgjggCrlURhqBYo6yiaYRZ1zt68EJ4TEm757MmjE=; b=T5zNu71wZXLwyZhW7dbG2ZAs8gvcAWTD3rsa1pGr20aOTnUZliIWETlfpnW2p/ohsR IPohQbp7BEgBl0zHO8146DUhvUOyLTMcuOsJLT4GIr2zuVT2hqsZzAlNAORlPfsKnIqb FaHazy+NU/2PVBXum5xy9jPZf1HUImrEw21P25J9twuzUUcJznHqrobPiRqVsr4vpLaQ mf6R1ivmaonmE+Zcsdp1MtNj6ben6qtznxBHVZxaal0htTu7xvGWViXjQ4QJ8vfclLlG IVIaF/aGbZL5nlmN4oD27lxPFxJ4z5a178IukmxFlMaB2Cy87bMOkX+b5ieAIkkdhVA5 AumA== X-Gm-Message-State: AKS2vOxPFgL+yrGXFB147ZHR3bioTxGhsE6iyCO38mfGOLphc2DA2+39 QNi6Dia+l4ehtt/6 X-Received: by 10.223.135.237 with SMTP id c42mr24103818wrc.10.1497994231145; Tue, 20 Jun 2017 14:30:31 -0700 (PDT) Received: from localhost.localdomain ([2a01:e35:879a:6cd0:ccbe:9633:e651:d298]) by smtp.gmail.com with ESMTPSA id o7sm13044234wmd.0.2017.06.20.14.30.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 20 Jun 2017 14:30:30 -0700 (PDT) From: Daniel Lezcano To: tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, Peter Zijlstra , "Rafael J . Wysocki" , Vincent Guittot , Nicolas Pitre , Hannes Reinecke , Jens Axboe , Bjorn Helgaas Subject: [PATCH 2/3] irq: Track the interrupt timings Date: Tue, 20 Jun 2017 23:28:56 +0200 Message-Id: <1497994139-19816-2-git-send-email-daniel.lezcano@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1497994139-19816-1-git-send-email-daniel.lezcano@linaro.org> References: <20170620212657.GB1812@mai> <1497994139-19816-1-git-send-email-daniel.lezcano@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The interrupt framework gives a lot of information about each interrupt. It does not keep track of when those interrupts occur though. This patch provides a mean to record the timestamp for each interrupt occurrences in a per-CPU circular buffer to help with the prediction of the next occurrence using a statistical model. Each CPU can store IRQ_TIMINGS_SIZE events , the current value of IRQ_TIMINGS_SIZE is 32. Each event is encoded into a single u64, where the 48 bits are used for the timestamp and the next 16 bits are for the irq number. A static key is introduced so when the irq prediction is switched off at runtime, we can reduce the overhead near to zero. It results in most of the code in internals.h for inline reason and a very few in the new file timings.c. The latter will contain more in the next patch which will provide the statistical model for the next event prediction. Note this code is by default *not* compiled in the kernel. Signed-off-by: Daniel Lezcano Acked-by: Nicolas Pitre Cc: Peter Zijlstra Cc: Rafael J. Wysocki Cc: Vincent Guittot --- Changelog: V11: - No changes V10: - No changes V9: - Changed indentation level by inverting the static key condition - Encoded interrupt and timestamp into a u64 variable - Boolean enable instead of refcount for the static key V8: - Replaced percpu field in the irqdesc by a percpu array containing the timings and the associated irq. The function irq_timings_get_next()is no longer needed, so it is removed - Removed all unused code resulting from the conversion irqdesc->percpu timings storage V7: - Mentionned in the irq_timings_get_next() function description, the function must be called inside a rcu read locked section V6: - Renamed handle_irq_timings to record_irq_time - Stored the event time instead of the interval time - Removed the 'timestamp' field from the timings structure - Moved _handle_irq_timings content inside record_irq_time V5: - Changed comment about 'deterministic' as the comment is confusing - Added license comment in the header - Replaced irq_timings_get/put by irq_timings_enable/disable - Moved IRQS_TIMINGS check in the handle_timings inline function - Dropped 'if !prev' as it is pointless - Stored time interval in nsec basis with u64 instead of u32 - Removed redundant store - Removed the math V4: - Added a static key - Added more comments for irq_timings_get_next() - Unified some function names to be prefixed by 'irq_timings_...' - Fixed a rebase error V3: - Replaced ktime_get() by local_clock() - Shared irq are not handled - Simplified code by adding the timing in the irqdesc struct - Added a function to browse the irq timings V2: - Fixed kerneldoc comment - Removed data field from the struct irq timing - Changed the lock section comment - Removed semi-colon style with empty stub - Replaced macro by static inline - Fixed static functions declaration RFC: - initial posting Signed-off-by: Daniel Lezcano --- include/linux/interrupt.h | 5 +++ kernel/irq/Kconfig | 3 ++ kernel/irq/Makefile | 1 + kernel/irq/handle.c | 2 ++ kernel/irq/internals.h | 84 +++++++++++++++++++++++++++++++++++++++++++++++ kernel/irq/manage.c | 3 ++ kernel/irq/timings.c | 30 +++++++++++++++++ 7 files changed, 128 insertions(+) create mode 100644 kernel/irq/timings.c -- 2.7.4 diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h index d4ae50e..5f98aa7 100644 --- a/include/linux/interrupt.h +++ b/include/linux/interrupt.h @@ -712,6 +712,11 @@ static inline void init_irq_proc(void) } #endif +#ifdef CONFIG_IRQ_TIMINGS +void irq_timings_enable(void); +void irq_timings_disable(void); +#endif + struct seq_file; int show_interrupts(struct seq_file *p, void *v); int arch_show_interrupts(struct seq_file *p, int prec); diff --git a/kernel/irq/Kconfig b/kernel/irq/Kconfig index 3bbfd6a..38e551d 100644 --- a/kernel/irq/Kconfig +++ b/kernel/irq/Kconfig @@ -81,6 +81,9 @@ config GENERIC_MSI_IRQ_DOMAIN config HANDLE_DOMAIN_IRQ bool +config IRQ_TIMINGS + bool + config IRQ_DOMAIN_DEBUG bool "Expose hardware/virtual IRQ mapping via debugfs" depends on IRQ_DOMAIN && DEBUG_FS diff --git a/kernel/irq/Makefile b/kernel/irq/Makefile index 1d3ee31..efb5f14 100644 --- a/kernel/irq/Makefile +++ b/kernel/irq/Makefile @@ -10,3 +10,4 @@ obj-$(CONFIG_PM_SLEEP) += pm.o obj-$(CONFIG_GENERIC_MSI_IRQ) += msi.o obj-$(CONFIG_GENERIC_IRQ_IPI) += ipi.o obj-$(CONFIG_SMP) += affinity.o +obj-$(CONFIG_IRQ_TIMINGS) += timings.o diff --git a/kernel/irq/handle.c b/kernel/irq/handle.c index d3f2490..eb4d3e8 100644 --- a/kernel/irq/handle.c +++ b/kernel/irq/handle.c @@ -138,6 +138,8 @@ irqreturn_t __handle_irq_event_percpu(struct irq_desc *desc, unsigned int *flags unsigned int irq = desc->irq_data.irq; struct irqaction *action; + record_irq_time(desc); + for_each_action_of_desc(desc, action) { irqreturn_t res; diff --git a/kernel/irq/internals.h b/kernel/irq/internals.h index bc226e7..df51b5e0 100644 --- a/kernel/irq/internals.h +++ b/kernel/irq/internals.h @@ -8,6 +8,7 @@ #include #include #include +#include #ifdef CONFIG_SPARSE_IRQ # define IRQ_BITMAP_BITS (NR_IRQS + 8196) @@ -57,6 +58,7 @@ enum { IRQS_WAITING = 0x00000080, IRQS_PENDING = 0x00000200, IRQS_SUSPENDED = 0x00000800, + IRQS_TIMINGS = 0x00001000, }; #include "debug.h" @@ -226,3 +228,85 @@ irq_pm_install_action(struct irq_desc *desc, struct irqaction *action) { } static inline void irq_pm_remove_action(struct irq_desc *desc, struct irqaction *action) { } #endif + +#ifdef CONFIG_IRQ_TIMINGS + +#define IRQ_TIMINGS_SHIFT 5 +#define IRQ_TIMINGS_SIZE (1 << IRQ_TIMINGS_SHIFT) +#define IRQ_TIMINGS_MASK (IRQ_TIMINGS_SIZE - 1) + +struct irq_timings { + u64 values[IRQ_TIMINGS_SIZE]; /* our circular buffer */ + unsigned int count; /* Number of interruptions since last inspection */ +}; + +DECLARE_PER_CPU(struct irq_timings, irq_timings); + +static inline void remove_timings(struct irq_desc *desc) +{ + desc->istate &= ~IRQS_TIMINGS; +} + +static inline void setup_timings(struct irq_desc *desc, struct irqaction *act) +{ + /* + * We don't need the measurement because the idle code already + * knows the next expiry event. + */ + if (act->flags & __IRQF_TIMER) + return; + + desc->istate |= IRQS_TIMINGS; +} + +extern void irq_timings_enable(void); +extern void irq_timings_disable(void); + +extern struct static_key_false irq_timing_enabled; + +/* + * The interrupt number and the timestamp are encoded into a single + * u64 variable to optimize the size. + * 48 bit time stamp and 16 bit IRQ number is way sufficient. + * Who cares an IRQ after 78 hours of idle time? + */ +static inline u64 irq_timing_encode(u64 timestamp, int irq) +{ + return (timestamp << 16) | irq; +} + +static inline void irq_timing_decode(u64 value, u64 *timestamp, int *irq) +{ + *timestamp = value >> 16; + *irq = value & U16_MAX; +} + +/* + * The function record_irq_time is only called in one place in the + * interrupts handler. We want this function always inline so the code + * inside is embedded in the function and the static key branching + * code can act at the higher level. Without the explicit + * __always_inline we can end up with a function call and a small + * overhead in the hotpath for nothing. + */ +static __always_inline void record_irq_time(struct irq_desc *desc) +{ + if (!static_branch_likely(&irq_timing_enabled)) + return; + + if (desc->istate & IRQS_TIMINGS) { + struct irq_timings *timings = this_cpu_ptr(&irq_timings); + + timings->values[timings->count & IRQ_TIMINGS_MASK] = + irq_timing_encode(local_clock(), + irq_desc_get_irq(desc)); + + timings->count++; + } +} +#else +static inline void remove_timings(struct irq_desc *desc) {} +static inline void setup_timings(struct irq_desc *desc, + struct irqaction *act) {}; +static inline void record_irq_time(struct irq_desc *desc) {} +#endif /* CONFIG_IRQ_TIMINGS */ diff --git a/kernel/irq/manage.c b/kernel/irq/manage.c index 5424cfb..d3f63f1 100644 --- a/kernel/irq/manage.c +++ b/kernel/irq/manage.c @@ -1388,6 +1388,8 @@ __setup_irq(unsigned int irq, struct irq_desc *desc, struct irqaction *new) raw_spin_unlock_irqrestore(&desc->lock, flags); + setup_timings(desc, new); + /* * Strictly no need to wake it up, but hung_task complains * when no hard interrupt wakes the thread up. @@ -1516,6 +1518,7 @@ static struct irqaction *__free_irq(unsigned int irq, void *dev_id) irq_settings_clr_disable_unlazy(desc); irq_shutdown(desc); irq_release_resources(desc); + remove_timings(desc); } #ifdef CONFIG_SMP diff --git a/kernel/irq/timings.c b/kernel/irq/timings.c new file mode 100644 index 0000000..56cf687 --- /dev/null +++ b/kernel/irq/timings.c @@ -0,0 +1,30 @@ +/* + * linux/kernel/irq/timings.c + * + * Copyright (C) 2016, Linaro Ltd - Daniel Lezcano + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + */ +#include +#include +#include +#include + +#include "internals.h" + +DEFINE_STATIC_KEY_FALSE(irq_timing_enabled); + +DEFINE_PER_CPU(struct irq_timings, irq_timings); + +void irq_timings_enable(void) +{ + static_branch_enable(&irq_timing_enabled); +} + +void irq_timings_disable(void) +{ + static_branch_disable(&irq_timing_enabled); +} From patchwork Tue Jun 20 21:28:57 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Lezcano X-Patchwork-Id: 106031 Delivered-To: patch@linaro.org Received: by 10.140.91.2 with SMTP id y2csp1608014qgd; Tue, 20 Jun 2017 14:31:01 -0700 (PDT) X-Received: by 10.84.137.3 with SMTP id 3mr36825170plm.68.1497994261010; Tue, 20 Jun 2017 14:31:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1497994261; cv=none; d=google.com; s=arc-20160816; b=TegQQszj3ppbZIh/FP5HFAEohiZcx7XSw4x/mt3Th7BqECiP6DGBQi1ANyTKBOg9i3 kVTTgSVgqWzYxGufZepQ2A/1gAr35Sk3IRonqV0qwNGsW+WAZt7IgHJdURCfEUK5PtZP ib5UV70Ekgai7PO9d3dyqCUHLAO5oXxnbYSlHFBpmSL1MuQWvSzKlu0OQpwqnF1yHj5+ QDI/JIfoAzqrtJv0vzyXglBQRgEVpV9O7orWAxILNvymGm6HSS+sx8OKjXl5QuzjvOTy V1c10Thm43QuQgoS34xJ8b2owYU939dnLvaWLJOSA9m9wonbaf0iumb/V/7Z+4DFN/lu V+Tw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=VdqovIFYXRj7LcrFokdkDuQZritdGHgq9o+qMxYPYHc=; b=euwBVu2aZrGX9rQMiVRwy5SxFO8sG0a1XkVkHMTZ4jGlGwj+1T5n8GoT5zY75tITFX ygZOcKIvxd0jweWbzXcZ6i0iz3yNVonJRB0zwErCDnEQMQ8hGa69574XYbn3prnF+L9U tPjfGJtxRlLS7vmOYQNnevjXnt6VBZoervWa2c6oZriovnN3zgHi/9XS4+0615+jBwcF d8ZZ49+X2yut2/GJpXZPtv1NIi+hLyxWL7DKpLP1fqFAy0258+PFo1xhdGwO52T9a+vw UMitIzbCD6x9nT1J/Iw30UQy8Vh8iIIKfYR8Q4ipl2F42xGfet9ROn6vRKH+kTDW9boD MaTw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.b=ERYJJf5h; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 5si13142985plc.632.2017.06.20.14.31.00; Tue, 20 Jun 2017 14:31:00 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.b=ERYJJf5h; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752747AbdFTVas (ORCPT + 25 others); Tue, 20 Jun 2017 17:30:48 -0400 Received: from mail-wr0-f170.google.com ([209.85.128.170]:36631 "EHLO mail-wr0-f170.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752479AbdFTVaq (ORCPT ); Tue, 20 Jun 2017 17:30:46 -0400 Received: by mail-wr0-f170.google.com with SMTP id c11so53008331wrc.3 for ; Tue, 20 Jun 2017 14:30:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=VdqovIFYXRj7LcrFokdkDuQZritdGHgq9o+qMxYPYHc=; b=ERYJJf5hrqCPwsxt6u3PRQruMlw3ghvDmUbRHoeidb4PDsBUHN3UbBbiRgFaQoUcN4 Thbsykgg9r67HeCG92fHRJ3zUaKiOSNO28p79uz8fSNRPmYnYvTqzxILO42pS67XTR5o p7lDP1RaSGAJfvwkkgGwB0WTQNbx8Z2A75Fb4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=VdqovIFYXRj7LcrFokdkDuQZritdGHgq9o+qMxYPYHc=; b=neQ82Xz7E18ez7Zg6kOdjfgLZEXrqRuJx4LeIMzF23J/UbF8dIk8JX6sqkJQtiBG2J 2v3ZDUtfumZe6uaqlUDvPpP0kLjvzg8ZYzGhnyeSgRqgfRnZ2VHsU8EDTLlqCVERyhNQ GysX7+mwRtgQ8LkLzdmCc85F3zn5o09yKwZTTCiSOkYI8mb8vrQZSIt/pGSh8h09DqNA wDd1fM3PLW5PeS6scnkV8PC03l4iui58Wg92j7B/4Aky+rAZEyF4kMqT3+hQtagLl+I1 LenscT9o+kyuzzZ7Z9Twzex0dkaiJpABi1E3aYFBvWYNU8jl7RFXIB+PZFp6Byuhl2iY qdaw== X-Gm-Message-State: AKS2vOw0RF1KAlHVcIsd2PI4SxHNLqALX/OU8j6M371+/ZTZ3kLwhnSc oQ5tF391TLUCM4nG X-Received: by 10.223.162.208 with SMTP id t16mr5704157wra.151.1497994239738; Tue, 20 Jun 2017 14:30:39 -0700 (PDT) Received: from localhost.localdomain ([2a01:e35:879a:6cd0:ccbe:9633:e651:d298]) by smtp.gmail.com with ESMTPSA id o7sm13044234wmd.0.2017.06.20.14.30.38 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 20 Jun 2017 14:30:39 -0700 (PDT) From: Daniel Lezcano To: tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, Peter Zijlstra , "Rafael J . Wysocki" , Vincent Guittot , Nicolas Pitre , Jens Axboe , Hannes Reinecke , Bjorn Helgaas Subject: [PATCH 3/3] irq: Compute the periodic interval for interrupts Date: Tue, 20 Jun 2017 23:28:57 +0200 Message-Id: <1497994139-19816-3-git-send-email-daniel.lezcano@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1497994139-19816-1-git-send-email-daniel.lezcano@linaro.org> References: <20170620212657.GB1812@mai> <1497994139-19816-1-git-send-email-daniel.lezcano@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org An interrupt behaves with a burst of activity with periodic interval of time followed by one or two peaks of longer interval. As the time intervals are periodic, statistically speaking they follow a normal distribution and each interrupts can be tracked individually. This patch does statistics on all interrupts, except the timers which are deterministic by essence. The goal is to extract the periodicity for each interrupt, with the last timestamp and sum them, so we have the next event. Taking the earliest prediction gives the expected wakeup on the system (assuming a timer won't expire before). As stated in the previous patch, this code is not enabled in the kernel by default. Signed-off-by: Daniel Lezcano Cc: Peter Zijlstra Cc: Rafael J. Wysocki Cc: Vincent Guittot Cc: Nicolas Pitre --- Changelog: V11: - No changes V10: - Simplified index/count computation V9: - Deal with 48+16 bits encoded values - Changed irq_stat => irqt_stat to prevent name collision on s390 - Changed div64 by constant IRQ_TIMINGS_SHIFT bit shift for average - Changed div64 by constant IRQ_TIMINGS_SHIFT bit shift for variance Signed-off-by: Daniel Lezcano --- include/linux/interrupt.h | 1 + kernel/irq/internals.h | 21 ++- kernel/irq/timings.c | 336 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 357 insertions(+), 1 deletion(-) -- 2.7.4 diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h index 5f98aa7..5ac6e23 100644 --- a/include/linux/interrupt.h +++ b/include/linux/interrupt.h @@ -715,6 +715,7 @@ static inline void init_irq_proc(void) #ifdef CONFIG_IRQ_TIMINGS void irq_timings_enable(void); void irq_timings_disable(void); +u64 irq_timings_next_event(u64 now); #endif struct seq_file; diff --git a/kernel/irq/internals.h b/kernel/irq/internals.h index df51b5e0..bf3d827 100644 --- a/kernel/irq/internals.h +++ b/kernel/irq/internals.h @@ -237,18 +237,26 @@ irq_pm_remove_action(struct irq_desc *desc, struct irqaction *action) { } struct irq_timings { u64 values[IRQ_TIMINGS_SIZE]; /* our circular buffer */ - unsigned int count; /* Number of interruptions since last inspection */ + int count; /* Number of interruptions since last inspection */ }; DECLARE_PER_CPU(struct irq_timings, irq_timings); +extern void irq_timings_free(int irq); +extern int irq_timings_alloc(int irq); + static inline void remove_timings(struct irq_desc *desc) { desc->istate &= ~IRQS_TIMINGS; + + irq_timings_free(irq_desc_get_irq(desc)); } static inline void setup_timings(struct irq_desc *desc, struct irqaction *act) { + int irq = irq_desc_get_irq(desc); + int ret; + /* * We don't need the measurement because the idle code already * knows the next expiry event. @@ -256,6 +264,17 @@ static inline void setup_timings(struct irq_desc *desc, struct irqaction *act) if (act->flags & __IRQF_TIMER) return; + /* + * In case the timing allocation fails, we just want to warn, + * not fail, so letting the system boot anyway. + */ + ret = irq_timings_alloc(irq); + if (ret) { + pr_warn("Failed to allocate irq timing stats for irq%d (%d)", + irq, ret); + return; + } + desc->istate |= IRQS_TIMINGS; } diff --git a/kernel/irq/timings.c b/kernel/irq/timings.c index 56cf687..4af7703 100644 --- a/kernel/irq/timings.c +++ b/kernel/irq/timings.c @@ -8,10 +8,16 @@ * published by the Free Software Foundation. * */ +#include #include +#include #include #include +#include #include +#include + +#include #include "internals.h" @@ -19,6 +25,18 @@ DEFINE_STATIC_KEY_FALSE(irq_timing_enabled); DEFINE_PER_CPU(struct irq_timings, irq_timings); +struct irqt_stat { + u64 ne; /* next event */ + u64 lts; /* last timestamp */ + u64 variance; /* variance */ + u32 avg; /* mean value */ + u32 count; /* number of samples */ + int anomalies; /* number of consecutives anomalies */ + int valid; /* behaviour of the interrupt */ +}; + +static DEFINE_IDR(irqt_stats); + void irq_timings_enable(void) { static_branch_enable(&irq_timing_enabled); @@ -28,3 +46,321 @@ void irq_timings_disable(void) { static_branch_disable(&irq_timing_enabled); } + +/** + * irqs_update - update the irq timing statistics with a new timestamp + * + * @irqs: an irqt_stat struct pointer + * @ts: the new timestamp + * + * ** This function must be called with the local irq disabled ** + * + * The statistics are computed online, in other words, the code is + * designed to compute the statistics on a stream of values rather + * than doing multiple passes on the values to compute the average, + * then the variance. The integer division introduces a loss of + * precision but with an acceptable error margin regarding the results + * we would have with the double floating precision: we are dealing + * with nanosec, so big numbers, consequently the mantisse is + * negligeable, especially when converting the time in usec + * afterwards. + * + * The computation happens at idle time. When the CPU is not idle, the + * interrupts' timestamps are stored in the circular buffer, when the + * CPU goes idle and this routine is called, all the buffer's values + * are injected in the statistical model continuying to extend the + * statistics from the previous busy-idle cycle. + * + * The observations showed a device will trigger a burst of periodic + * interrupts followed by one or two peaks of longer time, for + * instance when a SD card device flushes its cache, then the periodic + * intervals occur again. A one second inactivity period resets the + * stats, that gives us the certitude the statistical values won't + * exceed 1x10^9, thus the computation won't overflow. + * + * Basically, the purpose of the algorithm is to watch the periodic + * interrupts and eliminate the peaks. + * + * An interrupt is considered periodically stable if the interval of + * its occurences follow the normal distribution, thus the values + * comply with: + * + * avg - 3 x stddev < value < avg + 3 x stddev + * + * Which can be simplified to: + * + * -3 x stddev < value - avg < 3 x stddev + * + * abs(value - avg) < 3 x stddev + * + * In order to save a costly square root computation, we use the + * variance. For the record, stddev = sqrt(variance). The equation + * above becomes: + * + * abs(value - avg) < 3 x sqrt(variance) + * + * And finally we square it: + * + * (value - avg) ^ 2 < (3 x sqrt(variance)) ^ 2 + * + * (value - avg) x (value - avg) < 9 x variance + * + * Statistically speaking, any values out of this interval is + * considered as an anomaly and is discarded. However, a normal + * distribution appears when the number of samples is 30 (it is the + * rule of thumb in statistics, cf. "30 samples" on Internet). When + * there are three consecutive anomalies, the statistics are resetted. + * + */ +static void irqs_update(struct irqt_stat *irqs, u64 ts) +{ + u64 old_ts = irqs->lts; + u64 variance = 0; + u64 interval; + s64 diff; + + /* + * The timestamps are absolute time values, we need to compute + * the timing interval between two interrupts. + */ + irqs->lts = ts; + + /* + * The interval type is u64 in order to deal with the same + * type in our computation, that prevent mindfuck issues with + * overflow, sign and division. + */ + interval = ts - old_ts; + + /* + * The interrupt triggered more than one second apart, that + * ends the sequence as predictible for our purpose. In this + * case, assume we have the beginning of a sequence and the + * timestamp is the first value. As it is impossible to + * predict anything at this point, return. + * + * Note the first timestamp of the sequence will always fall + * in this test because the old_ts is zero. That is what we + * want as we need another timestamp to compute an interval. + */ + if (interval >= NSEC_PER_SEC) { + memset(irqs, 0, sizeof(*irqs)); + irqs->lts = ts; + return; + } + + /* + * Pre-compute the delta with the average as the result is + * used several times in this function. + */ + diff = interval - irqs->avg; + + /* + * Increment the number of samples. + */ + irqs->count++; + + /* + * Online variance divided by the number of elements if there + * is more than one sample. Normally the formula is division + * by count - 1 but we assume the number of element will be + * more than 32 and dividing by 32 instead of 31 is enough + * precise. + */ + if (likely(irqs->count > 1)) + variance = irqs->variance >> IRQ_TIMINGS_SHIFT; + + /* + * The rule of thumb in statistics for the normal distribution + * is having at least 30 samples in order to have the model to + * apply. Values outside the interval are considered as an + * anomaly. + */ + if ((irqs->count >= 30) && ((diff * diff) > (9 * variance))) { + /* + * After three consecutive anomalies, we reset the + * stats as it is no longer stable enough. + */ + if (irqs->anomalies++ >= 3) { + memset(irqs, 0, sizeof(*irqs)); + irqs->lts = ts; + return; + } + } else { + /* + * The anomalies must be consecutives, so at this + * point, we reset the anomalies counter. + */ + irqs->anomalies = 0; + } + + /* + * The interrupt is considered stable enough to try to predict + * the next event on it. + */ + irqs->valid = 1; + + /* + * Online average algorithm: + * + * new_average = average + ((value - average) / count) + * + * The variance computation depends on the new average + * to be computed here first. + * + */ + irqs->avg = irqs->avg + (diff >> IRQ_TIMINGS_SHIFT); + + /* + * Online variance algorithm: + * + * new_variance = variance + (value - average) x (value - new_average) + * + * Warning: irqs->avg is updated with the line above, hence + * 'interval - irqs->avg' is no longer equal to 'diff' + */ + irqs->variance = irqs->variance + (diff * (interval - irqs->avg)); + + /* + * Update the next event + */ + irqs->ne = ts + irqs->avg; +} + +/** + * irq_timings_next_event - Return when the next event is supposed to arrive + * + * *** This function must be called with the local irq disabled *** + * + * During the last busy cycle, the number of interrupts is incremented + * and stored in the irq_timings structure. This information is + * necessary to: + * + * - know if the index in the table wrapped up: + * + * If more than the array size interrupts happened during the + * last busy/idle cycle, the index wrapped up and we have to + * begin with the next element in the array which is the last one + * in the sequence, otherwise it is a the index 0. + * + * - have an indication of the interrupts activity on this CPU + * (eg. irq/sec) + * + * The values are 'consumed' after inserting in the statistical model, + * thus the count is reinitialized. + * + * The array of values **must** be browsed in the time direction, the + * timestamp must increase between an element and the next one. + * + * Returns a nanosec time based estimation of the earliest interrupt, + * U64_MAX otherwise. + */ +u64 irq_timings_next_event(u64 now) +{ + struct irq_timings *irqts = this_cpu_ptr(&irq_timings); + struct irqt_stat *irqs; + struct irqt_stat __percpu *s; + u64 ts, ne = U64_MAX; + int i, irq = 0; + + /* + * Number of elements in the circular buffer: If it happens it + * was flushed before, then the number of elements could be + * smaller than IRQ_TIMINGS_SIZE, so the count is used, + * otherwise the array size is used as we wrapped. The index + * begins from zero when we did not wrap. That could be done + * in a nicer way with the proper circular array structure + * type but with the cost of extra computation in the + * interrupt handler hot path. We choose efficiency. + * + * Inject measured irq/timestamp to the statistical model + * while decrementing the counter because we consume the data + * from our circular buffer. + */ + for (i = irqts->count & IRQ_TIMINGS_MASK, + irqts->count = min(IRQ_TIMINGS_SIZE, irqts->count); + irqts->count > 0; irqts->count--, i = (i + 1) & IRQ_TIMINGS_MASK) { + + irq_timing_decode(irqts->values[i], &ts, &irq); + + s = idr_find(&irqt_stats, irq); + if (s) { + irqs = this_cpu_ptr(s); + irqs_update(irqs, ts); + } + } + + /* + * Look in the list of interrupts' statistics, the earliest + * next event. + */ + idr_for_each_entry(&irqt_stats, s, i) { + + irqs = this_cpu_ptr(s); + + if (!irqs->valid) + continue; + + if (irqs->ne <= now) { + irq = i; + ne = now; + + /* + * This interrupt mustn't use in the future + * until new events occur and update the + * statistics. + */ + irqs->valid = 0; + break; + } + + if (irqs->ne < ne) { + irq = i; + ne = irqs->ne; + } + } + + return ne; +} + +void irq_timings_free(int irq) +{ + struct irqt_stat __percpu *s; + + s = idr_find(&irqt_stats, irq); + if (s) { + free_percpu(s); + idr_remove(&irqt_stats, irq); + } +} + +int irq_timings_alloc(int irq) +{ + int id; + struct irqt_stat __percpu *s; + + /* + * Some platforms can have the same private interrupt per cpu, + * so this function may be be called several times with the + * same interrupt number. Just bail out in case the per cpu + * stat structure is already allocated. + */ + s = idr_find(&irqt_stats, irq); + if (s) + return 0; + + s = alloc_percpu(*s); + if (!s) + return -ENOMEM; + + idr_preload(GFP_KERNEL); + id = idr_alloc(&irqt_stats, s, irq, irq + 1, GFP_NOWAIT); + idr_preload_end(); + + if (id < 0) { + free_percpu(s); + return id; + } + + return 0; +}