From patchwork Thu Jun 23 13:18:38 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Arnd Bergmann X-Patchwork-Id: 102110 Delivered-To: patch@linaro.org Received: by 10.140.28.4 with SMTP id 4csp428695qgy; Thu, 23 Jun 2016 06:17:13 -0700 (PDT) X-Received: by 10.98.206.77 with SMTP id y74mr42929028pfg.55.1466687833676; Thu, 23 Jun 2016 06:17:13 -0700 (PDT) Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id vx8si114557pac.107.2016.06.23.06.17.13; Thu, 23 Jun 2016 06:17:13 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751664AbcFWNQu (ORCPT + 30 others); Thu, 23 Jun 2016 09:16:50 -0400 Received: from mout.kundenserver.de ([212.227.17.13]:51786 "EHLO mout.kundenserver.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751037AbcFWNQt (ORCPT ); Thu, 23 Jun 2016 09:16:49 -0400 Received: from wuerfel.lan. ([78.42.132.4]) by mrelayeu.kundenserver.de (mreue103) with ESMTPA (Nemesis) id 0Lb8B7-1bdpaZ0QDR-00khNS; Thu, 23 Jun 2016 15:16:38 +0200 From: Arnd Bergmann To: Mel Gorman Cc: Vlastimil Babka , Johannes Weiner , Rik van Riel , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Arnd Bergmann Subject: [RFC, DEBUGGING v2 1/2] mm: pass NR_FILE_PAGES/NR_SHMEM into node_page_state Date: Thu, 23 Jun 2016 15:18:38 +0200 Message-Id: <20160623131839.3579472-1-arnd@arndb.de> X-Mailer: git-send-email 2.9.0 In-Reply-To: <3817461.6pThRKgN9N@wuerfel> References: <3817461.6pThRKgN9N@wuerfel> X-Provags-ID: V03:K0:hSvRT+3vWJCI3z9gL/+UXsFKaNIjM0w77zqX+dO5h2JV0LOX12Q oSbrvi7kgD1pMxyj8Pba15ce70N7KgROLc3HJy0goR/5rPMv33aKDzVm9vPFrl8VdF3rKC2 PDhX+XySlLQaCms6tt1aVx8QqM5syTNqAbbe/1hPh1bvUDgIiz93HRsijldApIJOJnhQ0bi U3cvf+K79zZ6EYQ1IF/FQ== X-UI-Out-Filterresults: notjunk:1; V01:K0:1R1jBCciglI=:DJG6Am9UCzYciiBMfZ5MkH 0SYsDo6gp/8dpB1XfUzmPQn68FOjuZjQ7SpUpvfaPKnqFUZj8q/NZl0pMTX8/fZrx7QeKX5db uM2n8tHPArINi5cwbTrEn3v5tqVHrl9w17NgavhI7WEuS7k61mTCmZW0pLOp/JOpi/b8jInbB fB2dA3jM5ZOsE4/rx7odK2mCe1ofckjDup38wr3mMBO0CFpNPoVijCVQqzCiAh5LqM1P9gV1O /zIX//07xFdI2UgK5Qd/oawm4mhLvMx9g9jZp623kwlYq4ArV84wmvKRU4m3drga374qC3Vud KH4j/OOg7DgeQqVCmzoMzz7T9w+dwhg9b2xixeYm6LeKNkDc09wT6ZkxFfD90fHLE6Sf/1ivr n04F+egN7MxqMhCOQnjr6RSTizhqRcVNOKcGT0PmPnJmJXK5M6k/C46ED5tdIUqNUBIOkm0a+ No/yG8Gc4eVMfhz/aZEwCDahf945P3a3tsvErc29jWvV5zMmoPi5J/4toINzjQ1SxC6zuYB2N wjursbYxd9FvOimOTJGhMhaSy+P5NVOj0nygIDxriDMDqOcRAMyXpW8ew7pyBgl7Aml196wkx y1hszDe/8IMFw8B/uSHY2n2t3ByAekwZ+/d/GWe1mrPXzCLbtK6Q4lstrWhsl+ALFAplf61hJ iAu8fL2HcyLtShoUZyMCOoMwcPJfaFmYWs0yISMkdiIoTblFr+cpi/3LC0kkeKL2dQp4= Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org I see some new warnings from a recent mm change: mm/filemap.c: In function '__delete_from_page_cache': include/linux/vmstat.h:116:2: error: array subscript is above array bounds [-Werror=array-bounds] atomic_long_add(x, &zone->vm_stat[item]); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ include/linux/vmstat.h:116:35: error: array subscript is above array bounds [-Werror=array-bounds] atomic_long_add(x, &zone->vm_stat[item]); ~~~~~~~~~~~~~^~~~~~ include/linux/vmstat.h:116:35: error: array subscript is above array bounds [-Werror=array-bounds] include/linux/vmstat.h:117:2: error: array subscript is above array bounds [-Werror=array-bounds] Looking deeper into it, I find that we pass the wrong enum into some functions after the type for the symbol has changed. This changes the code to use the other function for those that are using the incorrect type. I've done this blindly just going by warnings I got from a debug patch I did for this, so it's likely that some cases are more subtle and need another change, so please treat this as a bug-report rather than a patch for applying. Signed-off-by: Arnd Bergmann Fixes: e426f7b4ade5 ("mm: move most file-based accounting to the node") --- mm/filemap.c | 4 ++-- mm/khugepaged.c | 4 ++-- mm/page_alloc.c | 15 ++++++++------- mm/rmap.c | 4 ++-- mm/shmem.c | 4 ++-- mm/vmscan.c | 2 +- 6 files changed, 17 insertions(+), 16 deletions(-) -- 2.9.0 diff --git a/mm/filemap.c b/mm/filemap.c index 6cb19e012887..e0fe47e9ea44 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -218,9 +218,9 @@ void __delete_from_page_cache(struct page *page, void *shadow) /* hugetlb pages do not participate in page cache accounting. */ if (!PageHuge(page)) - __mod_zone_page_state(page_zone(page), NR_FILE_PAGES, -nr); + __mod_node_page_state(page_pgdat(page), NR_FILE_PAGES, -nr); if (PageSwapBacked(page)) { - __mod_zone_page_state(page_zone(page), NR_SHMEM, -nr); + __mod_node_page_state(page_pgdat(page), NR_SHMEM, -nr); if (PageTransHuge(page)) __dec_zone_page_state(page, NR_SHMEM_THPS); } else { diff --git a/mm/khugepaged.c b/mm/khugepaged.c index af256d599080..0efda0345aed 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1476,8 +1476,8 @@ tree_unlocked: local_irq_save(flags); __inc_zone_page_state(new_page, NR_SHMEM_THPS); if (nr_none) { - __mod_zone_page_state(zone, NR_FILE_PAGES, nr_none); - __mod_zone_page_state(zone, NR_SHMEM, nr_none); + __mod_node_page_state(zone->zone_pgdat, NR_FILE_PAGES, nr_none); + __mod_node_page_state(zone->zone_pgdat, NR_SHMEM, nr_none); } local_irq_restore(flags); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 23b5044f5ced..277dc0cbe780 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3484,9 +3484,10 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order, unsigned long writeback; unsigned long dirty; - writeback = zone_page_state_snapshot(zone, + writeback = node_page_state_snapshot(zone->zone_pgdat, NR_WRITEBACK); - dirty = zone_page_state_snapshot(zone, NR_FILE_DIRTY); + dirty = node_page_state_snapshot(zone->zone_pgdat, + NR_FILE_DIRTY); if (2*(writeback + dirty) > reclaimable) { congestion_wait(BLK_RW_ASYNC, HZ/10); @@ -4396,9 +4397,9 @@ void show_free_areas(unsigned int filter) K(zone->present_pages), K(zone->managed_pages), K(zone_page_state(zone, NR_MLOCK)), - K(zone_page_state(zone, NR_FILE_DIRTY)), - K(zone_page_state(zone, NR_WRITEBACK)), - K(zone_page_state(zone, NR_SHMEM)), + K(node_page_state(zone->zone_pgdat, NR_FILE_DIRTY)), + K(node_page_state(zone->zone_pgdat, NR_WRITEBACK)), + K(node_page_state(zone->zone_pgdat, NR_SHMEM)), #ifdef CONFIG_TRANSPARENT_HUGEPAGE K(zone_page_state(zone, NR_SHMEM_THPS) * HPAGE_PMD_NR), K(zone_page_state(zone, NR_SHMEM_PMDMAPPED) @@ -4410,12 +4411,12 @@ void show_free_areas(unsigned int filter) zone_page_state(zone, NR_KERNEL_STACK) * THREAD_SIZE / 1024, K(zone_page_state(zone, NR_PAGETABLE)), - K(zone_page_state(zone, NR_UNSTABLE_NFS)), + K(node_page_state(zone->zone_pgdat, NR_UNSTABLE_NFS)), K(zone_page_state(zone, NR_BOUNCE)), K(free_pcp), K(this_cpu_read(zone->pageset->pcp.count)), K(zone_page_state(zone, NR_FREE_CMA_PAGES)), - K(zone_page_state(zone, NR_WRITEBACK_TEMP)), + K(node_page_state(zone->zone_pgdat, NR_WRITEBACK_TEMP)), K(node_page_state(zone->zone_pgdat, NR_PAGES_SCANNED))); printk("lowmem_reserve[]:"); for (i = 0; i < MAX_NR_ZONES; i++) diff --git a/mm/rmap.c b/mm/rmap.c index 4deff963ea8a..a66f80bc8703 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1296,7 +1296,7 @@ void page_add_file_rmap(struct page *page, bool compound) if (!atomic_inc_and_test(&page->_mapcount)) goto out; } - __mod_zone_page_state(page_zone(page), NR_FILE_MAPPED, nr); + __mod_node_page_state(page_pgdat(page), NR_FILE_MAPPED, nr); mem_cgroup_inc_page_stat(page, MEM_CGROUP_STAT_FILE_MAPPED); out: unlock_page_memcg(page); @@ -1336,7 +1336,7 @@ static void page_remove_file_rmap(struct page *page, bool compound) * these counters are not modified in interrupt context, and * pte lock(a spinlock) is held, which implies preemption disabled. */ - __mod_zone_page_state(page_zone(page), NR_FILE_MAPPED, -nr); + __mod_node_page_state(page_pgdat(page), NR_FILE_MAPPED, -nr); mem_cgroup_dec_page_stat(page, MEM_CGROUP_STAT_FILE_MAPPED); if (unlikely(PageMlocked(page))) diff --git a/mm/shmem.c b/mm/shmem.c index e5c50fb0d4a4..a03c087f71fe 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -576,8 +576,8 @@ static int shmem_add_to_page_cache(struct page *page, mapping->nrpages += nr; if (PageTransHuge(page)) __inc_zone_page_state(page, NR_SHMEM_THPS); - __mod_zone_page_state(page_zone(page), NR_FILE_PAGES, nr); - __mod_zone_page_state(page_zone(page), NR_SHMEM, nr); + __mod_node_page_state(page_pgdat(page), NR_FILE_PAGES, nr); + __mod_node_page_state(page_pgdat(page), NR_SHMEM, nr); spin_unlock_irq(&mapping->tree_lock); } else { page->mapping = NULL; diff --git a/mm/vmscan.c b/mm/vmscan.c index 07e17dac1793..4702069cc80b 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2079,7 +2079,7 @@ static void get_scan_count(struct lruvec *lruvec, struct mem_cgroup *memcg, int z; unsigned long total_high_wmark = 0; - pgdatfree = sum_zone_node_page_state(pgdat->node_id, NR_FREE_PAGES); + pgdatfree = global_page_state(NR_FREE_PAGES); pgdatfile = node_page_state(pgdat, NR_ACTIVE_FILE) + node_page_state(pgdat, NR_INACTIVE_FILE); From patchwork Thu Jun 23 13:18:39 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Arnd Bergmann X-Patchwork-Id: 102111 Delivered-To: patch@linaro.org Received: by 10.140.28.4 with SMTP id 4csp428540qgy; Thu, 23 Jun 2016 06:16:53 -0700 (PDT) X-Received: by 10.66.135.40 with SMTP id pp8mr42869808pab.113.1466687813682; Thu, 23 Jun 2016 06:16:53 -0700 (PDT) Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id zq4si105432pac.130.2016.06.23.06.16.53; Thu, 23 Jun 2016 06:16:53 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751717AbcFWNQv (ORCPT + 30 others); Thu, 23 Jun 2016 09:16:51 -0400 Received: from mout.kundenserver.de ([212.227.17.13]:52617 "EHLO mout.kundenserver.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751040AbcFWNQt (ORCPT ); Thu, 23 Jun 2016 09:16:49 -0400 Received: from wuerfel.lan. ([78.42.132.4]) by mrelayeu.kundenserver.de (mreue103) with ESMTPA (Nemesis) id 0MQcnt-1aoCuB1ne1-00TzsE; Thu, 23 Jun 2016 15:16:38 +0200 From: Arnd Bergmann To: Mel Gorman Cc: Vlastimil Babka , Johannes Weiner , Rik van Riel , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Arnd Bergmann Subject: [RFC, DEBUGGING v2 2/2] mm: add type checking for page state functions Date: Thu, 23 Jun 2016 15:18:39 +0200 Message-Id: <20160623131839.3579472-2-arnd@arndb.de> X-Mailer: git-send-email 2.9.0 In-Reply-To: <20160623131839.3579472-1-arnd@arndb.de> References: <3817461.6pThRKgN9N@wuerfel> <20160623131839.3579472-1-arnd@arndb.de> X-Provags-ID: V03:K0:h1gLrZu7gZ9fMh3eZKRHIwxwAOrclqUsLNHhEZd6qXawBri7eMJ uw1a9Qm4HmqGipWQmuj6VjDf6mcWynvxXgT6I0+XF0aenQvf0qk19qKk8m9NL2jOpAS5384 uHg1QDCifJQCdoA5sVg2ROhrxKvN9CwFwOUHQE9VyOX4YXcmx9MxZ6ZWeVlvVi6REbTVx49 0GFKCr3uKkYkNHn3Qa9lw== X-UI-Out-Filterresults: notjunk:1; V01:K0:NmNYboeZMjo=:Nztql5m88l+qUlrahGB/MI a1i8FqDLi+Eo//85nzZXTcEGNQBJh2xNyAOAFl8uh3U5ZwYIeoBnV857ViHTsFiXbs2Ryr3Xe DzOaD4sYd66bgIjVld7+AglVaTkVdXhQmxqR23xfbVGRjKzFr2Mtfmp/+N+M2SufWHQWgCPHi G0RrFB1MM5EvTjdhOz6uO42qN5vDUjZRKWrm2pgvFfbEXiRgA27K9Vftgpf0Mb3zR5b40i3FD UW9oLRmqSvEgblrDISaU/8/6RB4braK+ownVmmuN5vPN6NvzCBkoStwZwmY89ZdWPUgbtPBjT MvtUmcXWNquvsn4BASkJqm4T+MvwXdabrRigjU9d9+I0niPzpMCga/ShtqO4p/UkxSK8leDa8 62snpvrYv0qIe6BmmqRsFQ+wkfXeVQ29NekRnxk73eyjK6sFjsbcMF7cUc/Ith7LoxrnsnnDF fIyguB1nOLoYMdKY19L+HZBcm1nQaNqlNsNX6M98vfcfcEoC31iCilIjw346qP2l8mY1YXngS G3sSkKDeitGAG2GfucqWd1UEBQwjo2VqL1rcm/rqM4b9vUrNQbXDrvBUlCYSIRCQHf8C8iO7g TIVUSwumP37eSaSrqIw4HQmviVyCZOi99JgwFgSGTI9HN19V+te5k/XZS2Dtu2pfPiqRpdA7v ERXablLqDGs4PNG921+neksX+mmFHZURu4OUQ1rRF3MmGCCTSFIgxtzaiSpcN1JZyTBY= Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org We had a couple of bugs where we pass the incorrect 'enum' into one of the statistics functions, and unfortunately gcc can only warn about comparing distinct enum types rather than warning about passing an enum of the wrong type into a function. This wraps all the stats calls inside of macros that add the type checking using a comparison. This second version is better compile-tested, but it's also really ugly. Signed-off-by: Arnd Bergmann --- include/linux/vmstat.h | 158 ++++++++++++++++++++++++++++++++++--------------- mm/vmstat.c | 136 +++++++++++++++++++++--------------------- 2 files changed, 178 insertions(+), 116 deletions(-) -- 2.9.0 diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index c799073fe1c4..390b7ae3efb2 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -110,21 +110,25 @@ static inline void vm_events_fold_cpu(int cpu) extern atomic_long_t vm_zone_stat[NR_VM_ZONE_STAT_ITEMS]; extern atomic_long_t vm_node_stat[NR_VM_NODE_STAT_ITEMS]; -static inline void zone_page_state_add(long x, struct zone *zone, +static inline void zone_page_state_add_check(long x, struct zone *zone, enum zone_stat_item item) { atomic_long_add(x, &zone->vm_stat[item]); atomic_long_add(x, &vm_zone_stat[item]); } +#define zone_page_state_add(x, zone, item) \ + zone_page_state_add_check(x, zone, ((item) == (enum zone_stat_item )0) ? (item) : (item)) -static inline void node_page_state_add(long x, struct pglist_data *pgdat, +static inline void node_page_state_add_check(long x, struct pglist_data *pgdat, enum node_stat_item item) { atomic_long_add(x, &pgdat->vm_stat[item]); atomic_long_add(x, &vm_node_stat[item]); } +#define node_page_state_add(x, node, item) \ + node_page_state_add_check(x, node, ((item) == (enum node_stat_item )0) ? (item) : (item)) -static inline unsigned long global_page_state(enum zone_stat_item item) +static inline unsigned long global_page_state_check(enum zone_stat_item item) { long x = atomic_long_read(&vm_zone_stat[item]); #ifdef CONFIG_SMP @@ -133,8 +137,10 @@ static inline unsigned long global_page_state(enum zone_stat_item item) #endif return x; } +#define global_page_state(item) \ + global_page_state_check(((item) == (enum zone_stat_item )0) ? (item) : (item)) -static inline unsigned long global_node_page_state(enum node_stat_item item) +static inline unsigned long global_node_page_state_check(enum node_stat_item item) { long x = atomic_long_read(&vm_node_stat[item]); #ifdef CONFIG_SMP @@ -143,8 +149,10 @@ static inline unsigned long global_node_page_state(enum node_stat_item item) #endif return x; } +#define global_node_page_state(item) \ + global_node_page_state_check(((item) == (enum node_stat_item )0) ? (item) : (item)) -static inline unsigned long zone_page_state(struct zone *zone, +static inline unsigned long zone_page_state_check(struct zone *zone, enum zone_stat_item item) { long x = atomic_long_read(&zone->vm_stat[item]); @@ -154,6 +162,8 @@ static inline unsigned long zone_page_state(struct zone *zone, #endif return x; } +#define zone_page_state(zone, item) \ + zone_page_state_check(zone, ((item) == (enum zone_stat_item )0) ? (item) : (item)) /* * More accurate version that also considers the currently pending @@ -161,7 +171,7 @@ static inline unsigned long zone_page_state(struct zone *zone, * deltas. There is no synchronization so the result cannot be * exactly accurate either. */ -static inline unsigned long zone_page_state_snapshot(struct zone *zone, +static inline unsigned long zone_page_state_snapshot_check(struct zone *zone, enum zone_stat_item item) { long x = atomic_long_read(&zone->vm_stat[item]); @@ -176,8 +186,10 @@ static inline unsigned long zone_page_state_snapshot(struct zone *zone, #endif return x; } +#define zone_page_state_snapshot(zone, item) \ + zone_page_state_snapshot_check(zone, ((item) == (enum zone_stat_item )0) ? (item) : (item)) -static inline unsigned long node_page_state_snapshot(pg_data_t *pgdat, +static inline unsigned long node_page_state_snapshot_check(pg_data_t *pgdat, enum zone_stat_item item) { long x = atomic_long_read(&pgdat->vm_stat[item]); @@ -192,13 +204,18 @@ static inline unsigned long node_page_state_snapshot(pg_data_t *pgdat, #endif return x; } - +#define node_page_state_snapshot(zone, item) \ + node_page_state_snapshot_check(zone, ((item) == (enum node_stat_item )0) ? (item) : (item)) #ifdef CONFIG_NUMA -extern unsigned long sum_zone_node_page_state(int node, +extern unsigned long sum_zone_node_page_state_check(int node, enum zone_stat_item item); -extern unsigned long node_page_state(struct pglist_data *pgdat, +#define sum_zone_node_page_state(node, item) \ + sum_zone_node_page_state_check(node, ((item) == (enum node_stat_item )0) ? (item) : (item)) +extern unsigned long node_page_state_check(struct pglist_data *pgdat, enum node_stat_item item); +#define node_page_state(pgdat, item) \ + node_page_state_check(pgdat, ((item) == (enum node_stat_item )0) ? (item) : (item)) #else #define sum_zone_node_page_state(node, item) global_node_page_state(item) #define node_page_state(node, item) global_node_page_state(item) @@ -210,28 +227,52 @@ extern unsigned long node_page_state(struct pglist_data *pgdat, #define sub_node_page_state(__p, __i, __d) mod_node_page_state(__p, __i, -(__d)) #ifdef CONFIG_SMP -void __mod_zone_page_state(struct zone *, enum zone_stat_item item, long); -void __inc_zone_page_state(struct page *, enum zone_stat_item); -void __dec_zone_page_state(struct page *, enum zone_stat_item); - -void __mod_node_page_state(struct pglist_data *, enum node_stat_item item, long); -void __inc_node_page_state(struct page *, enum node_stat_item); -void __dec_node_page_state(struct page *, enum node_stat_item); - -void mod_zone_page_state(struct zone *, enum zone_stat_item, long); -void inc_zone_page_state(struct page *, enum zone_stat_item); -void dec_zone_page_state(struct page *, enum zone_stat_item); - -void mod_node_page_state(struct pglist_data *, enum node_stat_item, long); -void inc_node_page_state(struct page *, enum node_stat_item); -void dec_node_page_state(struct page *, enum node_stat_item); - -extern void inc_node_state(struct pglist_data *, enum node_stat_item); -extern void __inc_zone_state(struct zone *, enum zone_stat_item); -extern void __inc_node_state(struct pglist_data *, enum node_stat_item); -extern void dec_zone_state(struct zone *, enum zone_stat_item); -extern void __dec_zone_state(struct zone *, enum zone_stat_item); -extern void __dec_node_state(struct pglist_data *, enum node_stat_item); +void __mod_zone_page_state_check(struct zone *, enum zone_stat_item item, long); +void __inc_zone_page_state_check(struct page *, enum zone_stat_item); +void __dec_zone_page_state_check(struct page *, enum zone_stat_item); + +void __mod_node_page_state_check(struct pglist_data *, enum node_stat_item item, long); +void __inc_node_page_state_check(struct page *, enum node_stat_item); +void __dec_node_page_state_check(struct page *, enum node_stat_item); + +void mod_zone_page_state_check(struct zone *, enum zone_stat_item, long); +void inc_zone_page_state_check(struct page *, enum zone_stat_item); +void dec_zone_page_state_check(struct page *, enum zone_stat_item); + +#define mod_zone_page_state(zone, item, delta) \ + mod_zone_page_state_check(zone, ((item) == (enum zone_stat_item )0) ? (item) : (item), delta) +#define inc_zone_page_state(page, item) \ + inc_zone_page_state_check(page, ((item) == (enum zone_stat_item )0) ? (item) : (item)) +#define dec_zone_page_state(page, item) \ + dec_zone_page_state_check(page, ((item) == (enum zone_stat_item )0) ? (item) : (item)) + +void mod_node_page_state_check(struct pglist_data *, enum node_stat_item, long); +void inc_node_page_state_check(struct page *, enum node_stat_item); +void dec_node_page_state_check(struct page *, enum node_stat_item); + +#define mod_node_page_state(pgdat, item, delta) \ + mod_node_page_state_check(pgdat, ((item) == (enum node_stat_item )0) ? (item) : (item), delta) +#define inc_node_page_state(page, item) \ + inc_node_page_state_check(page, ((item) == (enum node_stat_item )0) ? (item) : (item)) +#define dec_node_page_state(page, item) \ + dec_node_page_state_check(page, ((item) == (enum node_stat_item )0) ? (item) : (item)) + +extern void inc_node_state_check(struct pglist_data *, enum node_stat_item); +extern void __inc_zone_state_check(struct zone *, enum zone_stat_item); +extern void __inc_node_state_check(struct pglist_data *, enum node_stat_item); +extern void dec_zone_state_check(struct zone *, enum zone_stat_item); +extern void __dec_zone_state_check(struct zone *, enum zone_stat_item); +extern void __dec_node_state_check(struct pglist_data *, enum node_stat_item); + +#define inc_node_state(pgdat, item) \ + inc_node_state_check(pgdat, ((item) == (enum node_stat_item )0) ? (item) : (item)) +#define dec_node_state(pgdat, item) \ + dec_node_state_check(pgdat, ((item) == (enum node_stat_item )0) ? (item) : (item)) + +#define inc_zone_state(zone, item) \ + inc_zone_state_check(zone, ((item) == (enum zone_stat_item )0) ? (item) : (item)) +#define dec_zone_state(zone, item) \ + dec_zone_state_check(zone, ((item) == (enum zone_stat_item )0) ? (item) : (item)) void quiet_vmstat(void); void cpu_vm_stats_fold(int cpu); @@ -253,65 +294,65 @@ void set_pgdat_percpu_threshold(pg_data_t *pgdat, * We do not maintain differentials in a single processor configuration. * The functions directly modify the zone and global counters. */ -static inline void __mod_zone_page_state(struct zone *zone, +static inline void __mod_zone_page_state_check(struct zone *zone, enum zone_stat_item item, long delta) { - zone_page_state_add(delta, zone, item); + zone_page_state_add_check(delta, zone, item); } -static inline void __mod_node_page_state(struct pglist_data *pgdat, +static inline void __mod_node_page_state_check(struct pglist_data *pgdat, enum node_stat_item item, int delta) { - node_page_state_add(delta, pgdat, item); + node_page_state_add_check(delta, pgdat, item); } -static inline void __inc_zone_state(struct zone *zone, enum zone_stat_item item) +static inline void __inc_zone_state_check(struct zone *zone, enum zone_stat_item item) { atomic_long_inc(&zone->vm_stat[item]); atomic_long_inc(&vm_zone_stat[item]); } -static inline void __inc_node_state(struct pglist_data *pgdat, enum node_stat_item item) +static inline void __inc_node_state_check(struct pglist_data *pgdat, enum node_stat_item item) { atomic_long_inc(&pgdat->vm_stat[item]); atomic_long_inc(&vm_node_stat[item]); } -static inline void __dec_zone_state(struct zone *zone, enum zone_stat_item item) +static inline void __dec_zone_state_check(struct zone *zone, enum zone_stat_item item) { atomic_long_dec(&zone->vm_stat[item]); atomic_long_dec(&vm_zone_stat[item]); } -static inline void __dec_node_state(struct pglist_data *pgdat, enum node_stat_item item) +static inline void __dec_node_state_check(struct pglist_data *pgdat, enum node_stat_item item) { atomic_long_dec(&pgdat->vm_stat[item]); atomic_long_dec(&vm_node_stat[item]); } -static inline void __inc_zone_page_state(struct page *page, +static inline void __inc_zone_page_state_check(struct page *page, enum zone_stat_item item) { - __inc_zone_state(page_zone(page), item); + __inc_zone_state_check(page_zone(page), item); } -static inline void __inc_node_page_state(struct page *page, +static inline void __inc_node_page_state_check(struct page *page, enum node_stat_item item) { - __inc_node_state(page_pgdat(page), item); + __inc_node_state_check(page_pgdat(page), item); } -static inline void __dec_zone_page_state(struct page *page, +static inline void __dec_zone_page_state_check(struct page *page, enum zone_stat_item item) { - __dec_zone_state(page_zone(page), item); + __dec_zone_state_check(page_zone(page), item); } -static inline void __dec_node_page_state(struct page *page, +static inline void __dec_node_page_state_check(struct page *page, enum node_stat_item item) { - __dec_node_state(page_pgdat(page), item); + __dec_node_state_check(page_pgdat(page), item); } @@ -341,6 +382,27 @@ static inline void drain_zonestat(struct zone *zone, struct per_cpu_pageset *pset) { } #endif /* CONFIG_SMP */ +#define __mod_zone_page_state(zone, item, delta) \ + __mod_zone_page_state_check(zone, ((item) == (enum zone_stat_item )0) ? (item) : (item), delta) +#define __mod_node_page_state(pgdat, item, delta) \ + __mod_node_page_state_check(pgdat, ((item) == (enum node_stat_item )0) ? (item) : (item), delta) +#define __inc_zone_state(zone, item) \ + __inc_zone_state_check(zone, ((item) == (enum zone_stat_item )0) ? (item) : (item)) +#define __inc_node_state(pgdat, item) \ + __inc_node_state_check(pgdat, ((item) == (enum node_stat_item )0) ? (item) : (item)) +#define __dec_zone_state(zone, item) \ + __dec_zone_state_check(zone, ((item) == (enum zone_stat_item )0) ? (item) : (item)) +#define __dec_node_state(pgdat, item) \ + __dec_node_state_check(pgdat, ((item) == (enum node_stat_item )0) ? (item) : (item)) +#define __inc_zone_page_state(page, item) \ + __inc_zone_page_state_check(page, ((item) == (enum zone_stat_item )0) ? (item) : (item)) +#define __inc_node_page_state(page, item) \ + __inc_node_page_state_check(page, ((item) == (enum node_stat_item )0) ? (item) : (item)) +#define __dec_zone_page_state(page, item) \ + __dec_zone_page_state_check(page, ((item) == (enum zone_stat_item )0) ? (item) : (item)) +#define __dec_node_page_state(page, item) \ + __dec_node_page_state_check(page, ((item) == (enum node_stat_item )0) ? (item) : (item)) + static inline void __mod_zone_freepage_state(struct zone *zone, int nr_pages, int migratetype) { diff --git a/mm/vmstat.c b/mm/vmstat.c index 78c682ade326..c309a701d953 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -224,7 +224,7 @@ void set_pgdat_percpu_threshold(pg_data_t *pgdat, * or when we know that preemption is disabled and that * particular counter cannot be updated from interrupt context. */ -void __mod_zone_page_state(struct zone *zone, enum zone_stat_item item, +void __mod_zone_page_state_check(struct zone *zone, enum zone_stat_item item, long delta) { struct per_cpu_pageset __percpu *pcp = zone->pageset; @@ -242,9 +242,9 @@ void __mod_zone_page_state(struct zone *zone, enum zone_stat_item item, } __this_cpu_write(*p, x); } -EXPORT_SYMBOL(__mod_zone_page_state); +EXPORT_SYMBOL(__mod_zone_page_state_check); -void __mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item, +void __mod_node_page_state_check(struct pglist_data *pgdat, enum node_stat_item item, long delta) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; @@ -262,7 +262,7 @@ void __mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item, } __this_cpu_write(*p, x); } -EXPORT_SYMBOL(__mod_node_page_state); +EXPORT_SYMBOL(__mod_node_page_state_check); /* * Optimized increment and decrement functions. @@ -287,7 +287,7 @@ EXPORT_SYMBOL(__mod_node_page_state); * in between and therefore the atomicity vs. interrupt cannot be exploited * in a useful way here. */ -void __inc_zone_state(struct zone *zone, enum zone_stat_item item) +void __inc_zone_state_check(struct zone *zone, enum zone_stat_item item) { struct per_cpu_pageset __percpu *pcp = zone->pageset; s8 __percpu *p = pcp->vm_stat_diff + item; @@ -303,7 +303,7 @@ void __inc_zone_state(struct zone *zone, enum zone_stat_item item) } } -void __inc_node_state(struct pglist_data *pgdat, enum node_stat_item item) +void __inc_node_state_check(struct pglist_data *pgdat, enum node_stat_item item) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; s8 __percpu *p = pcp->vm_node_stat_diff + item; @@ -314,24 +314,24 @@ void __inc_node_state(struct pglist_data *pgdat, enum node_stat_item item) if (unlikely(v > t)) { s8 overstep = t >> 1; - node_page_state_add(v + overstep, pgdat, item); + node_page_state_add_check(v + overstep, pgdat, item); __this_cpu_write(*p, -overstep); } } -void __inc_zone_page_state(struct page *page, enum zone_stat_item item) +void __inc_zone_page_state_check(struct page *page, enum zone_stat_item item) { - __inc_zone_state(page_zone(page), item); + __inc_zone_state_check(page_zone(page), item); } -EXPORT_SYMBOL(__inc_zone_page_state); +EXPORT_SYMBOL(__inc_zone_page_state_check); -void __inc_node_page_state(struct page *page, enum node_stat_item item) +void __inc_node_page_state_check(struct page *page, enum node_stat_item item) { - __inc_node_state(page_pgdat(page), item); + __inc_node_state_check(page_pgdat(page), item); } -EXPORT_SYMBOL(__inc_node_page_state); +EXPORT_SYMBOL(__inc_node_page_state_check); -void __dec_zone_state(struct zone *zone, enum zone_stat_item item) +void __dec_zone_state_check(struct zone *zone, enum zone_stat_item item) { struct per_cpu_pageset __percpu *pcp = zone->pageset; s8 __percpu *p = pcp->vm_stat_diff + item; @@ -342,12 +342,12 @@ void __dec_zone_state(struct zone *zone, enum zone_stat_item item) if (unlikely(v < - t)) { s8 overstep = t >> 1; - zone_page_state_add(v - overstep, zone, item); + zone_page_state_add_check(v - overstep, zone, item); __this_cpu_write(*p, overstep); } } -void __dec_node_state(struct pglist_data *pgdat, enum node_stat_item item) +void __dec_node_state_check(struct pglist_data *pgdat, enum node_stat_item item) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; s8 __percpu *p = pcp->vm_node_stat_diff + item; @@ -358,22 +358,22 @@ void __dec_node_state(struct pglist_data *pgdat, enum node_stat_item item) if (unlikely(v < - t)) { s8 overstep = t >> 1; - node_page_state_add(v - overstep, pgdat, item); + node_page_state_add_check(v - overstep, pgdat, item); __this_cpu_write(*p, overstep); } } -void __dec_zone_page_state(struct page *page, enum zone_stat_item item) +void __dec_zone_page_state_check(struct page *page, enum zone_stat_item item) { - __dec_zone_state(page_zone(page), item); + __dec_zone_state_check(page_zone(page), item); } -EXPORT_SYMBOL(__dec_zone_page_state); +EXPORT_SYMBOL(__dec_zone_page_state_check); -void __dec_node_page_state(struct page *page, enum node_stat_item item) +void __dec_node_page_state_check(struct page *page, enum node_stat_item item) { - __dec_node_state(page_pgdat(page), item); + __dec_node_state_check(page_pgdat(page), item); } -EXPORT_SYMBOL(__dec_node_page_state); +EXPORT_SYMBOL(__dec_node_page_state_check); #ifdef CONFIG_HAVE_CMPXCHG_LOCAL /* @@ -426,26 +426,26 @@ static inline void mod_zone_state(struct zone *zone, zone_page_state_add(z, zone, item); } -void mod_zone_page_state(struct zone *zone, enum zone_stat_item item, +void mod_zone_page_state_check(struct zone *zone, enum zone_stat_item item, long delta) { - mod_zone_state(zone, item, delta, 0); + mod_zone_state_check(zone, item, delta, 0); } -EXPORT_SYMBOL(mod_zone_page_state); +EXPORT_SYMBOL(mod_zone_page_state_check); -void inc_zone_page_state(struct page *page, enum zone_stat_item item) +void inc_zone_page_state_check(struct page *page, enum zone_stat_item item) { - mod_zone_state(page_zone(page), item, 1, 1); + mod_zone_state_check(page_zone(page), item, 1, 1); } -EXPORT_SYMBOL(inc_zone_page_state); +EXPORT_SYMBOL(inc_zone_page_state_check); -void dec_zone_page_state(struct page *page, enum zone_stat_item item) +void dec_zone_page_state_check(struct page *page, enum zone_stat_item item) { - mod_zone_state(page_zone(page), item, -1, -1); + mod_zone_state_check(page_zone(page), item, -1, -1); } -EXPORT_SYMBOL(dec_zone_page_state); +EXPORT_SYMBOL(dec_zone_page_state_check); -static inline void mod_node_state(struct pglist_data *pgdat, +static inline void mod_node_state_check(struct pglist_data *pgdat, enum node_stat_item item, int delta, int overstep_mode) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; @@ -480,111 +480,111 @@ static inline void mod_node_state(struct pglist_data *pgdat, } while (this_cpu_cmpxchg(*p, o, n) != o); if (z) - node_page_state_add(z, pgdat, item); + node_page_state_add_check(z, pgdat, item); } -void mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item, +void mod_node_page_state_check(struct pglist_data *pgdat, enum node_stat_item item, long delta) { - mod_node_state(pgdat, item, delta, 0); + mod_node_state_check(pgdat, item, delta, 0); } -EXPORT_SYMBOL(mod_node_page_state); +EXPORT_SYMBOL(mod_node_page_state_check); -void inc_node_state(struct pglist_data *pgdat, enum node_stat_item item) +void inc_node_state_check(struct pglist_data *pgdat, enum node_stat_item item) { - mod_node_state(pgdat, item, 1, 1); + mod_node_state_check(pgdat, item, 1, 1); } -void inc_node_page_state(struct page *page, enum node_stat_item item) +void inc_node_page_state_check(struct page *page, enum node_stat_item item) { - mod_node_state(page_pgdat(page), item, 1, 1); + mod_node_state_check(page_pgdat(page), item, 1, 1); } -EXPORT_SYMBOL(inc_node_page_state); +EXPORT_SYMBOL(inc_node_page_state_check); -void dec_node_page_state(struct page *page, enum node_stat_item item) +void dec_node_page_state_check(struct page *page, enum node_stat_item item) { - mod_node_state(page_pgdat(page), item, -1, -1); + mod_node_state_check(page_pgdat(page), item, -1, -1); } -EXPORT_SYMBOL(dec_node_page_state); +EXPORT_SYMBOL(dec_node_page_state_check); #else /* * Use interrupt disable to serialize counter updates */ -void mod_zone_page_state(struct zone *zone, enum zone_stat_item item, +void mod_zone_page_state_check(struct zone *zone, enum zone_stat_item item, long delta) { unsigned long flags; local_irq_save(flags); - __mod_zone_page_state(zone, item, delta); + __mod_zone_page_state_check(zone, item, delta); local_irq_restore(flags); } -EXPORT_SYMBOL(mod_zone_page_state); +EXPORT_SYMBOL(mod_zone_page_state_check); -void inc_zone_page_state(struct page *page, enum zone_stat_item item) +void inc_zone_page_state_check(struct page *page, enum zone_stat_item item) { unsigned long flags; struct zone *zone; zone = page_zone(page); local_irq_save(flags); - __inc_zone_state(zone, item); + __inc_zone_state_check(zone, item); local_irq_restore(flags); } -EXPORT_SYMBOL(inc_zone_page_state); +EXPORT_SYMBOL(inc_zone_page_state_check); -void dec_zone_page_state(struct page *page, enum zone_stat_item item) +void dec_zone_page_state_check(struct page *page, enum zone_stat_item item) { unsigned long flags; local_irq_save(flags); - __dec_zone_page_state(page, item); + __dec_zone_page_state_check(page, item); local_irq_restore(flags); } -EXPORT_SYMBOL(dec_zone_page_state); +EXPORT_SYMBOL(dec_zone_page_state_check); -void inc_node_state(struct pglist_data *pgdat, enum node_stat_item item) +void inc_node_state_check(struct pglist_data *pgdat, enum node_stat_item item) { unsigned long flags; local_irq_save(flags); - __inc_node_state(pgdat, item); + __inc_node_state_check(pgdat, item); local_irq_restore(flags); } -EXPORT_SYMBOL(inc_node_state); +EXPORT_SYMBOL(inc_node_state_check); -void mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item, +void mod_node_page_state_check(struct pglist_data *pgdat, enum node_stat_item item, long delta) { unsigned long flags; local_irq_save(flags); - __mod_node_page_state(pgdat, item, delta); + __mod_node_page_state_check(pgdat, item, delta); local_irq_restore(flags); } -EXPORT_SYMBOL(mod_node_page_state); +EXPORT_SYMBOL(mod_node_page_state_check); -void inc_node_page_state(struct page *page, enum node_stat_item item) +void inc_node_page_state_check(struct page *page, enum node_stat_item item) { unsigned long flags; struct pglist_data *pgdat; pgdat = page_pgdat(page); local_irq_save(flags); - __inc_node_state(pgdat, item); + __inc_node_state_check(pgdat, item); local_irq_restore(flags); } -EXPORT_SYMBOL(inc_node_page_state); +EXPORT_SYMBOL(inc_node_page_state_check); -void dec_node_page_state(struct page *page, enum node_stat_item item) +void dec_node_page_state_check(struct page *page, enum node_stat_item item) { unsigned long flags; local_irq_save(flags); - __dec_node_page_state(page, item); + __dec_node_page_state_check(page, item); local_irq_restore(flags); } -EXPORT_SYMBOL(dec_node_page_state); +EXPORT_SYMBOL(dec_node_page_state_check); #endif /* @@ -775,7 +775,7 @@ void drain_zonestat(struct zone *zone, struct per_cpu_pageset *pset) * is called frequently in a NUMA machine, so try to be as * frugal as possible. */ -unsigned long sum_zone_node_page_state(int node, +unsigned long sum_zone_node_page_state_check(int node, enum zone_stat_item item) { struct zone *zones = NODE_DATA(node)->node_zones; @@ -791,7 +791,7 @@ unsigned long sum_zone_node_page_state(int node, /* * Determine the per node value of a stat item. */ -unsigned long node_page_state(struct pglist_data *pgdat, +unsigned long node_page_state_check(struct pglist_data *pgdat, enum node_stat_item item) { long x = atomic_long_read(&pgdat->vm_stat[item]);