tests/heapshrink: disable malloc per-thread cache for accuracy

Message ID 20180828112628.9462-1-rafael.tinoco@linaro.org
State New
Headers show
Series
  • tests/heapshrink: disable malloc per-thread cache for accuracy
Related show

Commit Message

Rafael David Tinoco Aug. 28, 2018, 11:26 a.m.
After upstream commit: (glibc-2.25.90-688-gd5c3fafc43) glibc has a new
per-thread caching mechanism that will NOT allow this test to
successfully measure if heap has shrunk, or not, due to the fact that
heap won't have its sized reduced right away, as the tests expects.

Only way to test heap shrinkness in an accurate way is to disable this
per-thread cache. In order to disable it you need to have an environment
tunable GLIBC in place:

GLIBC_TUNABLES=glibc.malloc.tcache_count=0

Unfortunately it requires to be set before program is loaded, since it
is evaluated during malloc() initialization, and there is no way to
re-initialize malloc() from the program context (not even using
constructor functions).

This commit sets GLIBC_TUNABLES environment variable for the heapshrink
tests in order to make the test to succeed in environments with newer
GLIBC. Older environments won't suffer any consequences since the
environment tunable will just be ignored.

Link: https://bugs.linaro.org/show_bug.cgi?id=3950
Signed-off-by: Rafael David Tinoco <rafael.tinoco@linaro.org>

---
 tests/heapshrink.c | 28 +++++++++++++++++++++++++---
 tests/run_tests.py | 41 ++++++++++++++++++++++++++++++++++-------
 2 files changed, 59 insertions(+), 10 deletions(-)

-- 
2.18.0

-- 
You received this message because you are subscribed to the Google Groups "libhugetlbfs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to libhugetlbfs+unsubscribe@googlegroups.com.
To post to this group, send email to libhugetlbfs@googlegroups.com.
Visit this group at https://groups.google.com/group/libhugetlbfs.
For more options, visit https://groups.google.com/d/optout.

Comments

Rafael David Tinoco Sept. 16, 2018, 7:10 p.m. | #1
On Tue, Aug 28, 2018 at 8:26 AM Rafael David Tinoco
<rafael.tinoco@linaro.org> wrote:
>

> After upstream commit: (glibc-2.25.90-688-gd5c3fafc43) glibc has a new

> per-thread caching mechanism that will NOT allow this test to

> successfully measure if heap has shrunk, or not, due to the fact that

> heap won't have its sized reduced right away, as the tests expects.

>

> Only way to test heap shrinkness in an accurate way is to disable this

> per-thread cache. In order to disable it you need to have an environment

> tunable GLIBC in place:

>

> GLIBC_TUNABLES=glibc.malloc.tcache_count=0

>

> Unfortunately it requires to be set before program is loaded, since it

> is evaluated during malloc() initialization, and there is no way to

> re-initialize malloc() from the program context (not even using

> constructor functions).

>

> This commit sets GLIBC_TUNABLES environment variable for the heapshrink

> tests in order to make the test to succeed in environments with newer

> GLIBC. Older environments won't suffer any consequences since the

> environment tunable will just be ignored.

>

> Link: https://bugs.linaro.org/show_bug.cgi?id=3950

> Signed-off-by: Rafael David Tinoco <rafael.tinoco@linaro.org>

> ---

>  tests/heapshrink.c | 28 +++++++++++++++++++++++++---

>  tests/run_tests.py | 41 ++++++++++++++++++++++++++++++++++-------

>  2 files changed, 59 insertions(+), 10 deletions(-)


Anyone able to review/push this fix for me ?

Is this list the correct place for such suggestion ?

Thanks,

-Rafael

-- 
You received this message because you are subscribed to the Google Groups "libhugetlbfs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to libhugetlbfs+unsubscribe@googlegroups.com.
To post to this group, send email to libhugetlbfs@googlegroups.com.
Visit this group at https://groups.google.com/group/libhugetlbfs.
For more options, visit https://groups.google.com/d/optout.
Eric B Munson Sept. 18, 2018, 3:15 a.m. | #2
On Tue, 28 Aug 2018, Rafael David Tinoco wrote:

> After upstream commit: (glibc-2.25.90-688-gd5c3fafc43) glibc has a new

> per-thread caching mechanism that will NOT allow this test to

> successfully measure if heap has shrunk, or not, due to the fact that

> heap won't have its sized reduced right away, as the tests expects.

> 

> Only way to test heap shrinkness in an accurate way is to disable this

> per-thread cache. In order to disable it you need to have an environment

> tunable GLIBC in place:

> 

> GLIBC_TUNABLES=glibc.malloc.tcache_count=0

> 

> Unfortunately it requires to be set before program is loaded, since it

> is evaluated during malloc() initialization, and there is no way to

> re-initialize malloc() from the program context (not even using

> constructor functions).

> 

> This commit sets GLIBC_TUNABLES environment variable for the heapshrink

> tests in order to make the test to succeed in environments with newer

> GLIBC. Older environments won't suffer any consequences since the

> environment tunable will just be ignored.

> 

> Link: https://bugs.linaro.org/show_bug.cgi?id=3950

> Signed-off-by: Rafael David Tinoco <rafael.tinoco@linaro.org>


Applied, thanks
Eric

-- 
You received this message because you are subscribed to the Google Groups "libhugetlbfs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to libhugetlbfs+unsubscribe@googlegroups.com.
To post to this group, send email to libhugetlbfs@googlegroups.com.
Visit this group at https://groups.google.com/group/libhugetlbfs.
For more options, visit https://groups.google.com/d/optout.
Eric B Munson Sept. 18, 2018, 3:16 a.m. | #3
Sorry for taking so long to apply this.  I just finished an
international move and my workspace is now finally setup.

Eric

-- 
You received this message because you are subscribed to the Google Groups "libhugetlbfs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to libhugetlbfs+unsubscribe@googlegroups.com.
To post to this group, send email to libhugetlbfs@googlegroups.com.
Visit this group at https://groups.google.com/group/libhugetlbfs.
For more options, visit https://groups.google.com/d/optout.
Rafael David Tinoco Sept. 19, 2018, 2:10 a.m. | #4
No problem Eric! I was just confirming, for any future needs, if any.
Tks a lot! All the best in your new place!
On Tue, Sep 18, 2018 at 12:16 AM Eric B Munson <emunson@mgebm.net> wrote:
>

> Sorry for taking so long to apply this.  I just finished an

> international move and my workspace is now finally setup.

>

> Eric

>

> --

> You received this message because you are subscribed to the Google Groups "libhugetlbfs" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to libhugetlbfs+unsubscribe@googlegroups.com.

> To post to this group, send email to libhugetlbfs@googlegroups.com.

> Visit this group at https://groups.google.com/group/libhugetlbfs.

> For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups "libhugetlbfs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to libhugetlbfs+unsubscribe@googlegroups.com.
To post to this group, send email to libhugetlbfs@googlegroups.com.
Visit this group at https://groups.google.com/group/libhugetlbfs.
For more options, visit https://groups.google.com/d/optout.

Patch

diff --git a/tests/heapshrink.c b/tests/heapshrink.c
index 16f233b..d2934aa 100644
--- a/tests/heapshrink.c
+++ b/tests/heapshrink.c
@@ -33,7 +33,7 @@ 
 
 int main(int argc, char **argv)
 {
-	int is_huge, have_env, shrink_ok, have_helper;
+	int is_huge, have_env, shrink_ok, have_helper, tcache_enabled;
 	unsigned long long mapping_size;
 	void *p;
 	long size = MAX(32*1024*1024, kernel_default_hugepage_size());
@@ -45,6 +45,23 @@  int main(int argc, char **argv)
 	p = getenv("LD_PRELOAD");
 	have_helper = p != NULL && strstr(p, "heapshrink") != NULL;
 
+	/*
+	 * After upstream commit: (glibc-2.25.90-688-gd5c3fafc43) glibc has a
+	 * new per-thread caching mechanism that will NOT allow this test to
+	 * successfully measure if heap has shrunk or not due to the fact that
+	 * heap won't have its sized reduced right away.
+	 *
+	 * In order to disable it you need to have the tunable GLIBC in place.
+	 * Unfortunately, it requires to be set before program is loaded, as an
+	 * environment variable, since we can't re-initialize malloc() from the
+	 * program context (not even with a constructor function), and the
+	 * tunable is only evaluated during malloc() initialization.
+	 *
+	 * GLIBC_TUNABLES=glibc.malloc.tcache_count=0
+	 */
+	p = getenv("GLIBC_TUNABLES");
+	tcache_enabled = p != NULL && strstr(p, "malloc.tcache_count=0");
+
 	p = malloc(size);
 	if (!p) {
 		if (shrink_ok && have_helper) {
@@ -68,7 +85,12 @@  int main(int argc, char **argv)
 
 	free(p);
 	mapping_size = get_mapping_page_size(p+size-1);
-	if (shrink_ok && mapping_size > MIN_PAGE_SIZE)
-		FAIL("Heap did not shrink");
+	if (shrink_ok && mapping_size > MIN_PAGE_SIZE) {
+		if (tcache_enabled)
+			FAIL("Heap did not shrink");
+		else
+			FAIL("Heap didn't shrink. Check malloc.tcache_count=0");
+	}
+
 	PASS();
 }
diff --git a/tests/run_tests.py b/tests/run_tests.py
index 617ed93..3c95a03 100755
--- a/tests/run_tests.py
+++ b/tests/run_tests.py
@@ -572,15 +572,42 @@  def functional_tests():
     do_test("malloc_manysmall")
     do_test("malloc_manysmall", LD_PRELOAD="libhugetlbfs.so",
             HUGETLB_MORECORE="yes")
-    do_test("heapshrink")
-    do_test("heapshrink", LD_PRELOAD="libheapshrink.so")
-    do_test("heapshrink", LD_PRELOAD="libhugetlbfs.so", HUGETLB_MORECORE="yes")
-    do_test("heapshrink", LD_PRELOAD="libhugetlbfs.so libheapshrink.so",
+
+    # After upstream commit: (glibc-2.25.90-688-gd5c3fafc43) glibc has a
+    # new per-thread caching mechanism that will NOT allow heapshrink test to
+    # successfully measure if heap has shrunk or not due to the fact that
+    # heap won't have its sized reduced right away.
+    #
+    # In order to disable it, you need to have the tunable GLIBC in place.
+    # Unfortunately, it requires to be set before program is loaded, as an
+    # environment variable, since we can't re-initialize malloc() from the
+    # program context (not even with a constructor function), and the tunable
+    # is only evaluated during malloc() initialization.
+
+    do_test("heapshrink",
+            GLIBC_TUNABLES="glibc.malloc.tcache_count=0")
+    do_test("heapshrink",
+            GLIBC_TUNABLES="glibc.malloc.tcache_count=0",
+            LD_PRELOAD="libheapshrink.so")
+    do_test("heapshrink",
+            GLIBC_TUNABLES="glibc.malloc.tcache_count=0",
+            LD_PRELOAD="libhugetlbfs.so",
+            HUGETLB_MORECORE="yes")
+    do_test("heapshrink",
+            GLIBC_TUNABLES="glibc.malloc.tcache_count=0",
+            LD_PRELOAD="libhugetlbfs.so libheapshrink.so",
             HUGETLB_MORECORE="yes")
-    do_test("heapshrink", LD_PRELOAD="libheapshrink.so", HUGETLB_MORECORE="yes",
+    do_test("heapshrink",
+            GLIBC_TUNABLES="glibc.malloc.tcache_count=0",
+            LD_PRELOAD="libheapshrink.so",
+            HUGETLB_MORECORE="yes",
             HUGETLB_MORECORE_SHRINK="yes")
-    do_test("heapshrink", LD_PRELOAD="libhugetlbfs.so libheapshrink.so",
-            HUGETLB_MORECORE="yes", HUGETLB_MORECORE_SHRINK="yes")
+    do_test("heapshrink",
+            GLIBC_TUNABLES="glibc.malloc.tcache_count=0",
+            LD_PRELOAD="libhugetlbfs.so libheapshrink.so",
+            HUGETLB_MORECORE="yes",
+            HUGETLB_MORECORE_SHRINK="yes")
+
     do_test("heap-overflow", HUGETLB_VERBOSE="1", HUGETLB_MORECORE="yes")
 
     # Run the remapping tests' up-front checks