From patchwork Fri Jul 22 12:41:52 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Maydell X-Patchwork-Id: 72622 Delivered-To: patches@linaro.org Received: by 10.140.29.52 with SMTP id a49csp976697qga; Fri, 22 Jul 2016 05:41:55 -0700 (PDT) X-Received: by 10.28.166.140 with SMTP id p134mr24023772wme.21.1469191315047; Fri, 22 Jul 2016 05:41:55 -0700 (PDT) Return-Path: Received: from orth.archaic.org.uk (orth.archaic.org.uk. [2001:8b0:1d0::2]) by mx.google.com with ESMTPS id tm3si698667wjc.108.2016.07.22.05.41.54 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 22 Jul 2016 05:41:54 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of pm215@archaic.org.uk designates 2001:8b0:1d0::2 as permitted sender) client-ip=2001:8b0:1d0::2; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of pm215@archaic.org.uk designates 2001:8b0:1d0::2 as permitted sender) smtp.mailfrom=pm215@archaic.org.uk; dmarc=fail (p=NONE dis=NONE) header.from=linaro.org Received: from pm215 by orth.archaic.org.uk with local (Exim 4.84_2) (envelope-from ) id 1bQZm5-0004VB-0F; Fri, 22 Jul 2016 13:41:53 +0100 From: Peter Maydell To: qemu-devel@nongnu.org Cc: patches@linaro.org, Riku Voipio Subject: [PATCH] linux-user: Handle brk() attempts with very large sizes Date: Fri, 22 Jul 2016 13:41:52 +0100 Message-Id: <1469191312-8330-1-git-send-email-peter.maydell@linaro.org> X-Mailer: git-send-email 1.9.1 In do_brk(), we were inadvertently truncating the size of a requested brk() from the guest by putting it into an 'int' variable. This meant that we would incorrectly report success back to the guest rather than a failed allocation, typically resulting in the guest then segfaulting. Use abi_ulong instead. This fixes a crash in the '31370.cc' test in the gcc libstdc++ test suite (the test case starts by trying to allocate a very large size and reduces the size until the allocation succeeds). Signed-off-by: Peter Maydell --- linux-user/syscall.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- 1.9.1 diff --git a/linux-user/syscall.c b/linux-user/syscall.c index b4dc721..0e98593 100644 --- a/linux-user/syscall.c +++ b/linux-user/syscall.c @@ -830,7 +830,7 @@ void target_set_brk(abi_ulong new_brk) abi_long do_brk(abi_ulong new_brk) { abi_long mapped_addr; - int new_alloc_size; + abi_ulong new_alloc_size; DEBUGF_BRK("do_brk(" TARGET_ABI_FMT_lx ") -> ", new_brk);