From patchwork Fri Feb 22 19:27:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adhemerval Zanella Netto X-Patchwork-Id: 159071 Delivered-To: patch@linaro.org Received: by 2002:a02:48:0:0:0:0:0 with SMTP id 69csp2103936jaa; Fri, 22 Feb 2019 11:27:25 -0800 (PST) X-Google-Smtp-Source: AHgI3IYLHpt07Def5bmuAS2ht4yGuIjVF53rmtuD9NRWqYkGp6PbOq1GOIOLzwu/zLP3wOd7INou X-Received: by 2002:a62:f598:: with SMTP id b24mr5762264pfm.72.1550863645526; Fri, 22 Feb 2019 11:27:25 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1550863645; cv=none; d=google.com; s=arc-20160816; b=HSoNDnjMHibt9dN2CUnwoL65yjllvNS9uNqpuMdwzIUfa/2dg94xdJG+fn8s+wdJ4K 09PuXduq6W4KU3ACOYUXixr5OAk2pHAgKCfhUJqaS7G1eL0njPd3KEBUdlRNoHt4OaA+ E1Rw4Np4JzgLwJmXCTVdB80ro+pDTmL8DQhe4moWT67VDIZztmm3pEjQCml2RywfGL3w 6qTc9ehdod//wSf9SfyRfecl39dDWsiqAVgpsmilhSYRZUJfRO791z0qP2R/fB50Aq1X CmkjZEOShRvfI4IqNTazRn68zjiIHVw6iXIXFZ8p4kIAGH4WWSU3yshEv9b0te1fptqf 8X+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:date:subject:to:from:dkim-signature:delivered-to:sender :list-help:list-post:list-archive:list-subscribe:list-unsubscribe :list-id:precedence:mailing-list:dkim-signature:domainkey-signature; bh=40d/JWbdqKdeyYWjlS/Eogcgyc2VnVJZHqWdOEb0P2Y=; b=QyQdCkBb1jKz0MoE7pTn8VrdMMSeRq4WN9sXXZnO+WU3oHYhWP6CCr8ziAUKhbGOiR W2il7uwoss+MqAlUtq8GYrCb48/zwTFs1s+/M9voIjlZmxrLH6QPwzWvCsYQuL5gLCjw rJgkwzBmYY0y03QWaaDosrvx+vaW3pKyqGUSkFaSBD0LmIJkbCAoAL+GNz2TGdu4wOnO x2HTOUf6RLAdeQC36PU2Cq8tcw7qimMCED5c1sTGVPUY/IzY61juBB3jG8avzVYizp0g 1tbgkJZc2HIcRVfpMpgniia7QGDnk8oLz5ScqPS/VS7NR8ZwDHddkVEuOv/CopFmBMlf 8GoA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@sourceware.org header.s=default header.b=Yhx3OCbD; dkim=pass header.i=@linaro.org header.s=google header.b=AANdl6vX; spf=pass (google.com: domain of libc-alpha-return-100226-patch=linaro.org@sourceware.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="libc-alpha-return-100226-patch=linaro.org@sourceware.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id k14si2099226pgl.40.2019.02.22.11.27.25 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 22 Feb 2019 11:27:25 -0800 (PST) Received-SPF: pass (google.com: domain of libc-alpha-return-100226-patch=linaro.org@sourceware.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@sourceware.org header.s=default header.b=Yhx3OCbD; dkim=pass header.i=@linaro.org header.s=google header.b=AANdl6vX; spf=pass (google.com: domain of libc-alpha-return-100226-patch=linaro.org@sourceware.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="libc-alpha-return-100226-patch=linaro.org@sourceware.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:from:to:subject:date:message-id; q=dns; s= default; b=mV52oQSisHxJfadSQ6pHlDTb2sMcV3RwZXmvROe2GwaR+2DPVRgva RarkB8VyWEqS9P6kNDnMA2G0k4x7VIJNVXLcII7pLWzrwepm+W+PPd0CikcZ1zLE wdZktchIwIiTDlB1tdhveoTbqWqymkZvRP0QJqJUyAp5lh5OHhmIM4= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:from:to:subject:date:message-id; s=default; bh=OGPKnudch+coQdro7MOO+6GfLeo=; b=Yhx3OCbDWlHIouR7wpQXlsnPjvn0 Li0t+TfCtXCKeARl9L+yCm83DHQypkv/JhT+hZQm5A43Gr3FXfJDpP7y5hk7rkIU p4MkHG8f7Fmn/iopFU4Zz57zRtIL15pWol89wZcVlTgfcWqFyyiBOYozn9Lptj1F wHCjXkeVjcZnwDA= Received: (qmail 104230 invoked by alias); 22 Feb 2019 19:27:14 -0000 Mailing-List: contact libc-alpha-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: libc-alpha-owner@sourceware.org Delivered-To: mailing list libc-alpha@sourceware.org Received: (qmail 104059 invoked by uid 89); 22 Feb 2019 19:27:14 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-26.9 required=5.0 tests=BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy=inclusion, nonexported, cancellation, Single X-HELO: mail-qt1-f196.google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id; bh=40d/JWbdqKdeyYWjlS/Eogcgyc2VnVJZHqWdOEb0P2Y=; b=AANdl6vXUdrVM5k+1eZEgyUMP6aqaTVUZWZr/E55fSRbjtoohuY/drKDmEkKX8lZTB WAnAkOD2GRqPeKGTs3A7NuWNnUbTxjZGzk6CVBxeypUz6zHp7n9FOyfokalAgBi8zmHX tMlazJ6fKWtfG4bFHprqYw2JnzgawlHL/ZyRNRPexKqvgNRWtu5cYAgd8L9iaQDqyUVT 6GUNPlqCnj1NkJgeqYPuW/eoNoO8NydslXJwpZufPaK5CwU6l0kyi9k/QEm7yAQ7KL2Q KFnPaa1ZKo8dlhR8LgNyOw9SWtpvy3KWdUaRLeEUUJ1yeq9r+LJgdEP7pW79hOr6NHxA xT+w== Return-Path: From: Adhemerval Zanella To: libc-alpha@sourceware.org Subject: [PATCH 1/4] Add single-thread.h header Date: Fri, 22 Feb 2019 16:27:00 -0300 Message-Id: <20190222192703.18177-1-adhemerval.zanella@linaro.org> This patch move the single-thread syscall optimization defintions from syscall-cancel.h to new header file single-thread.h and also move the cancellation definitions from pthreadP.h to syscall-cancel.h. The idea is just simplify the inclusion of both syscall-cancel.h and single-thread.h (without the requirement of including all pthreadP.h defintions). No semantic changes expected, checked on a build for all major ABIs. * nptl/pthreadP.h (CANCEL_ASYNC, CANCEL_RESET, LIBC_CANCEL_ASYNC, LIBC_CANCEL_RESET, __libc_enable_asynccancel, __libc_disable_asynccancel, __librt_enable_asynccancel, __libc_disable_asynccancel, __librt_enable_asynccancel, __librt_disable_asynccancel): Move to ... * sysdeps/unix/sysv/linux/sysdep-cancel.h: ... here. (SINGLE_THREAD_P, RTLD_SINGLE_THREAD_P): Move to ... * sysdeps/unix/sysv/linux/single-thread.h: ... here. * sysdeps/generic/single-thread.h: New file. * sysdeps/unix/sysdep.h: Include single-thread.h. * sysdeps/unix/sysv/linux/futex-internal.h: Include sysdep-cancel.h. * sysdeps/unix/sysv/linux/lowlevellock-futex.h: Likewise. --- nptl/pthreadP.h | 37 --------- sysdeps/generic/single-thread.h | 24 ++++++ sysdeps/unix/sysdep.h | 2 +- sysdeps/unix/sysv/linux/futex-internal.h | 2 +- sysdeps/unix/sysv/linux/lowlevellock-futex.h | 2 +- sysdeps/unix/sysv/linux/single-thread.h | 64 ++++++++++++++++ sysdeps/unix/sysv/linux/sysdep-cancel.h | 81 ++++++++++---------- 7 files changed, 133 insertions(+), 79 deletions(-) create mode 100644 sysdeps/generic/single-thread.h create mode 100644 sysdeps/unix/sysv/linux/single-thread.h -- 2.17.1 diff --git a/nptl/pthreadP.h b/nptl/pthreadP.h index 626bd4b096..0beacd1266 100644 --- a/nptl/pthreadP.h +++ b/nptl/pthreadP.h @@ -311,34 +311,6 @@ __do_cancel (void) } -/* Set cancellation mode to asynchronous. */ -#define CANCEL_ASYNC() \ - __pthread_enable_asynccancel () -/* Reset to previous cancellation mode. */ -#define CANCEL_RESET(oldtype) \ - __pthread_disable_asynccancel (oldtype) - -#if IS_IN (libc) -/* Same as CANCEL_ASYNC, but for use in libc.so. */ -# define LIBC_CANCEL_ASYNC() \ - __libc_enable_asynccancel () -/* Same as CANCEL_RESET, but for use in libc.so. */ -# define LIBC_CANCEL_RESET(oldtype) \ - __libc_disable_asynccancel (oldtype) -#elif IS_IN (libpthread) -# define LIBC_CANCEL_ASYNC() CANCEL_ASYNC () -# define LIBC_CANCEL_RESET(val) CANCEL_RESET (val) -#elif IS_IN (librt) -# define LIBC_CANCEL_ASYNC() \ - __librt_enable_asynccancel () -# define LIBC_CANCEL_RESET(val) \ - __librt_disable_asynccancel (val) -#else -# define LIBC_CANCEL_ASYNC() 0 /* Just a dummy value. */ -# define LIBC_CANCEL_RESET(val) ((void)(val)) /* Nothing, but evaluate it. */ -#endif - - /* Internal prototypes. */ /* Thread list handling. */ @@ -545,15 +517,6 @@ extern int __pthread_cond_wait_2_0 (pthread_cond_2_0_t *cond, extern int __pthread_getaffinity_np (pthread_t th, size_t cpusetsize, cpu_set_t *cpuset); -/* The two functions are in libc.so and not exported. */ -extern int __libc_enable_asynccancel (void) attribute_hidden; -extern void __libc_disable_asynccancel (int oldtype) attribute_hidden; - - -/* The two functions are in librt.so and not exported. */ -extern int __librt_enable_asynccancel (void) attribute_hidden; -extern void __librt_disable_asynccancel (int oldtype) attribute_hidden; - #if IS_IN (libpthread) /* Special versions which use non-exported functions. */ extern void __pthread_cleanup_push (struct _pthread_cleanup_buffer *buffer, diff --git a/sysdeps/generic/single-thread.h b/sysdeps/generic/single-thread.h new file mode 100644 index 0000000000..0a1520ec49 --- /dev/null +++ b/sysdeps/generic/single-thread.h @@ -0,0 +1,24 @@ +/* Single thread optimization, generic version. + Copyright (C) 2019 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#ifndef _SINGLE_THREAD_H +#define _SINGLE_THREAD_H + +#define SINGLE_THREAD_P (0) + +#endif /* _SINGLE_THREAD_H */ diff --git a/sysdeps/unix/sysdep.h b/sysdeps/unix/sysdep.h index 7cb6c89825..6e503d7688 100644 --- a/sysdeps/unix/sysdep.h +++ b/sysdeps/unix/sysdep.h @@ -16,7 +16,7 @@ . */ #include - +#include #include #define HAVE_SYSCALLS diff --git a/sysdeps/unix/sysv/linux/futex-internal.h b/sysdeps/unix/sysv/linux/futex-internal.h index 55f0fab77c..501f993853 100644 --- a/sysdeps/unix/sysv/linux/futex-internal.h +++ b/sysdeps/unix/sysv/linux/futex-internal.h @@ -22,7 +22,7 @@ #include #include #include -#include +#include /* See sysdeps/nptl/futex-internal.h for documentation; this file only contains Linux-specific comments. diff --git a/sysdeps/unix/sysv/linux/lowlevellock-futex.h b/sysdeps/unix/sysv/linux/lowlevellock-futex.h index 6f060b1739..030a14b8dc 100644 --- a/sysdeps/unix/sysv/linux/lowlevellock-futex.h +++ b/sysdeps/unix/sysv/linux/lowlevellock-futex.h @@ -21,7 +21,7 @@ #ifndef __ASSEMBLER__ #include -#include +#include #include #endif diff --git a/sysdeps/unix/sysv/linux/single-thread.h b/sysdeps/unix/sysv/linux/single-thread.h new file mode 100644 index 0000000000..8248c5d8b8 --- /dev/null +++ b/sysdeps/unix/sysv/linux/single-thread.h @@ -0,0 +1,64 @@ +/* Single thread optimization, Linux version. + Copyright (C) 2019 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#ifndef _SINGLE_THREAD_H +#define _SINGLE_THREAD_H + +/* The default way to check if the process is single thread is by using the + pthread_t 'multiple_threads' field. However for some architectures it + is faster to either use an extra field on TCB or global varibles + (the TCB field is also used on x86 for some single-thread atomic + optimizations). + + The ABI might define SINGLE_THREAD_BY_GLOBAL to enable the single + thread check to use global variables instead of the pthread_t + field. */ + +#ifdef SINGLE_THREAD_BY_GLOBAL +# if IS_IN (libc) +extern int __libc_multiple_threads; +# define SINGLE_THREAD_P \ + __glibc_likely (__libc_multiple_threads == 0) +# elif IS_IN (libpthread) +extern int __pthread_multiple_threads; +# define SINGLE_THREAD_P \ + __glibc_likely (__pthread_multiple_threads == 0) +# elif IS_IN (librt) +# define SINGLE_THREAD_P \ + __glibc_likely (THREAD_GETMEM (THREAD_SELF, \ + header.multiple_threads) == 0) +# else +/* For rtld, et cetera. */ +# define SINGLE_THREAD_P (1) +# endif +#else /* SINGLE_THREAD_BY_GLOBAL */ +# if IS_IN (libc) || IS_IN (libpthread) || IS_IN (librt) +# define SINGLE_THREAD_P \ + __glibc_likely (THREAD_GETMEM (THREAD_SELF, \ + header.multiple_threads) == 0) +# else +/* For rtld, et cetera. */ +# define SINGLE_THREAD_P (1) +# endif +#endif /* SINGLE_THREAD_BY_GLOBAL */ + +#define RTLD_SINGLE_THREAD_P \ + __glibc_likely (THREAD_GETMEM (THREAD_SELF, \ + header.multiple_threads) == 0) + +#endif /* _SINGLE_THREAD_H */ diff --git a/sysdeps/unix/sysv/linux/sysdep-cancel.h b/sysdeps/unix/sysv/linux/sysdep-cancel.h index 896549c7f7..a831b30106 100644 --- a/sysdeps/unix/sysv/linux/sysdep-cancel.h +++ b/sysdeps/unix/sysv/linux/sysdep-cancel.h @@ -17,48 +17,51 @@ License along with the GNU C Library; if not, see . */ +#ifndef _SYSDEP_CANCEL_H +#define _SYSDEP_CANCEL_H + #include #include -#include +#include + +/* The two functions are in libc.so and not exported. */ +extern int __libc_enable_asynccancel (void) attribute_hidden; +extern void __libc_disable_asynccancel (int oldtype) attribute_hidden; + +/* The two functions are in librt.so and not exported. */ +extern int __librt_enable_asynccancel (void) attribute_hidden; +extern void __librt_disable_asynccancel (int oldtype) attribute_hidden; + +/* The two functions are in libpthread.so and not exported. */ +extern int __pthread_enable_asynccancel (void) attribute_hidden; +extern void __pthread_disable_asynccancel (int oldtype) attribute_hidden; -/* The default way to check if the process is single thread is by using the - pthread_t 'multiple_threads' field. However for some architectures it - is faster to either use an extra field on TCB or global varibles - (the TCB field is also used on x86 for some single-thread atomic - optimizations). +/* Set cancellation mode to asynchronous. */ +#define CANCEL_ASYNC() \ + __pthread_enable_asynccancel () +/* Reset to previous cancellation mode. */ +#define CANCEL_RESET(oldtype) \ + __pthread_disable_asynccancel (oldtype) - The ABI might define SINGLE_THREAD_BY_GLOBAL to enable the single - thread check to use global variables instead of the pthread_t - field. */ +#if IS_IN (libc) +/* Same as CANCEL_ASYNC, but for use in libc.so. */ +# define LIBC_CANCEL_ASYNC() \ + __libc_enable_asynccancel () +/* Same as CANCEL_RESET, but for use in libc.so. */ +# define LIBC_CANCEL_RESET(oldtype) \ + __libc_disable_asynccancel (oldtype) +#elif IS_IN (libpthread) +# define LIBC_CANCEL_ASYNC() CANCEL_ASYNC () +# define LIBC_CANCEL_RESET(val) CANCEL_RESET (val) +#elif IS_IN (librt) +# define LIBC_CANCEL_ASYNC() \ + __librt_enable_asynccancel () +# define LIBC_CANCEL_RESET(val) \ + __librt_disable_asynccancel (val) +#else +# define LIBC_CANCEL_ASYNC() 0 /* Just a dummy value. */ +# define LIBC_CANCEL_RESET(val) ((void)(val)) /* Nothing, but evaluate it. */ +#endif -#ifdef SINGLE_THREAD_BY_GLOBAL -# if IS_IN (libc) -extern int __libc_multiple_threads; -# define SINGLE_THREAD_P \ - __glibc_likely (__libc_multiple_threads == 0) -# elif IS_IN (libpthread) -extern int __pthread_multiple_threads; -# define SINGLE_THREAD_P \ - __glibc_likely (__pthread_multiple_threads == 0) -# elif IS_IN (librt) -# define SINGLE_THREAD_P \ - __glibc_likely (THREAD_GETMEM (THREAD_SELF, \ - header.multiple_threads) == 0) -# else -/* For rtld, et cetera. */ -# define SINGLE_THREAD_P (1) -# endif -#else /* SINGLE_THREAD_BY_GLOBAL */ -# if IS_IN (libc) || IS_IN (libpthread) || IS_IN (librt) -# define SINGLE_THREAD_P \ - __glibc_likely (THREAD_GETMEM (THREAD_SELF, \ - header.multiple_threads) == 0) -# else -/* For rtld, et cetera. */ -# define SINGLE_THREAD_P (1) -# endif -#endif /* SINGLE_THREAD_BY_GLOBAL */ -#define RTLD_SINGLE_THREAD_P \ - __glibc_likely (THREAD_GETMEM (THREAD_SELF, \ - header.multiple_threads) == 0) +#endif From patchwork Fri Feb 22 19:27:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adhemerval Zanella Netto X-Patchwork-Id: 159072 Delivered-To: patch@linaro.org Received: by 2002:a02:48:0:0:0:0:0 with SMTP id 69csp2104078jaa; Fri, 22 Feb 2019 11:27:34 -0800 (PST) X-Google-Smtp-Source: AHgI3IZcqN8tTO1Yft2qYkByhvPVZWlAFv5YAuJWd6jsiHO6ecQQX8epZ58jyXeMs2RwkJuUfdje X-Received: by 2002:a17:902:b20c:: with SMTP id t12mr2661301plr.340.1550863654226; Fri, 22 Feb 2019 11:27:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1550863654; cv=none; d=google.com; s=arc-20160816; b=jxB8ZStLul6zmC013m2zYEgqa/6KddNwC/FhU52m0XIILvKguiigYxvBCQO54UMf/5 qVxuGSFJ3VmlK5s60kePQWPqZZO7m7Ob9Jv/LQbenKA3sTbHCip/amchiWO1csP9gGgG PTWjC6B4OQQY9S7JL0oxFg6hNy5zBp4fmDRt5FY9SYPr/lAuAD+eBtnIL+FHV/AzFLey uysVFTQECq6vB4Sa+JuQIrxIfv/UM69o81OIcKcxXd0YnNlYPnF2jyQHm7fl/mNvaTtb ZKAjguil3gug/I2MHpAGNsehfHdETy/4OYIJ2AZCkl1Aqw2fxw2VJqnT/fE078DBStd3 1fKg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:to:from :dkim-signature:delivered-to:sender:list-help:list-post:list-archive :list-subscribe:list-unsubscribe:list-id:precedence:mailing-list :dkim-signature:domainkey-signature; bh=S7BlCjhdv4DtbR4F1QTfiCNpajUpygrWUXDByD8zu/4=; b=sBhpguWDfqFwpfB+DoKKEwTeSWTrzWrErqtOp5Exx5rQ5O4UJKnyBSinV3SmTsRj0f jYZatG+GxxZEKd6BK9COYmpOwk5rbhM/iyE5eT/AuS6NUeS9ICE2zm2cVisfrXD9VzM6 BROo0lygJwNii5ih7HBT1ndzGWjDdplbScfR3VhtPfX8qN5S92QRSs0ygsDFaG9rV7HR 4NOMBVTFqyAnbFJV7YDDeJBHkmWpEgxuHbM56o8cm85XurytkQ1vEY0m8CZ+ed1y+N8T UdVcFU0ZEm+EsvAtasFjyuZ6bVxQEqeanp2k8kLG68qNY/MqPxA+3NAbBmS4YlgRAPCF bX4w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@sourceware.org header.s=default header.b=VqqeP4BW; dkim=pass header.i=@linaro.org header.s=google header.b=F2CPD0qs; spf=pass (google.com: domain of libc-alpha-return-100227-patch=linaro.org@sourceware.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="libc-alpha-return-100227-patch=linaro.org@sourceware.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id x24si1985313plr.304.2019.02.22.11.27.33 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 22 Feb 2019 11:27:34 -0800 (PST) Received-SPF: pass (google.com: domain of libc-alpha-return-100227-patch=linaro.org@sourceware.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@sourceware.org header.s=default header.b=VqqeP4BW; dkim=pass header.i=@linaro.org header.s=google header.b=F2CPD0qs; spf=pass (google.com: domain of libc-alpha-return-100227-patch=linaro.org@sourceware.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="libc-alpha-return-100227-patch=linaro.org@sourceware.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:from:to:subject:date:message-id:in-reply-to :references; q=dns; s=default; b=pyVxVQU7AD0rUY9gqDQrK1OSJ0VrqAS 3IXLtLkRjOVo4aLZopyPj6Ryk+riQv/aR1XX6o+aHM3xlWrsW9lx8P4x+ToNZpSi mZGN/24VgOQ4pbY7uZghhawFrRsrxefbMuM4chGgUWFy8+DDtAqvzsay66Yv6oqE 8YsLWr5qfAqM= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:from:to:subject:date:message-id:in-reply-to :references; s=default; bh=sUJd8m9Cqog9kYuR08OhF3OgsBU=; b=VqqeP 4BWUOvosdTQG5R+afLHvzIsMsmntBYrigOcdIsp8P33ZZRUlv1yu8OvjcnkxUeGw Y52iBwz/fKhe7UXZb4QmQBNbzue6je68jmZJrbYJInQ3bruudaZlg7gDOY9Qci88 4bExIz5+SRZRSLi6zuve2/EcKZTnYvMCK18Lfw= Received: (qmail 104382 invoked by alias); 22 Feb 2019 19:27:15 -0000 Mailing-List: contact libc-alpha-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: libc-alpha-owner@sourceware.org Delivered-To: mailing list libc-alpha@sourceware.org Received: (qmail 104254 invoked by uid 89); 22 Feb 2019 19:27:15 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-26.9 required=5.0 tests=BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy=0x48, Small, 0x58, 0x28 X-HELO: mail-qk1-f173.google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=S7BlCjhdv4DtbR4F1QTfiCNpajUpygrWUXDByD8zu/4=; b=F2CPD0qs5ejHEUKcuWQhL0xZGV/vdseOP01/fXiSJvxY7vLSNcW/AfqowewvsDDPmQ XjBhPTccU4k+vr6HcWfADYYf682uEVd3ZoKz8golKzSfRsN81AHf9zw8F5ad+uWUL3mj Xszh0ajgxNrdfVUw6PlPi4uObWVF6lr2slVOvXJGwCKG3mp1N9oXUXPuwpmQAVOvEycR TsA6cG00fVJq/+uSqYPW+asFcaZAGhGicTEvC7sEjAX6+iAq6qK2upOD/N4M8os15jrM 1Xya0ELseDmke6mYSmejy3RQMKTWoFMkXWx78bO1h47cV9/iG5fNTiwmxYt5Ps3XR+F/ lwGA== Return-Path: From: Adhemerval Zanella To: libc-alpha@sourceware.org Subject: [PATCH 2/4] Small optimization for lowlevellock Date: Fri, 22 Feb 2019 16:27:01 -0300 Message-Id: <20190222192703.18177-2-adhemerval.zanella@linaro.org> In-Reply-To: <20190222192703.18177-1-adhemerval.zanella@linaro.org> References: <20190222192703.18177-1-adhemerval.zanella@linaro.org> This patch optimizes both __lll_lock_wait_private and __lll_lock_wait by issuing only one lll_futex_wait. Since it is defined as an inlined syscall and inlined syscalls are defined using inlined assembly the compiler usually can not see both calls are equal and optimize accordingly. On aarch64 the resulting binary is change from: 0000000000000060 <__lll_lock_wait>: 60: 2a0103e5 mov w5, w1 64: b9400001 ldr w1, [x0] 68: aa0003e4 mov x4, x0 6c: 7100083f cmp w1, #0x2 70: 540000e1 b.ne 8c <__lll_lock_wait+0x2c> // b.any 74: 521900a1 eor w1, w5, #0x80 78: d2800042 mov x2, #0x2 // #2 7c: 93407c21 sxtw x1, w1 80: d2800003 mov x3, #0x0 // #0 84: d2800c48 mov x8, #0x62 // #98 88: d4000001 svc #0x0 8c: 521900a5 eor w5, w5, #0x80 90: 52800046 mov w6, #0x2 // #2 94: 93407ca5 sxtw x5, w5 98: 14000008 b b8 <__lll_lock_wait+0x58> 9c: d503201f nop a0: aa0403e0 mov x0, x4 a4: aa0503e1 mov x1, x5 a8: d2800042 mov x2, #0x2 // #2 ac: d2800003 mov x3, #0x0 // #0 b0: d2800c48 mov x8, #0x62 // #98 b4: d4000001 svc #0x0 b8: 885ffc80 ldaxr w0, [x4] bc: 88017c86 stxr w1, w6, [x4] c0: 35ffffc1 cbnz w1, b8 <__lll_lock_wait+0x58> c4: 35fffee0 cbnz w0, a0 <__lll_lock_wait+0x40> c8: d65f03c0 ret To: 0000000000000048 <__lll_lock_wait>: 48: aa0003e4 mov x4, x0 4c: 2a0103e5 mov w5, w1 50: b9400000 ldr w0, [x0] 54: 7100081f cmp w0, #0x2 58: 540000c0 b.eq 70 <__lll_lock_wait+0x28> // b.none 5c: 52800041 mov w1, #0x2 // #2 60: 885ffc80 ldaxr w0, [x4] 64: 88027c81 stxr w2, w1, [x4] 68: 35ffffc2 cbnz w2, 60 <__lll_lock_wait+0x18> 6c: 34000120 cbz w0, 90 <__lll_lock_wait+0x48> 70: 521900a1 eor w1, w5, #0x80 74: aa0403e0 mov x0, x4 78: 93407c21 sxtw x1, w1 7c: d2800042 mov x2, #0x2 // #2 80: d2800003 mov x3, #0x0 // #0 84: d2800c48 mov x8, #0x62 // #98 88: d4000001 svc #0x0 8c: 17fffff4 b 5c <__lll_lock_wait+0x14> 90: d65f03c0 ret I see similar changes on powerpc and other architectures. It also aligns with x86_64 implementation by adding the systemtap probes. Checker on aarch64-linux-gnu. * nptl/lowlevellock.c (__lll_lock_wait, __lll_lock_wait_private): Optimize futex call and add systemtap probe. --- nptl/lowlevellock.c | 31 +++++++++++++++++++------------ 1 file changed, 19 insertions(+), 12 deletions(-) -- 2.17.1 Reviewed-by: Carlos O'Donell diff --git a/nptl/lowlevellock.c b/nptl/lowlevellock.c index 5eaa3807ea..47548ff121 100644 --- a/nptl/lowlevellock.c +++ b/nptl/lowlevellock.c @@ -17,20 +17,23 @@ License along with the GNU C Library; if not, see . */ -#include #include #include -#include #include +#include void __lll_lock_wait_private (int *futex) { - if (*futex == 2) - lll_futex_wait (futex, 2, LLL_PRIVATE); /* Wait if *futex == 2. */ - - while (atomic_exchange_acq (futex, 2) != 0) - lll_futex_wait (futex, 2, LLL_PRIVATE); /* Wait if *futex == 2. */ + if (atomic_load_relaxed (futex) == 2) + goto futex; + + while (atomic_exchange_acquire (futex, 2) != 0) + { + futex: + LIBC_PROBE (lll_lock_wait_private, 1, futex); + lll_futex_wait (futex, 2, LLL_PRIVATE); /* Wait if *futex == 2. */ + } } @@ -39,10 +42,14 @@ __lll_lock_wait_private (int *futex) void __lll_lock_wait (int *futex, int private) { - if (*futex == 2) - lll_futex_wait (futex, 2, private); /* Wait if *futex == 2. */ - - while (atomic_exchange_acq (futex, 2) != 0) - lll_futex_wait (futex, 2, private); /* Wait if *futex == 2. */ + if (atomic_load_relaxed (futex) == 2) + goto futex; + + while (atomic_exchange_acquire (futex, 2) != 0) + { + futex: + LIBC_PROBE (lll_lock_wait, 1, futex); + lll_futex_wait (futex, 2, private); /* Wait if *futex == 2. */ + } } #endif From patchwork Fri Feb 22 19:27:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adhemerval Zanella Netto X-Patchwork-Id: 159073 Delivered-To: patch@linaro.org Received: by 2002:a02:48:0:0:0:0:0 with SMTP id 69csp2104227jaa; Fri, 22 Feb 2019 11:27:43 -0800 (PST) X-Google-Smtp-Source: AHgI3IYiKFHVsuB95CF8iL0m819m/f+Tfx8CfRI/24JSztii9ljUI7/aBwttSVqDlyFgYnYrMriR X-Received: by 2002:a62:e719:: with SMTP id s25mr5771182pfh.12.1550863663137; Fri, 22 Feb 2019 11:27:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1550863663; cv=none; d=google.com; s=arc-20160816; b=miuriommW/rBa0FWidmaavdGsJZlvYhXeQiSbfDR4/g9O5RT03CSAzgrpePT0kZ+EF GVNeOUz+M00UwTNyGlQa62eM/WXFwbX43KLcev3vJKpHymrP4VCRMFPbT3ePQviDIDyE 7L+8FNCBXGcx1rKkhHO9gZx67RuWfnM5q+WRhL+XxgHhxXLcZ0GMHSGnfeLsQatgr951 niKT9VSnqmL8sYuP/edf0g/0pr4BiaUooaqUIrQd9LLk9h05wz4NrArV0rypTKubTE6Q +osTLIcuA+upQWZ7dHL/QpdTxki0TXroQCf+Tc5R9JNlZ2vVwELJreRN9hCMHN5X+vTT ugrg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:to:from :dkim-signature:delivered-to:sender:list-help:list-post:list-archive :list-subscribe:list-unsubscribe:list-id:precedence:mailing-list :dkim-signature:domainkey-signature; bh=uQcausP/p13fIUWxfrAMuOVxhKUEpTEwsjNKt4lsswk=; b=h8i5kC0xJM3bQg7jcruO1HU9H/hIdrhrLNw/Qyo1IdtJHASAP4XQBH9ZcFNHXoPvr/ HUhEOVkCmM1VYRp1LjgTfF0cH7QQK9RKEW2XuI0fBKNEtpgxo2M5eWpGGLIHKySXXb3R Ms2H/R/pEihg0teFHneWmwDKYSIBRL4m3SxvUOprK9TFV9MVFzfHLijKS8pSUbaPViGQ jLzsUd9mGuWzUBK6LE+PhCxjq5IqylD938kegFs2u8h15n/Ak72vZh+gzagvJfQilZBs 0zlX80Ok8ZmoU9moJzjv7eilxVP9CPuNGGfsWmI37mIaFGhYreoEu5Eyf2wQp8kQfYw2 Hx9g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@sourceware.org header.s=default header.b=gns8ruEb; dkim=pass header.i=@linaro.org header.s=google header.b=qO6DfiZ8; spf=pass (google.com: domain of libc-alpha-return-100228-patch=linaro.org@sourceware.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="libc-alpha-return-100228-patch=linaro.org@sourceware.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id 71si1970389pga.16.2019.02.22.11.27.42 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 22 Feb 2019 11:27:43 -0800 (PST) Received-SPF: pass (google.com: domain of libc-alpha-return-100228-patch=linaro.org@sourceware.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@sourceware.org header.s=default header.b=gns8ruEb; dkim=pass header.i=@linaro.org header.s=google header.b=qO6DfiZ8; spf=pass (google.com: domain of libc-alpha-return-100228-patch=linaro.org@sourceware.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="libc-alpha-return-100228-patch=linaro.org@sourceware.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:from:to:subject:date:message-id:in-reply-to :references; q=dns; s=default; b=lnmyhGmAp1zqLzvbVekY8ufOaxeQYm1 8PnOeXifQ4jBBgnLf2uZTxn7t4d0BryHUNzF+a9H1Cy4BiN3ahi9MS8CAE+eYYs+ gMTfIkWwIxpjTJC0fbdNkaCCa3W1uV6Ty+phMpGtdUcN6PBjby3OwleHhnyoBMWw zbBFezttx4n4= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:from:to:subject:date:message-id:in-reply-to :references; s=default; bh=LKfBI1VQgOFIOjbZhF2gzKZ/CFg=; b=gns8r uEba0MfVzkr/NykNHTafO8AmPA4vYU/fzZvQOG1xVSqI4c8akmBjdPALNRqpJQup ucxUBbWHNDv9zxvlUg/8p1AnA5aAb83eDwWg153P7v4dHXDMwFEzNngaJtm76GDd QgN1TMkYjchP/m6Fr4fEgaiFjGLg/M+QwfTsb4= Received: (qmail 105029 invoked by alias); 22 Feb 2019 19:27:20 -0000 Mailing-List: contact libc-alpha-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: libc-alpha-owner@sourceware.org Delivered-To: mailing list libc-alpha@sourceware.org Received: (qmail 104926 invoked by uid 89); 22 Feb 2019 19:27:19 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-26.4 required=5.0 tests=BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, KAM_NUMSUBJECT, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy=3118 X-HELO: mail-qk1-f174.google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=uQcausP/p13fIUWxfrAMuOVxhKUEpTEwsjNKt4lsswk=; b=qO6DfiZ8FwFuufg+7CRJNCRC5VWxDot+pM63SnG7W6nyMc6Sd2uRfhs8umnGmWEabu xHD0TJD3tCYSD1WNackQl/c9zq+PgfGS0Eh4os0GPFSRSjp5lpqwbxly15E6oZ0Dk1cF SeuVqHaJrKeg9gQr6MjrjrrbnWE1uyPKFUrEHIxSz7jNyI69RasQT6ir8nM+mtC+jBCb RMyG19PRTr7N0xyN2JmdrdwMJmQnZMqHLIQU+wqFoc022Ns6qKSs0bsyAXh9sAsGvs51 vHmOaxZ7glfz/VmuXSa2YKiHHn+F+OILMwudP+iA1GrebvvzLocElNPKSST806zXlbHl +aow== Return-Path: From: Adhemerval Zanella To: libc-alpha@sourceware.org Subject: [PATCH 3/4] Assume LLL_LOCK_INITIALIZER is 0 Date: Fri, 22 Feb 2019 16:27:02 -0300 Message-Id: <20190222192703.18177-3-adhemerval.zanella@linaro.org> In-Reply-To: <20190222192703.18177-1-adhemerval.zanella@linaro.org> References: <20190222192703.18177-1-adhemerval.zanella@linaro.org> Since hppa is not an outlier anymore regarding LLL_LOCK_INITIALIZER value, we can now assume it 0 for all architectures. Checked on a build for all major ABIs. * nptl/old_pthread_cond_broadcast.c (__pthread_cond_broadcast_2_0): Assume LLL_LOCK_INITIALIZER being 0. * nptl/old_pthread_cond_signal.c (__pthread_cond_signal_2_0): Likewise. * nptl/old_pthread_cond_timedwait.c (__pthread_cond_timedwait_2_0): Likewise. * nptl/old_pthread_cond_wait.c (__pthread_cond_wait_2_0): Likewise. * sysdeps/nptl/libc-lockP.h (__libc_lock_define_initialized): Likewise. --- nptl/old_pthread_cond_broadcast.c | 9 --------- nptl/old_pthread_cond_signal.c | 9 --------- nptl/old_pthread_cond_timedwait.c | 9 --------- nptl/old_pthread_cond_wait.c | 9 --------- sysdeps/nptl/libc-lockP.h | 17 +++-------------- 5 files changed, 3 insertions(+), 50 deletions(-) -- 2.17.1 diff --git a/nptl/old_pthread_cond_broadcast.c b/nptl/old_pthread_cond_broadcast.c index f561eb941f..04f285a1b1 100644 --- a/nptl/old_pthread_cond_broadcast.c +++ b/nptl/old_pthread_cond_broadcast.c @@ -31,18 +31,9 @@ __pthread_cond_broadcast_2_0 (pthread_cond_2_0_t *cond) { pthread_cond_t *newcond; -#if LLL_LOCK_INITIALIZER == 0 newcond = (pthread_cond_t *) calloc (sizeof (pthread_cond_t), 1); if (newcond == NULL) return ENOMEM; -#else - newcond = (pthread_cond_t *) malloc (sizeof (pthread_cond_t)); - if (newcond == NULL) - return ENOMEM; - - /* Initialize the condvar. */ - (void) pthread_cond_init (newcond, NULL); -#endif if (atomic_compare_and_exchange_bool_acq (&cond->cond, newcond, NULL)) /* Somebody else just initialized the condvar. */ diff --git a/nptl/old_pthread_cond_signal.c b/nptl/old_pthread_cond_signal.c index a167c8addc..9b67cb8a54 100644 --- a/nptl/old_pthread_cond_signal.c +++ b/nptl/old_pthread_cond_signal.c @@ -31,18 +31,9 @@ __pthread_cond_signal_2_0 (pthread_cond_2_0_t *cond) { pthread_cond_t *newcond; -#if LLL_LOCK_INITIALIZER == 0 newcond = (pthread_cond_t *) calloc (sizeof (pthread_cond_t), 1); if (newcond == NULL) return ENOMEM; -#else - newcond = (pthread_cond_t *) malloc (sizeof (pthread_cond_t)); - if (newcond == NULL) - return ENOMEM; - - /* Initialize the condvar. */ - (void) pthread_cond_init (newcond, NULL); -#endif if (atomic_compare_and_exchange_bool_acq (&cond->cond, newcond, NULL)) /* Somebody else just initialized the condvar. */ diff --git a/nptl/old_pthread_cond_timedwait.c b/nptl/old_pthread_cond_timedwait.c index f920320b13..a1fde85825 100644 --- a/nptl/old_pthread_cond_timedwait.c +++ b/nptl/old_pthread_cond_timedwait.c @@ -32,18 +32,9 @@ __pthread_cond_timedwait_2_0 (pthread_cond_2_0_t *cond, pthread_mutex_t *mutex, { pthread_cond_t *newcond; -#if LLL_LOCK_INITIALIZER == 0 newcond = (pthread_cond_t *) calloc (sizeof (pthread_cond_t), 1); if (newcond == NULL) return ENOMEM; -#else - newcond = (pthread_cond_t *) malloc (sizeof (pthread_cond_t)); - if (newcond == NULL) - return ENOMEM; - - /* Initialize the condvar. */ - (void) pthread_cond_init (newcond, NULL); -#endif if (atomic_compare_and_exchange_bool_acq (&cond->cond, newcond, NULL)) /* Somebody else just initialized the condvar. */ diff --git a/nptl/old_pthread_cond_wait.c b/nptl/old_pthread_cond_wait.c index 2be41b36d6..bb65340a17 100644 --- a/nptl/old_pthread_cond_wait.c +++ b/nptl/old_pthread_cond_wait.c @@ -31,18 +31,9 @@ __pthread_cond_wait_2_0 (pthread_cond_2_0_t *cond, pthread_mutex_t *mutex) { pthread_cond_t *newcond; -#if LLL_LOCK_INITIALIZER == 0 newcond = (pthread_cond_t *) calloc (sizeof (pthread_cond_t), 1); if (newcond == NULL) return ENOMEM; -#else - newcond = (pthread_cond_t *) malloc (sizeof (pthread_cond_t)); - if (newcond == NULL) - return ENOMEM; - - /* Initialize the condvar. */ - (void) pthread_cond_init (newcond, NULL); -#endif if (atomic_compare_and_exchange_bool_acq (&cond->cond, newcond, NULL)) /* Somebody else just initialized the condvar. */ diff --git a/sysdeps/nptl/libc-lockP.h b/sysdeps/nptl/libc-lockP.h index fc1bfe57e5..07d583f11a 100644 --- a/sysdeps/nptl/libc-lockP.h +++ b/sysdeps/nptl/libc-lockP.h @@ -71,23 +71,12 @@ typedef pthread_key_t __libc_key_t; For the C library we take a deeper look at the initializer. For this implementation all fields are initialized to zero. Therefore we don't initialize the variable which allows putting it into the - BSS section. (Except on PA-RISC and other odd architectures, where - initialized locks must be set to one due to the lack of normal - atomic operations.) */ + BSS section. */ +_Static_assert (LLL_LOCK_INITIALIZER == 0, "LLL_LOCK_INITIALIZER != 0"); #define _LIBC_LOCK_INITIALIZER LLL_LOCK_INITIALIZER -#if IS_IN (libc) || IS_IN (libpthread) -# if LLL_LOCK_INITIALIZER == 0 -# define __libc_lock_define_initialized(CLASS,NAME) \ - CLASS __libc_lock_t NAME; -# else -# define __libc_lock_define_initialized(CLASS,NAME) \ - CLASS __libc_lock_t NAME = LLL_LOCK_INITIALIZER; -# endif -#else -# define __libc_lock_define_initialized(CLASS,NAME) \ +#define __libc_lock_define_initialized(CLASS,NAME) \ CLASS __libc_lock_t NAME; -#endif #define __libc_rwlock_define_initialized(CLASS,NAME) \ CLASS __libc_rwlock_t NAME = PTHREAD_RWLOCK_INITIALIZER; From patchwork Fri Feb 22 19:27:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adhemerval Zanella Netto X-Patchwork-Id: 159074 Delivered-To: patch@linaro.org Received: by 2002:a02:48:0:0:0:0:0 with SMTP id 69csp2104415jaa; Fri, 22 Feb 2019 11:27:55 -0800 (PST) X-Google-Smtp-Source: AHgI3IZjB/WdpGT8RYvOwsCNzry/Of5UTMVPgOAL62Fe3DW74AsfDB3a4aP/b3s7LpTb7U6HZ35V X-Received: by 2002:a17:902:8f82:: with SMTP id z2mr5758316plo.163.1550863675006; Fri, 22 Feb 2019 11:27:55 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1550863675; cv=none; d=google.com; s=arc-20160816; b=KWBlOn/Vz+DbnCIOP0MOzzS72rG9z52rnI07u8Kthadq0A3d7DN4zA0eQdQYn3eUy/ v+kMq7wvAQX/IJRm0CyODARWgHZnu6gwzs7/tOyZtEaFhSa6pOyF3xsDAarmVZ3oRnQq qxChADZsfSUnozXZrbzI855+4zEaNIrBuhQNwIsryvvcC1fR7Q2RP0KtsjToFIxjI6Hc rg4mCK+kdBoZEZ3xHRPLHaYwOZikTr1QGpT5b9l6D9wKc19zJbLeltZJmDnm1SvigWwt Xy/b0+7sIkJnvHuSoPgWA+g/LKF7pYyU1uczEwivI4YMBBLZ/AzzkcicTXYJx42Vx/0c rhQw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:to:from :dkim-signature:delivered-to:sender:list-help:list-post:list-archive :list-subscribe:list-unsubscribe:list-id:precedence:mailing-list :dkim-signature:domainkey-signature; bh=9YRvtYJYXBM1mCGAJZkcDoWFNymFs9nOYAFC0HIotn8=; b=MepFmX8i5xbv/n9Mcs3f1efbn8hDHDnc3P9jbEulJtXYiZmE+unkjqiQuYoy6lZSUN 0gdZ6gNWVEWSo9j3bwtCEFDN8Jl73a61IMeD87znJ7ZuKyHNdZjkBWnotFWbXT57t4Ap ZayRoAvJVC8K8wiIIy9JgxAdZ98gMnUoG0icJYPSPXGjaOnec3QakuiCyuFabeW2YsIs g8IxfzP5MjoAoxI8QERoCWS2QuapIUJpqJ43G+VcmG6CtX4u/b7OFJ9XegEkjHoOd0RS njTUAgUnGe9JlUeVTx9qQK3Mbr96vB4n/5zXAeY+5/ZmxIXSQCjFFOMavxx25XaSyaD4 t6jw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@sourceware.org header.s=default header.b=fnr447Nr; dkim=pass header.i=@linaro.org header.s=google header.b=WP9V9WkQ; spf=pass (google.com: domain of libc-alpha-return-100229-patch=linaro.org@sourceware.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="libc-alpha-return-100229-patch=linaro.org@sourceware.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id b11si2082436pla.405.2019.02.22.11.27.54 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 22 Feb 2019 11:27:54 -0800 (PST) Received-SPF: pass (google.com: domain of libc-alpha-return-100229-patch=linaro.org@sourceware.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@sourceware.org header.s=default header.b=fnr447Nr; dkim=pass header.i=@linaro.org header.s=google header.b=WP9V9WkQ; spf=pass (google.com: domain of libc-alpha-return-100229-patch=linaro.org@sourceware.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="libc-alpha-return-100229-patch=linaro.org@sourceware.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:from:to:subject:date:message-id:in-reply-to :references; q=dns; s=default; b=s4/NxOXpSz6e/K/tHyTanpSc5h4Bd65 WF5lmgZOAb/5lzI7OVFw91mq9lt9ukvBAQgW4+zgqBeb4ehyBjFqPPkQJ0Wafy3q +by4bfotffQwo57nwdHQq9ILiX82T5O2Jqk0vMcyQj6A2XA84e+Qyx7WxEFJCpCo 1w78AvtL38ds= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:from:to:subject:date:message-id:in-reply-to :references; s=default; bh=v9s8yrw9vJJ5egyI8MqmY0cH2gY=; b=fnr44 7NrfBv83gK9QIXKnry+4IRXUHBwAyru7tk8Nfmz5rWbTSyCHqExssO/QSRSVcInR Itjas6ZiP7CkIXQFlH29QeMpccr0F3syHdJDJMkvnGps9psBVsqVT0bt/MAvQ47z QsP0lhSLG1fdW615skByIQIpDwL9f9WH2MZBxg= Received: (qmail 106248 invoked by alias); 22 Feb 2019 19:27:28 -0000 Mailing-List: contact libc-alpha-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: libc-alpha-owner@sourceware.org Delivered-To: mailing list libc-alpha@sourceware.org Received: (qmail 106140 invoked by uid 89); 22 Feb 2019 19:27:27 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-24.9 required=5.0 tests=BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, KAM_SHORT, LOTS_OF_MONEY, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy=indication, lock, nano, cmpl X-HELO: mail-qt1-f171.google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=9YRvtYJYXBM1mCGAJZkcDoWFNymFs9nOYAFC0HIotn8=; b=WP9V9WkQQwkDwmcDy9sv8ssWJlTo6jMoif9qwagMAoWPf9yjNPI2+j1Q4l4irQHtF5 E+X8ZeU9W19t6i74xLT2UhwpJPlvtLvx2b5hZAibQ6Os3CVz3fk1aMQoe6v5CGiOE+TR B8z6EjUg4f+l7yPCg1aaX4S+QzK5MhmHVqOIgVxSvwhoH0PmXcsN/f09eu1ixbrVOxI6 MEhbwU/BDnVneymRCk8M4EG0RjA0AXYgIPs+ORI5S9fo+Ju5Rbtaq8285cPTWLSEcP4X oGjc3D+vMZPXEjtPCsX87y32IPpRT/TcXXDOtcGhqU5gbZ3exlg9X2gfWBCzKnz5XHlB 4RnA== Return-Path: From: Adhemerval Zanella To: libc-alpha@sourceware.org Subject: [PATCH 4/4] x86: Remove arch-specific low level lock implementation Date: Fri, 22 Feb 2019 16:27:03 -0300 Message-Id: <20190222192703.18177-4-adhemerval.zanella@linaro.org> In-Reply-To: <20190222192703.18177-1-adhemerval.zanella@linaro.org> References: <20190222192703.18177-1-adhemerval.zanella@linaro.org> This patch removes the arch-specific x86 assembly implementation for low level locking and consolidate both 64 bits and 32 bits in a single implementation. Different than other architectures, x86 lll_trylock, lll_lock, and lll_unlock implements a single-thread optimization to avoid atomic operation, using cmpxchgl instead. This patch implements by using the new single-thread.h definitions in a generic way, although using the previous semantic. The lll_cond_trylock, lll_cond_lock, and lll_timedlock just use atomic operations plus calls to lll_lock_wait*. For __lll_lock_wait_private and __lll_lock_wait the generic implemtation there is no indication that assembly implementation is required performance-wise. Checked on x86_64-linux-gnu and i686-linux-gnu. * sysdeps/nptl/lowlevellock.h (__lll_trylock): New macro. (lll_trylock): Call __lll_trylock. * sysdeps/unix/sysv/linux/i386/libc-lowlevellock.S: Remove file. * sysdeps/unix/sysv/linux/i386/lll_timedlock_wait.c: Likewise. * sysdeps/unix/sysv/linux/i386/lowlevellock.S: Likewise. * sysdeps/unix/sysv/linux/i386/lowlevellock.h: Likewise. * sysdeps/unix/sysv/linux/x86_64/libc-lowlevellock.S: Likewise. * sysdeps/unix/sysv/linux/x86_64/lll_timedlock_wait.c: Likewise. * sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: Likewise. * sysdeps/unix/sysv/linux/x86_64/lowlevellock.h: Likewise. * sysdeps/unix/sysv/linux/x86/lowlevellock.h: New file. * sysdeps/unix/sysv/linux/x86_64/cancellation.S: Include lowlevellock-futex.h. --- sysdeps/nptl/lowlevellock.h | 4 +- .../unix/sysv/linux/i386/libc-lowlevellock.S | 19 - .../unix/sysv/linux/i386/lll_timedlock_wait.c | 1 - sysdeps/unix/sysv/linux/i386/lowlevellock.S | 368 ------------------ sysdeps/unix/sysv/linux/i386/lowlevellock.h | 240 ------------ sysdeps/unix/sysv/linux/x86/lowlevellock.h | 110 ++++++ sysdeps/unix/sysv/linux/x86_64/cancellation.S | 2 +- .../sysv/linux/x86_64/libc-lowlevellock.S | 19 - .../sysv/linux/x86_64/lll_timedlock_wait.c | 1 - sysdeps/unix/sysv/linux/x86_64/lowlevellock.S | 348 ----------------- sysdeps/unix/sysv/linux/x86_64/lowlevellock.h | 243 ------------ 11 files changed, 114 insertions(+), 1241 deletions(-) delete mode 100644 sysdeps/unix/sysv/linux/i386/libc-lowlevellock.S delete mode 100644 sysdeps/unix/sysv/linux/i386/lll_timedlock_wait.c delete mode 100644 sysdeps/unix/sysv/linux/i386/lowlevellock.S delete mode 100644 sysdeps/unix/sysv/linux/i386/lowlevellock.h create mode 100644 sysdeps/unix/sysv/linux/x86/lowlevellock.h delete mode 100644 sysdeps/unix/sysv/linux/x86_64/libc-lowlevellock.S delete mode 100644 sysdeps/unix/sysv/linux/x86_64/lll_timedlock_wait.c delete mode 100644 sysdeps/unix/sysv/linux/x86_64/lowlevellock.S delete mode 100644 sysdeps/unix/sysv/linux/x86_64/lowlevellock.h -- 2.17.1 diff --git a/sysdeps/nptl/lowlevellock.h b/sysdeps/nptl/lowlevellock.h index 6f017afdd5..e905829ee4 100644 --- a/sysdeps/nptl/lowlevellock.h +++ b/sysdeps/nptl/lowlevellock.h @@ -63,8 +63,10 @@ /* If LOCK is 0 (not acquired), set to 1 (acquired with no waiters) and return 0. Otherwise leave lock unchanged and return non-zero to indicate that the lock was not acquired. */ +#define __lll_trylock(lock) \ + __glibc_unlikely (atomic_compare_and_exchange_bool_acq ((lock), 1, 0)) #define lll_trylock(lock) \ - __glibc_unlikely (atomic_compare_and_exchange_bool_acq (&(lock), 1, 0)) + __lll_trylock (&(lock)) /* If LOCK is 0 (not acquired), set to 2 (acquired, possibly with waiters) and return 0. Otherwise leave lock unchanged and return non-zero to indicate diff --git a/sysdeps/unix/sysv/linux/i386/libc-lowlevellock.S b/sysdeps/unix/sysv/linux/i386/libc-lowlevellock.S deleted file mode 100644 index 32025277f3..0000000000 --- a/sysdeps/unix/sysv/linux/i386/libc-lowlevellock.S +++ /dev/null @@ -1,19 +0,0 @@ -/* Copyright (C) 2002-2019 Free Software Foundation, Inc. - This file is part of the GNU C Library. - Contributed by Ulrich Drepper , 2002. - - The GNU C Library is free software; you can redistribute it and/or - modify it under the terms of the GNU Lesser General Public - License as published by the Free Software Foundation; either - version 2.1 of the License, or (at your option) any later version. - - The GNU C Library is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - Lesser General Public License for more details. - - You should have received a copy of the GNU Lesser General Public - License along with the GNU C Library; if not, see - . */ - -#include "lowlevellock.S" diff --git a/sysdeps/unix/sysv/linux/i386/lll_timedlock_wait.c b/sysdeps/unix/sysv/linux/i386/lll_timedlock_wait.c deleted file mode 100644 index f6875b8f89..0000000000 --- a/sysdeps/unix/sysv/linux/i386/lll_timedlock_wait.c +++ /dev/null @@ -1 +0,0 @@ -/* __lll_timedlock_wait is in lowlevellock.S. */ diff --git a/sysdeps/unix/sysv/linux/i386/lowlevellock.S b/sysdeps/unix/sysv/linux/i386/lowlevellock.S deleted file mode 100644 index 83191a3899..0000000000 --- a/sysdeps/unix/sysv/linux/i386/lowlevellock.S +++ /dev/null @@ -1,368 +0,0 @@ -/* Copyright (C) 2002-2019 Free Software Foundation, Inc. - This file is part of the GNU C Library. - Contributed by Ulrich Drepper , 2002. - - The GNU C Library is free software; you can redistribute it and/or - modify it under the terms of the GNU Lesser General Public - License as published by the Free Software Foundation; either - version 2.1 of the License, or (at your option) any later version. - - The GNU C Library is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - Lesser General Public License for more details. - - You should have received a copy of the GNU Lesser General Public - License along with the GNU C Library; if not, see - . */ - -#include -#include -#include -#include - -#include - - .text - -#define LOAD_PRIVATE_FUTEX_WAIT(reg) \ - movl $(FUTEX_WAIT | FUTEX_PRIVATE_FLAG), reg -#define LOAD_PRIVATE_FUTEX_WAKE(reg) \ - movl $(FUTEX_WAKE | FUTEX_PRIVATE_FLAG), reg -#define LOAD_FUTEX_WAIT(reg) \ - xorl $(FUTEX_WAIT | FUTEX_PRIVATE_FLAG), reg -#define LOAD_FUTEX_WAIT_ABS(reg) \ - xorl $(FUTEX_WAIT_BITSET | FUTEX_PRIVATE_FLAG | FUTEX_CLOCK_REALTIME), reg -#define LOAD_FUTEX_WAKE(reg) \ - xorl $(FUTEX_WAKE | FUTEX_PRIVATE_FLAG), reg - - .globl __lll_lock_wait_private - .type __lll_lock_wait_private,@function - .hidden __lll_lock_wait_private - .align 16 -__lll_lock_wait_private: - cfi_startproc - pushl %edx - cfi_adjust_cfa_offset(4) - pushl %ebx - cfi_adjust_cfa_offset(4) - pushl %esi - cfi_adjust_cfa_offset(4) - cfi_offset(%edx, -8) - cfi_offset(%ebx, -12) - cfi_offset(%esi, -16) - - movl $2, %edx - movl %ecx, %ebx - xorl %esi, %esi /* No timeout. */ - LOAD_PRIVATE_FUTEX_WAIT (%ecx) - - cmpl %edx, %eax /* NB: %edx == 2 */ - jne 2f - -1: LIBC_PROBE (lll_lock_wait_private, 1, %ebx) - movl $SYS_futex, %eax - ENTER_KERNEL - -2: movl %edx, %eax - xchgl %eax, (%ebx) /* NB: lock is implied */ - - testl %eax, %eax - jnz 1b - - popl %esi - cfi_adjust_cfa_offset(-4) - cfi_restore(%esi) - popl %ebx - cfi_adjust_cfa_offset(-4) - cfi_restore(%ebx) - popl %edx - cfi_adjust_cfa_offset(-4) - cfi_restore(%edx) - ret - cfi_endproc - .size __lll_lock_wait_private,.-__lll_lock_wait_private - -#if !IS_IN (libc) - .globl __lll_lock_wait - .type __lll_lock_wait,@function - .hidden __lll_lock_wait - .align 16 -__lll_lock_wait: - cfi_startproc - pushl %edx - cfi_adjust_cfa_offset(4) - pushl %ebx - cfi_adjust_cfa_offset(4) - pushl %esi - cfi_adjust_cfa_offset(4) - cfi_offset(%edx, -8) - cfi_offset(%ebx, -12) - cfi_offset(%esi, -16) - - movl %edx, %ebx - movl $2, %edx - xorl %esi, %esi /* No timeout. */ - LOAD_FUTEX_WAIT (%ecx) - - cmpl %edx, %eax /* NB: %edx == 2 */ - jne 2f - -1: movl $SYS_futex, %eax - ENTER_KERNEL - -2: movl %edx, %eax - xchgl %eax, (%ebx) /* NB: lock is implied */ - - testl %eax, %eax - jnz 1b - - popl %esi - cfi_adjust_cfa_offset(-4) - cfi_restore(%esi) - popl %ebx - cfi_adjust_cfa_offset(-4) - cfi_restore(%ebx) - popl %edx - cfi_adjust_cfa_offset(-4) - cfi_restore(%edx) - ret - cfi_endproc - .size __lll_lock_wait,.-__lll_lock_wait - - /* %ecx: futex - %esi: flags - %edx: timeout - %eax: futex value - */ - .globl __lll_timedlock_wait - .type __lll_timedlock_wait,@function - .hidden __lll_timedlock_wait - .align 16 -__lll_timedlock_wait: - cfi_startproc - pushl %ebp - cfi_adjust_cfa_offset(4) - cfi_rel_offset(%ebp, 0) - pushl %ebx - cfi_adjust_cfa_offset(4) - cfi_rel_offset(%ebx, 0) - -# ifndef __ASSUME_FUTEX_CLOCK_REALTIME -# ifdef PIC - LOAD_PIC_REG (bx) - cmpl $0, __have_futex_clock_realtime@GOTOFF(%ebx) -# else - cmpl $0, __have_futex_clock_realtime -# endif - je .Lreltmo -# endif - - cmpl $0, (%edx) - js 8f - - movl %ecx, %ebx - movl %esi, %ecx - movl %edx, %esi - movl $0xffffffff, %ebp - LOAD_FUTEX_WAIT_ABS (%ecx) - - movl $2, %edx - cmpl %edx, %eax - jne 2f - -1: movl $SYS_futex, %eax - movl $2, %edx - ENTER_KERNEL - -2: xchgl %edx, (%ebx) /* NB: lock is implied */ - - testl %edx, %edx - jz 3f - - cmpl $-ETIMEDOUT, %eax - je 4f - cmpl $-EINVAL, %eax - jne 1b -4: movl %eax, %edx - negl %edx - -3: movl %edx, %eax -7: popl %ebx - cfi_adjust_cfa_offset(-4) - cfi_restore(%ebx) - popl %ebp - cfi_adjust_cfa_offset(-4) - cfi_restore(%ebp) - ret - -8: movl $ETIMEDOUT, %eax - jmp 7b - -# ifndef __ASSUME_FUTEX_CLOCK_REALTIME -.Lreltmo: - /* Check for a valid timeout value. */ - cmpl $1000000000, 4(%edx) - jae 3f - - pushl %esi - cfi_adjust_cfa_offset(4) - cfi_rel_offset(%esi, 0) - pushl %edi - cfi_adjust_cfa_offset(4) - cfi_rel_offset(%edi, 0) - - /* Stack frame for the timespec and timeval structs. */ - subl $8, %esp - cfi_adjust_cfa_offset(8) - - movl %ecx, %ebp - movl %edx, %edi - - movl $2, %edx - xchgl %edx, (%ebp) - - test %edx, %edx - je 6f - -1: - /* Get current time. */ - movl %esp, %ebx - xorl %ecx, %ecx - movl $__NR_gettimeofday, %eax - ENTER_KERNEL - - /* Compute relative timeout. */ - movl 4(%esp), %eax - movl $1000, %edx - mul %edx /* Milli seconds to nano seconds. */ - movl (%edi), %ecx - movl 4(%edi), %edx - subl (%esp), %ecx - subl %eax, %edx - jns 4f - addl $1000000000, %edx - subl $1, %ecx -4: testl %ecx, %ecx - js 2f /* Time is already up. */ - - /* Store relative timeout. */ - movl %ecx, (%esp) - movl %edx, 4(%esp) - - /* Futex call. */ - movl %ebp, %ebx - movl $2, %edx - movl %esp, %esi - movl 16(%esp), %ecx - LOAD_FUTEX_WAIT (%ecx) - movl $SYS_futex, %eax - ENTER_KERNEL - - /* NB: %edx == 2 */ - xchgl %edx, (%ebp) - - testl %edx, %edx - je 6f - - cmpl $-ETIMEDOUT, %eax - jne 1b -2: movl $ETIMEDOUT, %edx - -6: addl $8, %esp - cfi_adjust_cfa_offset(-8) - popl %edi - cfi_adjust_cfa_offset(-4) - cfi_restore(%edi) - popl %esi - cfi_adjust_cfa_offset(-4) - cfi_restore(%esi) -7: popl %ebx - cfi_adjust_cfa_offset(-4) - cfi_restore(%ebx) - popl %ebp - cfi_adjust_cfa_offset(-4) - cfi_restore(%ebp) - movl %edx, %eax - ret - -3: movl $EINVAL, %edx - jmp 7b -# endif - cfi_endproc - .size __lll_timedlock_wait,.-__lll_timedlock_wait -#endif - - .globl __lll_unlock_wake_private - .type __lll_unlock_wake_private,@function - .hidden __lll_unlock_wake_private - .align 16 -__lll_unlock_wake_private: - cfi_startproc - pushl %ebx - cfi_adjust_cfa_offset(4) - pushl %ecx - cfi_adjust_cfa_offset(4) - pushl %edx - cfi_adjust_cfa_offset(4) - cfi_offset(%ebx, -8) - cfi_offset(%ecx, -12) - cfi_offset(%edx, -16) - - movl %eax, %ebx - movl $0, (%eax) - LOAD_PRIVATE_FUTEX_WAKE (%ecx) - movl $1, %edx /* Wake one thread. */ - movl $SYS_futex, %eax - ENTER_KERNEL - - popl %edx - cfi_adjust_cfa_offset(-4) - cfi_restore(%edx) - popl %ecx - cfi_adjust_cfa_offset(-4) - cfi_restore(%ecx) - popl %ebx - cfi_adjust_cfa_offset(-4) - cfi_restore(%ebx) - ret - cfi_endproc - .size __lll_unlock_wake_private,.-__lll_unlock_wake_private - -#if !IS_IN (libc) - .globl __lll_unlock_wake - .type __lll_unlock_wake,@function - .hidden __lll_unlock_wake - .align 16 -__lll_unlock_wake: - cfi_startproc - pushl %ebx - cfi_adjust_cfa_offset(4) - pushl %ecx - cfi_adjust_cfa_offset(4) - pushl %edx - cfi_adjust_cfa_offset(4) - cfi_offset(%ebx, -8) - cfi_offset(%ecx, -12) - cfi_offset(%edx, -16) - - movl %eax, %ebx - movl $0, (%eax) - LOAD_FUTEX_WAKE (%ecx) - movl $1, %edx /* Wake one thread. */ - movl $SYS_futex, %eax - ENTER_KERNEL - - popl %edx - cfi_adjust_cfa_offset(-4) - cfi_restore(%edx) - popl %ecx - cfi_adjust_cfa_offset(-4) - cfi_restore(%ecx) - popl %ebx - cfi_adjust_cfa_offset(-4) - cfi_restore(%ebx) - ret - cfi_endproc - .size __lll_unlock_wake,.-__lll_unlock_wake -#endif diff --git a/sysdeps/unix/sysv/linux/i386/lowlevellock.h b/sysdeps/unix/sysv/linux/i386/lowlevellock.h deleted file mode 100644 index 94dccc4ce7..0000000000 --- a/sysdeps/unix/sysv/linux/i386/lowlevellock.h +++ /dev/null @@ -1,240 +0,0 @@ -/* Copyright (C) 2002-2019 Free Software Foundation, Inc. - This file is part of the GNU C Library. - Contributed by Ulrich Drepper , 2002. - - The GNU C Library is free software; you can redistribute it and/or - modify it under the terms of the GNU Lesser General Public - License as published by the Free Software Foundation; either - version 2.1 of the License, or (at your option) any later version. - - The GNU C Library is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - Lesser General Public License for more details. - - You should have received a copy of the GNU Lesser General Public - License along with the GNU C Library; if not, see - . */ - -#ifndef _LOWLEVELLOCK_H -#define _LOWLEVELLOCK_H 1 - -#ifndef __ASSEMBLER__ -# include -# include -# include -# include -/* is generated from tcb-offsets.sym to define offsets - and sizes of types in as well as which includes - via nptl/descr.h. Don't include - when generating to avoid circular dependency which - may lead to build hang on a many-core machine. */ -# ifndef GEN_AS_CONST_HEADERS -# include -# endif - -# ifndef LOCK_INSTR -# ifdef UP -# define LOCK_INSTR /* nothing */ -# else -# define LOCK_INSTR "lock;" -# endif -# endif -#else -# ifndef LOCK -# ifdef UP -# define LOCK -# else -# define LOCK lock -# endif -# endif -#endif - -#include - -/* XXX Remove when no assembler code uses futexes anymore. */ -#define SYS_futex __NR_futex - -#ifndef __ASSEMBLER__ - -/* Initializer for compatibility lock. */ -#define LLL_LOCK_INITIALIZER (0) -#define LLL_LOCK_INITIALIZER_LOCKED (1) -#define LLL_LOCK_INITIALIZER_WAITERS (2) - - -/* NB: in the lll_trylock macro we simply return the value in %eax - after the cmpxchg instruction. In case the operation succeded this - value is zero. In case the operation failed, the cmpxchg instruction - has loaded the current value of the memory work which is guaranteed - to be nonzero. */ -#if !IS_IN (libc) || defined UP -# define __lll_trylock_asm LOCK_INSTR "cmpxchgl %2, %1" -#else -# define __lll_trylock_asm "cmpl $0, %%gs:%P5\n\t" \ - "je 0f\n\t" \ - "lock\n" \ - "0:\tcmpxchgl %2, %1" -#endif - -#define lll_trylock(futex) \ - ({ int ret; \ - __asm __volatile (__lll_trylock_asm \ - : "=a" (ret), "=m" (futex) \ - : "r" (LLL_LOCK_INITIALIZER_LOCKED), "m" (futex), \ - "0" (LLL_LOCK_INITIALIZER), \ - "i" (MULTIPLE_THREADS_OFFSET) \ - : "memory"); \ - ret; }) - - -#define lll_cond_trylock(futex) \ - ({ int ret; \ - __asm __volatile (LOCK_INSTR "cmpxchgl %2, %1" \ - : "=a" (ret), "=m" (futex) \ - : "r" (LLL_LOCK_INITIALIZER_WAITERS), \ - "m" (futex), "0" (LLL_LOCK_INITIALIZER) \ - : "memory"); \ - ret; }) - -#if !IS_IN (libc) || defined UP -# define __lll_lock_asm_start LOCK_INSTR "cmpxchgl %1, %2\n\t" -#else -# define __lll_lock_asm_start "cmpl $0, %%gs:%P6\n\t" \ - "je 0f\n\t" \ - "lock\n" \ - "0:\tcmpxchgl %1, %2\n\t" -#endif - -#define lll_lock(futex, private) \ - (void) \ - ({ int ignore1, ignore2; \ - if (__builtin_constant_p (private) && (private) == LLL_PRIVATE) \ - __asm __volatile (__lll_lock_asm_start \ - "jz 18f\n\t" \ - "1:\tleal %2, %%ecx\n" \ - "2:\tcall __lll_lock_wait_private\n" \ - "18:" \ - : "=a" (ignore1), "=c" (ignore2), "=m" (futex) \ - : "0" (0), "1" (1), "m" (futex), \ - "i" (MULTIPLE_THREADS_OFFSET) \ - : "memory"); \ - else \ - { \ - int ignore3; \ - __asm __volatile (__lll_lock_asm_start \ - "jz 18f\n\t" \ - "1:\tleal %2, %%edx\n" \ - "0:\tmovl %8, %%ecx\n" \ - "2:\tcall __lll_lock_wait\n" \ - "18:" \ - : "=a" (ignore1), "=c" (ignore2), \ - "=m" (futex), "=&d" (ignore3) \ - : "1" (1), "m" (futex), \ - "i" (MULTIPLE_THREADS_OFFSET), "0" (0), \ - "g" ((int) (private)) \ - : "memory"); \ - } \ - }) - - -/* Special version of lll_lock which causes the unlock function to - always wakeup waiters. */ -#define lll_cond_lock(futex, private) \ - (void) \ - ({ int ignore1, ignore2, ignore3; \ - __asm __volatile (LOCK_INSTR "cmpxchgl %1, %2\n\t" \ - "jz 18f\n\t" \ - "1:\tleal %2, %%edx\n" \ - "0:\tmovl %7, %%ecx\n" \ - "2:\tcall __lll_lock_wait\n" \ - "18:" \ - : "=a" (ignore1), "=c" (ignore2), "=m" (futex), \ - "=&d" (ignore3) \ - : "0" (0), "1" (2), "m" (futex), "g" ((int) (private))\ - : "memory"); \ - }) - - -#define lll_timedlock(futex, timeout, private) \ - ({ int result, ignore1, ignore2, ignore3; \ - __asm __volatile (LOCK_INSTR "cmpxchgl %1, %3\n\t" \ - "jz 18f\n\t" \ - "1:\tleal %3, %%ecx\n" \ - "0:\tmovl %8, %%edx\n" \ - "2:\tcall __lll_timedlock_wait\n" \ - "18:" \ - : "=a" (result), "=c" (ignore1), "=&d" (ignore2), \ - "=m" (futex), "=S" (ignore3) \ - : "0" (0), "1" (1), "m" (futex), "m" (timeout), \ - "4" ((int) (private)) \ - : "memory"); \ - result; }) - -extern int __lll_timedlock_elision (int *futex, short *adapt_count, - const struct timespec *timeout, - int private) attribute_hidden; - -#define lll_timedlock_elision(futex, adapt_count, timeout, private) \ - __lll_timedlock_elision(&(futex), &(adapt_count), timeout, private) - -#if !IS_IN (libc) || defined UP -# define __lll_unlock_asm LOCK_INSTR "subl $1, %0\n\t" -#else -# define __lll_unlock_asm "cmpl $0, %%gs:%P3\n\t" \ - "je 0f\n\t" \ - "lock\n" \ - "0:\tsubl $1,%0\n\t" -#endif - -#define lll_unlock(futex, private) \ - (void) \ - ({ int ignore; \ - if (__builtin_constant_p (private) && (private) == LLL_PRIVATE) \ - __asm __volatile (__lll_unlock_asm \ - "je 18f\n\t" \ - "1:\tleal %0, %%eax\n" \ - "2:\tcall __lll_unlock_wake_private\n" \ - "18:" \ - : "=m" (futex), "=&a" (ignore) \ - : "m" (futex), "i" (MULTIPLE_THREADS_OFFSET) \ - : "memory"); \ - else \ - { \ - int ignore2; \ - __asm __volatile (__lll_unlock_asm \ - "je 18f\n\t" \ - "1:\tleal %0, %%eax\n" \ - "0:\tmovl %5, %%ecx\n" \ - "2:\tcall __lll_unlock_wake\n" \ - "18:" \ - : "=m" (futex), "=&a" (ignore), "=&c" (ignore2) \ - : "i" (MULTIPLE_THREADS_OFFSET), "m" (futex), \ - "g" ((int) (private)) \ - : "memory"); \ - } \ - }) - - -#define lll_islocked(futex) \ - (futex != LLL_LOCK_INITIALIZER) - -extern int __lll_lock_elision (int *futex, short *adapt_count, int private) - attribute_hidden; - -extern int __lll_unlock_elision(int *lock, int private) - attribute_hidden; - -extern int __lll_trylock_elision(int *lock, short *adapt_count) - attribute_hidden; - -#define lll_lock_elision(futex, adapt_count, private) \ - __lll_lock_elision (&(futex), &(adapt_count), private) -#define lll_unlock_elision(futex, adapt_count, private) \ - __lll_unlock_elision (&(futex), private) -#define lll_trylock_elision(futex, adapt_count) \ - __lll_trylock_elision(&(futex), &(adapt_count)) - -#endif /* !__ASSEMBLER__ */ - -#endif /* lowlevellock.h */ diff --git a/sysdeps/unix/sysv/linux/x86/lowlevellock.h b/sysdeps/unix/sysv/linux/x86/lowlevellock.h new file mode 100644 index 0000000000..2652a00736 --- /dev/null +++ b/sysdeps/unix/sysv/linux/x86/lowlevellock.h @@ -0,0 +1,110 @@ +/* Low-level lock implementation, x86 version. + Copyright (C) 2019 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#ifndef _X86_64_LOWLEVELLOCK_H +#define _X86_64_LOWLEVELLOCK_H 1 + +#ifndef __ASSEMBLER__ +#include +#include + +/* The lll_trylock, lll_lock, and lll_unlock implements a single-thread + optimization using the cmpxchgl instruction. It checks if the process + is single thread and avoid a more expensive atomic instruction. */ + +/* The single-thread optimization only works for libc itself, we need + atomicity for libpthread in case of shared futexes. */ +#if !IS_IN(libc) +# define is_single_thread 0 +#else +# define is_single_thread SINGLE_THREAD_P +#endif + +/* In the __lllc_as we simply return the value in %eax after the cmpxchg + instruction. In case the operation succeded this value is zero. In case + the operation failed, the cmpxchg instruction has loaded the current value + of the memory work which is guaranteed to be nonzero. */ +static inline int +__attribute__ ((always_inline)) +__lll_cas_lock (int *futex) +{ + int ret; + asm volatile ("cmpxchgl %2, %1" + : "=a" (ret), "=m" (*futex) + : "r" (1), "m" (*futex), "0" (0) + : "memory"); + return ret; +} + +#undef lll_trylock +#define lll_trylock(lock) \ + ({ \ + int __ret; \ + if (is_single_thread) \ + __ret = __lll_cas_lock (&(lock)); \ + else \ + __ret = __lll_trylock (&(lock)); \ + __ret; \ + }) + +#undef lll_lock +#define lll_lock(lock, private) \ + ((void) \ + ({ \ + if (is_single_thread) \ + __lll_cas_lock (&(lock)); \ + else \ + __lll_lock (&(lock), private); \ + })) + +#undef lll_unlock +#define lll_unlock(lock, private) \ + ((void) \ + ({ \ + if (is_single_thread) \ + (lock)--; \ + else \ + __lll_unlock (&(lock), private); \ + })) + +extern int __lll_timedlock_elision (int *futex, short *adapt_count, + const struct timespec *timeout, + int private) attribute_hidden; + +#define lll_timedlock_elision(futex, adapt_count, timeout, private) \ + __lll_timedlock_elision(&(futex), &(adapt_count), timeout, private) + +extern int __lll_lock_elision (int *futex, short *adapt_count, int private) + attribute_hidden; + +extern int __lll_unlock_elision (int *lock, int private) + attribute_hidden; + +extern int __lll_trylock_elision (int *lock, short *adapt_count) + attribute_hidden; + +#define lll_lock_elision(futex, adapt_count, private) \ + __lll_lock_elision (&(futex), &(adapt_count), private) +#define lll_unlock_elision(futex, adapt_count, private) \ + __lll_unlock_elision (&(futex), private) +#define lll_trylock_elision(futex, adapt_count) \ + __lll_trylock_elision (&(futex), &(adapt_count)) + +#endif /* !__ASSEMBLER__ */ + +#endif /* lowlevellock.h */ diff --git a/sysdeps/unix/sysv/linux/x86_64/cancellation.S b/sysdeps/unix/sysv/linux/x86_64/cancellation.S index 7d169d9aca..bb4910764a 100644 --- a/sysdeps/unix/sysv/linux/x86_64/cancellation.S +++ b/sysdeps/unix/sysv/linux/x86_64/cancellation.S @@ -19,7 +19,7 @@ #include #include #include -#include "lowlevellock.h" +#include #define PTHREAD_UNWIND JUMPTARGET(__pthread_unwind) #if IS_IN (libpthread) diff --git a/sysdeps/unix/sysv/linux/x86_64/libc-lowlevellock.S b/sysdeps/unix/sysv/linux/x86_64/libc-lowlevellock.S deleted file mode 100644 index 32025277f3..0000000000 --- a/sysdeps/unix/sysv/linux/x86_64/libc-lowlevellock.S +++ /dev/null @@ -1,19 +0,0 @@ -/* Copyright (C) 2002-2019 Free Software Foundation, Inc. - This file is part of the GNU C Library. - Contributed by Ulrich Drepper , 2002. - - The GNU C Library is free software; you can redistribute it and/or - modify it under the terms of the GNU Lesser General Public - License as published by the Free Software Foundation; either - version 2.1 of the License, or (at your option) any later version. - - The GNU C Library is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - Lesser General Public License for more details. - - You should have received a copy of the GNU Lesser General Public - License along with the GNU C Library; if not, see - . */ - -#include "lowlevellock.S" diff --git a/sysdeps/unix/sysv/linux/x86_64/lll_timedlock_wait.c b/sysdeps/unix/sysv/linux/x86_64/lll_timedlock_wait.c deleted file mode 100644 index f6875b8f89..0000000000 --- a/sysdeps/unix/sysv/linux/x86_64/lll_timedlock_wait.c +++ /dev/null @@ -1 +0,0 @@ -/* __lll_timedlock_wait is in lowlevellock.S. */ diff --git a/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S b/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S deleted file mode 100644 index 166dbcfd3b..0000000000 --- a/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S +++ /dev/null @@ -1,348 +0,0 @@ -/* Copyright (C) 2002-2019 Free Software Foundation, Inc. - This file is part of the GNU C Library. - Contributed by Ulrich Drepper , 2002. - - The GNU C Library is free software; you can redistribute it and/or - modify it under the terms of the GNU Lesser General Public - License as published by the Free Software Foundation; either - version 2.1 of the License, or (at your option) any later version. - - The GNU C Library is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - Lesser General Public License for more details. - - You should have received a copy of the GNU Lesser General Public - License along with the GNU C Library; if not, see - . */ - -#include -#include -#include -#include - -#include - - .text - -#define LOAD_PRIVATE_FUTEX_WAIT(reg) \ - movl $(FUTEX_WAIT | FUTEX_PRIVATE_FLAG), reg -#define LOAD_PRIVATE_FUTEX_WAKE(reg) \ - movl $(FUTEX_WAKE | FUTEX_PRIVATE_FLAG), reg -#define LOAD_FUTEX_WAIT(reg) \ - xorl $(FUTEX_WAIT | FUTEX_PRIVATE_FLAG), reg -#define LOAD_FUTEX_WAIT_ABS(reg) \ - xorl $(FUTEX_WAIT_BITSET | FUTEX_PRIVATE_FLAG | FUTEX_CLOCK_REALTIME), reg -#define LOAD_FUTEX_WAKE(reg) \ - xorl $(FUTEX_WAKE | FUTEX_PRIVATE_FLAG), reg - - - .globl __lll_lock_wait_private - .type __lll_lock_wait_private,@function - .hidden __lll_lock_wait_private - .align 16 -__lll_lock_wait_private: - cfi_startproc - pushq %r10 - cfi_adjust_cfa_offset(8) - pushq %rdx - cfi_adjust_cfa_offset(8) - cfi_offset(%r10, -16) - cfi_offset(%rdx, -24) - xorq %r10, %r10 /* No timeout. */ - movl $2, %edx - LOAD_PRIVATE_FUTEX_WAIT (%esi) - - cmpl %edx, %eax /* NB: %edx == 2 */ - jne 2f - -1: LIBC_PROBE (lll_lock_wait_private, 1, %rdi) - movl $SYS_futex, %eax - syscall - -2: movl %edx, %eax - xchgl %eax, (%rdi) /* NB: lock is implied */ - - testl %eax, %eax - jnz 1b - - popq %rdx - cfi_adjust_cfa_offset(-8) - cfi_restore(%rdx) - popq %r10 - cfi_adjust_cfa_offset(-8) - cfi_restore(%r10) - retq - cfi_endproc - .size __lll_lock_wait_private,.-__lll_lock_wait_private - -#if !IS_IN (libc) - .globl __lll_lock_wait - .type __lll_lock_wait,@function - .hidden __lll_lock_wait - .align 16 -__lll_lock_wait: - cfi_startproc - pushq %r10 - cfi_adjust_cfa_offset(8) - pushq %rdx - cfi_adjust_cfa_offset(8) - cfi_offset(%r10, -16) - cfi_offset(%rdx, -24) - xorq %r10, %r10 /* No timeout. */ - movl $2, %edx - LOAD_FUTEX_WAIT (%esi) - - cmpl %edx, %eax /* NB: %edx == 2 */ - jne 2f - -1: LIBC_PROBE (lll_lock_wait, 2, %rdi, %rsi) - movl $SYS_futex, %eax - syscall - -2: movl %edx, %eax - xchgl %eax, (%rdi) /* NB: lock is implied */ - - testl %eax, %eax - jnz 1b - - popq %rdx - cfi_adjust_cfa_offset(-8) - cfi_restore(%rdx) - popq %r10 - cfi_adjust_cfa_offset(-8) - cfi_restore(%r10) - retq - cfi_endproc - .size __lll_lock_wait,.-__lll_lock_wait - - /* %rdi: futex - %rsi: flags - %rdx: timeout - %eax: futex value - */ - .globl __lll_timedlock_wait - .type __lll_timedlock_wait,@function - .hidden __lll_timedlock_wait - .align 16 -__lll_timedlock_wait: - cfi_startproc -# ifndef __ASSUME_FUTEX_CLOCK_REALTIME -# ifdef PIC - cmpl $0, __have_futex_clock_realtime(%rip) -# else - cmpl $0, __have_futex_clock_realtime -# endif - je .Lreltmo -# endif - - cmpq $0, (%rdx) - js 5f - - pushq %r9 - cfi_adjust_cfa_offset(8) - cfi_rel_offset(%r9, 0) - - movq %rdx, %r10 - movl $0xffffffff, %r9d - LOAD_FUTEX_WAIT_ABS (%esi) - - movl $2, %edx - cmpl %edx, %eax - jne 2f - -1: movl $SYS_futex, %eax - movl $2, %edx - syscall - -2: xchgl %edx, (%rdi) /* NB: lock is implied */ - - testl %edx, %edx - jz 3f - - cmpl $-ETIMEDOUT, %eax - je 4f - cmpl $-EINVAL, %eax - jne 1b -4: movl %eax, %edx - negl %edx - -3: movl %edx, %eax - popq %r9 - cfi_adjust_cfa_offset(-8) - cfi_restore(%r9) - retq - -5: movl $ETIMEDOUT, %eax - retq - -# ifndef __ASSUME_FUTEX_CLOCK_REALTIME -.Lreltmo: - /* Check for a valid timeout value. */ - cmpq $1000000000, 8(%rdx) - jae 3f - - pushq %r8 - cfi_adjust_cfa_offset(8) - pushq %r9 - cfi_adjust_cfa_offset(8) - pushq %r12 - cfi_adjust_cfa_offset(8) - pushq %r13 - cfi_adjust_cfa_offset(8) - pushq %r14 - cfi_adjust_cfa_offset(8) - cfi_offset(%r8, -16) - cfi_offset(%r9, -24) - cfi_offset(%r12, -32) - cfi_offset(%r13, -40) - cfi_offset(%r14, -48) - pushq %rsi - cfi_adjust_cfa_offset(8) - - /* Stack frame for the timespec and timeval structs. */ - subq $24, %rsp - cfi_adjust_cfa_offset(24) - - movq %rdi, %r12 - movq %rdx, %r13 - - movl $2, %edx - xchgl %edx, (%r12) - - testl %edx, %edx - je 6f - -1: - /* Get current time. */ - movq %rsp, %rdi - xorl %esi, %esi - /* This call works because we directly jump to a system call entry - which preserves all the registers. */ - call JUMPTARGET(__gettimeofday) - - /* Compute relative timeout. */ - movq 8(%rsp), %rax - movl $1000, %edi - mul %rdi /* Milli seconds to nano seconds. */ - movq (%r13), %rdi - movq 8(%r13), %rsi - subq (%rsp), %rdi - subq %rax, %rsi - jns 4f - addq $1000000000, %rsi - decq %rdi -4: testq %rdi, %rdi - js 2f /* Time is already up. */ - - /* Store relative timeout. */ - movq %rdi, (%rsp) - movq %rsi, 8(%rsp) - - /* Futex call. */ - movl $2, %edx - movl $1, %eax - movq %rsp, %r10 - movl 24(%rsp), %esi - LOAD_FUTEX_WAIT (%esi) - movq %r12, %rdi - movl $SYS_futex, %eax - syscall - - /* NB: %edx == 2 */ - xchgl %edx, (%r12) - - testl %edx, %edx - je 6f - - cmpl $-ETIMEDOUT, %eax - jne 1b -2: movl $ETIMEDOUT, %edx - -6: addq $32, %rsp - cfi_adjust_cfa_offset(-32) - popq %r14 - cfi_adjust_cfa_offset(-8) - cfi_restore(%r14) - popq %r13 - cfi_adjust_cfa_offset(-8) - cfi_restore(%r13) - popq %r12 - cfi_adjust_cfa_offset(-8) - cfi_restore(%r12) - popq %r9 - cfi_adjust_cfa_offset(-8) - cfi_restore(%r9) - popq %r8 - cfi_adjust_cfa_offset(-8) - cfi_restore(%r8) - movl %edx, %eax - retq - -3: movl $EINVAL, %eax - retq -# endif - cfi_endproc - .size __lll_timedlock_wait,.-__lll_timedlock_wait -#endif - - - .globl __lll_unlock_wake_private - .type __lll_unlock_wake_private,@function - .hidden __lll_unlock_wake_private - .align 16 -__lll_unlock_wake_private: - cfi_startproc - pushq %rsi - cfi_adjust_cfa_offset(8) - pushq %rdx - cfi_adjust_cfa_offset(8) - cfi_offset(%rsi, -16) - cfi_offset(%rdx, -24) - - movl $0, (%rdi) - LOAD_PRIVATE_FUTEX_WAKE (%esi) - movl $1, %edx /* Wake one thread. */ - movl $SYS_futex, %eax - syscall - - popq %rdx - cfi_adjust_cfa_offset(-8) - cfi_restore(%rdx) - popq %rsi - cfi_adjust_cfa_offset(-8) - cfi_restore(%rsi) - retq - cfi_endproc - .size __lll_unlock_wake_private,.-__lll_unlock_wake_private - -#if !IS_IN (libc) - .globl __lll_unlock_wake - .type __lll_unlock_wake,@function - .hidden __lll_unlock_wake - .align 16 -__lll_unlock_wake: - cfi_startproc - pushq %rsi - cfi_adjust_cfa_offset(8) - pushq %rdx - cfi_adjust_cfa_offset(8) - cfi_offset(%rsi, -16) - cfi_offset(%rdx, -24) - - movl $0, (%rdi) - LOAD_FUTEX_WAKE (%esi) - movl $1, %edx /* Wake one thread. */ - movl $SYS_futex, %eax - syscall - - popq %rdx - cfi_adjust_cfa_offset(-8) - cfi_restore(%rdx) - popq %rsi - cfi_adjust_cfa_offset(-8) - cfi_restore(%rsi) - retq - cfi_endproc - .size __lll_unlock_wake,.-__lll_unlock_wake -#endif diff --git a/sysdeps/unix/sysv/linux/x86_64/lowlevellock.h b/sysdeps/unix/sysv/linux/x86_64/lowlevellock.h deleted file mode 100644 index 8cbc1caa5b..0000000000 --- a/sysdeps/unix/sysv/linux/x86_64/lowlevellock.h +++ /dev/null @@ -1,243 +0,0 @@ -/* Copyright (C) 2002-2019 Free Software Foundation, Inc. - This file is part of the GNU C Library. - Contributed by Ulrich Drepper , 2002. - - The GNU C Library is free software; you can redistribute it and/or - modify it under the terms of the GNU Lesser General Public - License as published by the Free Software Foundation; either - version 2.1 of the License, or (at your option) any later version. - - The GNU C Library is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - Lesser General Public License for more details. - - You should have received a copy of the GNU Lesser General Public - License along with the GNU C Library; if not, see - . */ - -#ifndef _LOWLEVELLOCK_H -#define _LOWLEVELLOCK_H 1 - -#ifndef __ASSEMBLER__ -# include -# include -# include -# include - -# ifndef LOCK_INSTR -# ifdef UP -# define LOCK_INSTR /* nothing */ -# else -# define LOCK_INSTR "lock;" -# endif -# endif -#else -# ifndef LOCK -# ifdef UP -# define LOCK -# else -# define LOCK lock -# endif -# endif -#endif - -#include - -/* XXX Remove when no assembler code uses futexes anymore. */ -#define SYS_futex __NR_futex - -#ifndef __ASSEMBLER__ - -/* Initializer for lock. */ -#define LLL_LOCK_INITIALIZER (0) -#define LLL_LOCK_INITIALIZER_LOCKED (1) -#define LLL_LOCK_INITIALIZER_WAITERS (2) - - -/* NB: in the lll_trylock macro we simply return the value in %eax - after the cmpxchg instruction. In case the operation succeded this - value is zero. In case the operation failed, the cmpxchg instruction - has loaded the current value of the memory work which is guaranteed - to be nonzero. */ -#if !IS_IN (libc) || defined UP -# define __lll_trylock_asm LOCK_INSTR "cmpxchgl %2, %1" -#else -# define __lll_trylock_asm "cmpl $0, __libc_multiple_threads(%%rip)\n\t" \ - "je 0f\n\t" \ - "lock; cmpxchgl %2, %1\n\t" \ - "jmp 1f\n\t" \ - "0:\tcmpxchgl %2, %1\n\t" \ - "1:" -#endif - -#define lll_trylock(futex) \ - ({ int ret; \ - __asm __volatile (__lll_trylock_asm \ - : "=a" (ret), "=m" (futex) \ - : "r" (LLL_LOCK_INITIALIZER_LOCKED), "m" (futex), \ - "0" (LLL_LOCK_INITIALIZER) \ - : "memory"); \ - ret; }) - -#define lll_cond_trylock(futex) \ - ({ int ret; \ - __asm __volatile (LOCK_INSTR "cmpxchgl %2, %1" \ - : "=a" (ret), "=m" (futex) \ - : "r" (LLL_LOCK_INITIALIZER_WAITERS), \ - "m" (futex), "0" (LLL_LOCK_INITIALIZER) \ - : "memory"); \ - ret; }) - -#if !IS_IN (libc) || defined UP -# define __lll_lock_asm_start LOCK_INSTR "cmpxchgl %4, %2\n\t" \ - "jz 24f\n\t" -#else -# define __lll_lock_asm_start "cmpl $0, __libc_multiple_threads(%%rip)\n\t" \ - "je 0f\n\t" \ - "lock; cmpxchgl %4, %2\n\t" \ - "jnz 1f\n\t" \ - "jmp 24f\n" \ - "0:\tcmpxchgl %4, %2\n\t" \ - "jz 24f\n\t" -#endif - -#define lll_lock(futex, private) \ - (void) \ - ({ int ignore1, ignore2, ignore3; \ - if (__builtin_constant_p (private) && (private) == LLL_PRIVATE) \ - __asm __volatile (__lll_lock_asm_start \ - "1:\tlea %2, %%" RDI_LP "\n" \ - "2:\tsub $128, %%" RSP_LP "\n" \ - ".cfi_adjust_cfa_offset 128\n" \ - "3:\tcallq __lll_lock_wait_private\n" \ - "4:\tadd $128, %%" RSP_LP "\n" \ - ".cfi_adjust_cfa_offset -128\n" \ - "24:" \ - : "=S" (ignore1), "=&D" (ignore2), "=m" (futex), \ - "=a" (ignore3) \ - : "0" (1), "m" (futex), "3" (0) \ - : "cx", "r11", "cc", "memory"); \ - else \ - __asm __volatile (__lll_lock_asm_start \ - "1:\tlea %2, %%" RDI_LP "\n" \ - "2:\tsub $128, %%" RSP_LP "\n" \ - ".cfi_adjust_cfa_offset 128\n" \ - "3:\tcallq __lll_lock_wait\n" \ - "4:\tadd $128, %%" RSP_LP "\n" \ - ".cfi_adjust_cfa_offset -128\n" \ - "24:" \ - : "=S" (ignore1), "=D" (ignore2), "=m" (futex), \ - "=a" (ignore3) \ - : "1" (1), "m" (futex), "3" (0), "0" (private) \ - : "cx", "r11", "cc", "memory"); \ - }) \ - -#define lll_cond_lock(futex, private) \ - (void) \ - ({ int ignore1, ignore2, ignore3; \ - __asm __volatile (LOCK_INSTR "cmpxchgl %4, %2\n\t" \ - "jz 24f\n" \ - "1:\tlea %2, %%" RDI_LP "\n" \ - "2:\tsub $128, %%" RSP_LP "\n" \ - ".cfi_adjust_cfa_offset 128\n" \ - "3:\tcallq __lll_lock_wait\n" \ - "4:\tadd $128, %%" RSP_LP "\n" \ - ".cfi_adjust_cfa_offset -128\n" \ - "24:" \ - : "=S" (ignore1), "=D" (ignore2), "=m" (futex), \ - "=a" (ignore3) \ - : "1" (2), "m" (futex), "3" (0), "0" (private) \ - : "cx", "r11", "cc", "memory"); \ - }) - -#define lll_timedlock(futex, timeout, private) \ - ({ int result, ignore1, ignore2, ignore3; \ - __asm __volatile (LOCK_INSTR "cmpxchgl %1, %4\n\t" \ - "jz 24f\n" \ - "1:\tlea %4, %%" RDI_LP "\n" \ - "0:\tmov %8, %%" RDX_LP "\n" \ - "2:\tsub $128, %%" RSP_LP "\n" \ - ".cfi_adjust_cfa_offset 128\n" \ - "3:\tcallq __lll_timedlock_wait\n" \ - "4:\tadd $128, %%" RSP_LP "\n" \ - ".cfi_adjust_cfa_offset -128\n" \ - "24:" \ - : "=a" (result), "=D" (ignore1), "=S" (ignore2), \ - "=&d" (ignore3), "=m" (futex) \ - : "0" (0), "1" (1), "m" (futex), "m" (timeout), \ - "2" (private) \ - : "memory", "cx", "cc", "r10", "r11"); \ - result; }) - -extern int __lll_timedlock_elision (int *futex, short *adapt_count, - const struct timespec *timeout, - int private) attribute_hidden; - -#define lll_timedlock_elision(futex, adapt_count, timeout, private) \ - __lll_timedlock_elision(&(futex), &(adapt_count), timeout, private) - -#if !IS_IN (libc) || defined UP -# define __lll_unlock_asm_start LOCK_INSTR "decl %0\n\t" \ - "je 24f\n\t" -#else -# define __lll_unlock_asm_start "cmpl $0, __libc_multiple_threads(%%rip)\n\t" \ - "je 0f\n\t" \ - "lock; decl %0\n\t" \ - "jne 1f\n\t" \ - "jmp 24f\n\t" \ - "0:\tdecl %0\n\t" \ - "je 24f\n\t" -#endif - -#define lll_unlock(futex, private) \ - (void) \ - ({ int ignore; \ - if (__builtin_constant_p (private) && (private) == LLL_PRIVATE) \ - __asm __volatile (__lll_unlock_asm_start \ - "1:\tlea %0, %%" RDI_LP "\n" \ - "2:\tsub $128, %%" RSP_LP "\n" \ - ".cfi_adjust_cfa_offset 128\n" \ - "3:\tcallq __lll_unlock_wake_private\n" \ - "4:\tadd $128, %%" RSP_LP "\n" \ - ".cfi_adjust_cfa_offset -128\n" \ - "24:" \ - : "=m" (futex), "=&D" (ignore) \ - : "m" (futex) \ - : "ax", "cx", "r11", "cc", "memory"); \ - else \ - __asm __volatile (__lll_unlock_asm_start \ - "1:\tlea %0, %%" RDI_LP "\n" \ - "2:\tsub $128, %%" RSP_LP "\n" \ - ".cfi_adjust_cfa_offset 128\n" \ - "3:\tcallq __lll_unlock_wake\n" \ - "4:\tadd $128, %%" RSP_LP "\n" \ - ".cfi_adjust_cfa_offset -128\n" \ - "24:" \ - : "=m" (futex), "=&D" (ignore) \ - : "m" (futex), "S" (private) \ - : "ax", "cx", "r11", "cc", "memory"); \ - }) - -#define lll_islocked(futex) \ - (futex != LLL_LOCK_INITIALIZER) - -extern int __lll_lock_elision (int *futex, short *adapt_count, int private) - attribute_hidden; - -extern int __lll_unlock_elision (int *lock, int private) - attribute_hidden; - -extern int __lll_trylock_elision (int *lock, short *adapt_count) - attribute_hidden; - -#define lll_lock_elision(futex, adapt_count, private) \ - __lll_lock_elision (&(futex), &(adapt_count), private) -#define lll_unlock_elision(futex, adapt_count, private) \ - __lll_unlock_elision (&(futex), private) -#define lll_trylock_elision(futex, adapt_count) \ - __lll_trylock_elision (&(futex), &(adapt_count)) - -#endif /* !__ASSEMBLER__ */ - -#endif /* lowlevellock.h */