From patchwork Tue Jan 10 21:00:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adhemerval Zanella Netto X-Patchwork-Id: 640899 Delivered-To: patch@linaro.org Received: by 2002:a17:522:f3c4:b0:4b4:3859:abed with SMTP id in4csp2916328pvb; Tue, 10 Jan 2023 13:02:22 -0800 (PST) X-Google-Smtp-Source: AMrXdXtw2Sw8qrhb3b+1MMOZR8ueD+OAtupuxy2GF4N/zaK9Xc9rovc2RFT/DqxIkzj7KeFEtrG/ X-Received: by 2002:a17:906:54d1:b0:84d:3819:79b9 with SMTP id c17-20020a17090654d100b0084d381979b9mr11837533ejp.71.1673384541922; Tue, 10 Jan 2023 13:02:21 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673384541; cv=none; d=google.com; s=arc-20160816; b=oUnJyd9KPFidPNUoQ3DruJPZaPVSSyFPZ6NSwRCY/62inSpg+H9agTEggtzQyMqyEi th+0WLNS0qreCXSo4x7bk16uRZD9hS8oolrP5MKqG0Y8xuVVdaY5VikDIUpkO9Lj7E/y 4ebWNIIW4PmAop4AOe+LcT5Q0wuAP+zHljxXCXwuyxN5Bm40ptMcxMgBxGVMbrm0ETdw xFQ93sNyqtZXCDO9wsRiZlzEutgMzqEXrx8g3G68qesDk3YFzRfsG3//uFSsFD2klgzS byMWaXHtcaBvG6ghlphCKCMazODXXg/S8MbFj825tr2WURbDLBmwS9YTXeEk1K3X6Q3l YV5Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:reply-to:from:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence :content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:dmarc-filter:delivered-to :dkim-signature:dkim-filter; bh=jsdgiZ4ZKa5otRpCpPYLEYCiKOevxnsN7LwyakEDfJI=; b=xy5sdwEtVlPPJR1n6Di9JCbNpsUUS+/BupfEmE3PgPWjx1E6MTYZk1vylw0NYBGsWZ 9nRdsq/SZ5Abdnek81WUvjbR33s+ib/x/4a6XkQiGCVacMHbr4NrNBFgsCL3fCuJmAc+ Q4aFCRIbzFSw1tkjBxSWslaIXflIrJkMgWHnU2k+sh1KMmzazfl8J4kFeqTVyp/5r53P hIWf1vXF+1JM1Bk1Mk3XLv/LkQfkt9SvebGWarP+AN1PX6D1o2uB4K9bqubQg5frZioM WcBpFuLO2QFoccjddlYvG/yWXvc/0tbilKUQz/trPTThZfdhe8JUTSdaPmd/gQyOzKU6 AIEA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@sourceware.org header.s=default header.b=wvMtxYML; spf=pass (google.com: domain of libc-alpha-bounces+patch=linaro.org@sourceware.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="libc-alpha-bounces+patch=linaro.org@sourceware.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=sourceware.org Return-Path: Received: from sourceware.org (server2.sourceware.org. [2620:52:3:1:0:246e:9693:128c]) by mx.google.com with ESMTPS id xb13-20020a170907070d00b007c1600359ecsi13053880ejb.441.2023.01.10.13.02.21 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Jan 2023 13:02:21 -0800 (PST) Received-SPF: pass (google.com: domain of libc-alpha-bounces+patch=linaro.org@sourceware.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) client-ip=2620:52:3:1:0:246e:9693:128c; Authentication-Results: mx.google.com; dkim=pass header.i=@sourceware.org header.s=default header.b=wvMtxYML; spf=pass (google.com: domain of libc-alpha-bounces+patch=linaro.org@sourceware.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="libc-alpha-bounces+patch=linaro.org@sourceware.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 5053A3832364 for ; Tue, 10 Jan 2023 21:02:20 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 5053A3832364 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1673384540; bh=jsdgiZ4ZKa5otRpCpPYLEYCiKOevxnsN7LwyakEDfJI=; h=To:Cc:Subject:Date:In-Reply-To:References:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=wvMtxYMLpOawXaJhri44FTzdCzgRF9oD4ElVeA9DTQYTuOfDytPi+6Y6hqxLckTDG Ves82m8GXDLqtGvaP1vS9z6gjrVYYcciMXCn61/duIW0M5y6J6793jpl4K7wgTuWAV 2T8grR7xLwTymW6o9KzWgCUomJzHOBlvTS4VImOM= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-oo1-xc31.google.com (mail-oo1-xc31.google.com [IPv6:2607:f8b0:4864:20::c31]) by sourceware.org (Postfix) with ESMTPS id 2ED1C3858404 for ; Tue, 10 Jan 2023 21:01:25 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 2ED1C3858404 Received: by mail-oo1-xc31.google.com with SMTP id h3-20020a4aa283000000b004ead187bd6eso3541424ool.5 for ; Tue, 10 Jan 2023 13:01:25 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jsdgiZ4ZKa5otRpCpPYLEYCiKOevxnsN7LwyakEDfJI=; b=IvMFPJMs8JCaJbgW5k0qhJAGWtwbc/0mogW2kQicbXTNUgWJhGVeMcE0fXtATZpDMx 0A4Un7qASHzxaxUgD0SoBCmra2d7nmJ33znwsEhxNpjszMEQNcP8pqpGrChe8NATKUWo b23EzvSurjuaBC5iKxf/rmX2qEndazfIkjoZgo+nHtfk8TdtnhdtAPYMV6f6EAwTtzjS X1TXsy1SETbnftMPHqKanx34bOTSysSNIfsrtrE+zED0444D5UphdPaE+5lN0M94mOOM RdwA1zz00FEiF/LM+3vZb8I4b2IyD8CylVF9jQ9EAIWL4i7uXVObxmqhCxxbppntaKS4 BcYw== X-Gm-Message-State: AFqh2kpvYqxwwdlOXTQjjOE2PXNjRYWA/UqjrAEXskbuPHICzreoxLaX MWRzOCVC8o3F/n+sPh6KvEqKFGFAG2CL//FxRt0= X-Received: by 2002:a4a:c54d:0:b0:4f2:ff6:6168 with SMTP id j13-20020a4ac54d000000b004f20ff66168mr1674540ooq.3.1673384483561; Tue, 10 Jan 2023 13:01:23 -0800 (PST) Received: from mandiga.. ([2804:1b3:a7c0:a93a:e8a0:dd55:3328:997]) by smtp.gmail.com with ESMTPSA id r5-20020a4a83c5000000b0049ee88e86f9sm6202193oog.10.2023.01.10.13.01.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Jan 2023 13:01:22 -0800 (PST) To: libc-alpha@sourceware.org, Noah Goldstein , Richard Henderson Cc: Adhemerval Zanella Netto Subject: [PATCH v6 04/17] Add string vectorized find and detection functions Date: Tue, 10 Jan 2023 18:00:53 -0300 Message-Id: <20230110210106.1457686-5-adhemerval.zanella@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230110210106.1457686-1-adhemerval.zanella@linaro.org> References: <20230110210106.1457686-1-adhemerval.zanella@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-12.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Adhemerval Zanella via Libc-alpha From: Adhemerval Zanella Netto Reply-To: Adhemerval Zanella Errors-To: libc-alpha-bounces+patch=linaro.org@sourceware.org Sender: "Libc-alpha" From: Adhemerval Zanella Netto This patch adds generic string find and detection meant to be used in generic vectorized string implementation. The idea is to decompose the basic string operation so each architecture can reimplement if it provides any specialized hardware instruction. The 'string-fza.h' provides zero byte detection functions (find_zero_low, find_zero_all, find_eq_low, find_eq_all, find_zero_eq_low, find_zero_eq_all, find_zero_ne_low, and find_zero_ne_all). They are used on both functions provided by 'string-fzb.h' and 'string-fzi'. The 'string-fzb.h' provides boolean zero byte detection with the functions: - has_zero: determine if any byte within a word is zero. - has_eq: determine byte equality between two words. - has_zero_eq: determine if any byte within a word is zero along with byte equality between two words. The 'string-fzi.h' provides zero byte detection along with its positions: - index_first_zero: return index of first zero byte within a word. - index_first_eq: return index of first byte different between two words. - index_first_zero_eq: return index of first zero byte within a word or first byte different between two words. - index_first_zero_ne: return index of first zero byte within a word or first byte equal between two words. - index_last_zero: return index of last zero byte within a word. - index_last_eq: return index of last byte different between two words. Co-authored-by: Richard Henderson --- sysdeps/generic/string-extbyte.h | 37 +++++++++ sysdeps/generic/string-fza.h | 102 +++++++++++++++++++++++ sysdeps/generic/string-fzb.h | 49 +++++++++++ sysdeps/generic/string-fzi.h | 135 +++++++++++++++++++++++++++++++ sysdeps/generic/string-maskoff.h | 9 +-- 5 files changed, 327 insertions(+), 5 deletions(-) create mode 100644 sysdeps/generic/string-extbyte.h create mode 100644 sysdeps/generic/string-fza.h create mode 100644 sysdeps/generic/string-fzb.h create mode 100644 sysdeps/generic/string-fzi.h diff --git a/sysdeps/generic/string-extbyte.h b/sysdeps/generic/string-extbyte.h new file mode 100644 index 0000000000..38b4674dca --- /dev/null +++ b/sysdeps/generic/string-extbyte.h @@ -0,0 +1,37 @@ +/* Extract by from memory word. Generic C version. + Copyright (C) 2023 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#ifndef _STRING_EXTBYTE_H +#define _STRING_EXTBYTE_H 1 + +#include +#include +#include + +/* Extract the byte at index IDX from word X, with index 0 being the + least significant byte. */ +static __always_inline unsigned char +extractbyte (op_t x, unsigned int idx) +{ + if (__BYTE_ORDER == __LITTLE_ENDIAN) + return x >> (idx * CHAR_BIT); + else + return x >> (sizeof (x) - 1 - idx) * CHAR_BIT; +} + +#endif /* _STRING_EXTBYTE_H */ diff --git a/sysdeps/generic/string-fza.h b/sysdeps/generic/string-fza.h new file mode 100644 index 0000000000..1ec51f7f3c --- /dev/null +++ b/sysdeps/generic/string-fza.h @@ -0,0 +1,102 @@ +/* Basic zero byte detection. Generic C version. + Copyright (C) 2023 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#ifndef _STRING_FZA_H +#define _STRING_FZA_H 1 + +#include +#include +#include + +/* This function returns non-zero if any byte in X is zero. + More specifically, at least one bit set within the least significant + byte that was zero; other bytes within the word are indeterminate. */ +static __always_inline op_t +find_zero_low (op_t x) +{ + /* This expression comes from + https://graphics.stanford.edu/~seander/bithacks.html#ZeroInWord + Subtracting 1 sets 0x80 in a byte that was 0; anding ~x clears + 0x80 in a byte that was >= 128; anding 0x80 isolates that test bit. */ + op_t lsb = repeat_bytes (0x01); + op_t msb = repeat_bytes (0x80); + return (x - lsb) & ~x & msb; +} + +/* This function returns at least one bit set within every byte of X that + is zero. The result is exact in that, unlike find_zero_low, all bytes + are determinate. This is usually used for finding the index of the + most significant byte that was zero. */ +static __always_inline op_t +find_zero_all (op_t x) +{ + /* For each byte, find not-zero by + (0) And 0x7f so that we cannot carry between bytes, + (1) Add 0x7f so that non-zero carries into 0x80, + (2) Or in the original byte (which might have had 0x80 set). + Then invert and mask such that 0x80 is set iff that byte was zero. */ + op_t m = repeat_bytes (0x7f); + return ~(((x & m) + m) | x | m); +} + +/* With similar caveats, identify bytes that are equal between X1 and X2. */ +static __always_inline op_t +find_eq_low (op_t x1, op_t x2) +{ + return find_zero_low (x1 ^ x2); +} + +static __always_inline op_t +find_eq_all (op_t x1, op_t x2) +{ + return find_zero_all (x1 ^ x2); +} + +/* With similar caveats, identify zero bytes in X1 and bytes that are + equal between in X1 and X2. */ +static __always_inline op_t +find_zero_eq_low (op_t x1, op_t x2) +{ + return find_zero_low (x1) | find_zero_low (x1 ^ x2); +} + +static __always_inline op_t +find_zero_eq_all (op_t x1, op_t x2) +{ + return find_zero_all (x1) | find_zero_all (x1 ^ x2); +} + +/* With similar caveats, identify zero bytes in X1 and bytes that are + not equal between in X1 and X2. */ +static __always_inline op_t +find_zero_ne_low (op_t x1, op_t x2) +{ + return (~find_zero_eq_low (x1, x2)) + 1; +} + +static __always_inline op_t +find_zero_ne_all (op_t x1, op_t x2) +{ + op_t m = repeat_bytes (0x7f); + op_t eq = x1 ^ x2; + op_t nz1 = ((x1 & m) + m) | x1; + op_t ne2 = ((eq & m) + m) | eq; + return (ne2 | ~nz1) & ~m; +} + +#endif /* _STRING_FZA_H */ diff --git a/sysdeps/generic/string-fzb.h b/sysdeps/generic/string-fzb.h new file mode 100644 index 0000000000..42de500d67 --- /dev/null +++ b/sysdeps/generic/string-fzb.h @@ -0,0 +1,49 @@ +/* Zero byte detection, boolean. Generic C version. + Copyright (C) 2023 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#ifndef _STRING_FZB_H +#define _STRING_FZB_H 1 + +#include +#include + +/* Determine if any byte within X is zero. This is a pure boolean test. */ + +static __always_inline _Bool +has_zero (op_t x) +{ + return find_zero_low (x) != 0; +} + +/* Likewise, but for byte equality between X1 and X2. */ + +static __always_inline _Bool +has_eq (op_t x1, op_t x2) +{ + return find_eq_low (x1, x2) != 0; +} + +/* Likewise, but for zeros in X1 and equal bytes between X1 and X2. */ + +static __always_inline _Bool +has_zero_eq (op_t x1, op_t x2) +{ + return find_zero_eq_low (x1, x2); +} + +#endif /* _STRING_FZB_H */ diff --git a/sysdeps/generic/string-fzi.h b/sysdeps/generic/string-fzi.h new file mode 100644 index 0000000000..b1fd4d34b3 --- /dev/null +++ b/sysdeps/generic/string-fzi.h @@ -0,0 +1,135 @@ +/* Zero byte detection; indexes. Generic C version. + Copyright (C) 2023 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#ifndef _STRING_FZI_H +#define _STRING_FZI_H 1 + +#include +#include +#include + +static __always_inline int +clz (op_t c) +{ + if (sizeof (op_t) == sizeof (unsigned long)) + return __builtin_clzl (c); + else + return __builtin_clzll (c); +} + +static __always_inline int +ctz (op_t c) +{ + if (sizeof (op_t) == sizeof (unsigned long)) + return __builtin_ctzl (c); + else + return __builtin_ctzll (c); +} + +/* A subroutine for the index_zero functions. Given a test word C, return + the (memory order) index of the first byte (in memory order) that is + non-zero. */ +static __always_inline unsigned int +index_first (op_t c) +{ + int r; + if (__BYTE_ORDER == __LITTLE_ENDIAN) + r = ctz (c); + else + r = clz (c); + return r / CHAR_BIT; +} + +/* Similarly, but return the (memory order) index of the last byte that is + non-zero. */ +static __always_inline unsigned int +index_last (op_t c) +{ + int r; + if (__BYTE_ORDER == __LITTLE_ENDIAN) + r = clz (c); + else + r = ctz (c); + return sizeof (op_t) - 1 - (r / CHAR_BIT); +} + +/* Given a word X that is known to contain a zero byte, return the index of + the first such within the word in memory order. */ +static __always_inline unsigned int +index_first_zero (op_t x) +{ + if (__BYTE_ORDER == __LITTLE_ENDIAN) + x = find_zero_low (x); + else + x = find_zero_all (x); + return index_first (x); +} + +/* Similarly, but perform the search for byte equality between X1 and X2. */ +static __always_inline unsigned int +index_first_eq (op_t x1, op_t x2) +{ + if (__BYTE_ORDER == __LITTLE_ENDIAN) + x1 = find_eq_low (x1, x2); + else + x1 = find_eq_all (x1, x2); + return index_first (x1); +} + +/* Similarly, but perform the search for zero within X1 or equality between + X1 and X2. */ +static __always_inline unsigned int +index_first_zero_eq (op_t x1, op_t x2) +{ + if (__BYTE_ORDER == __LITTLE_ENDIAN) + x1 = find_zero_eq_low (x1, x2); + else + x1 = find_zero_eq_all (x1, x2); + return index_first (x1); +} + +/* Similarly, but perform the search for zero within X1 or inequality between + X1 and X2. */ +static __always_inline unsigned int +index_first_zero_ne (op_t x1, op_t x2) +{ + if (__BYTE_ORDER == __LITTLE_ENDIAN) + x1 = find_zero_ne_low (x1, x2); + else + x1 = find_zero_ne_all (x1, x2); + return index_first (x1); +} + +/* Similarly, but search for the last zero within X. */ +static __always_inline unsigned int +index_last_zero (op_t x) +{ + if (__BYTE_ORDER == __LITTLE_ENDIAN) + x = find_zero_all (x); + else + x = find_zero_low (x); + return index_last (x); +} + +static __always_inline unsigned int +index_last_eq (op_t x1, op_t x2) +{ + return index_last_zero (x1 ^ x2); +} + +#endif /* STRING_FZI_H */ diff --git a/sysdeps/generic/string-maskoff.h b/sysdeps/generic/string-maskoff.h index 6ad4e0b6f9..16d3cc2ddb 100644 --- a/sysdeps/generic/string-maskoff.h +++ b/sysdeps/generic/string-maskoff.h @@ -23,7 +23,6 @@ #include #include #include -#include /* Provide a mask based on the pointer alignment that sets up non-zero bytes before the beginning of the word. It is used to mask off @@ -44,12 +43,12 @@ create_mask (uintptr_t i) /* Return the mask WORD shifted based on S_INT address value, to ignore values not presented in the aligned word read. */ static __always_inline op_t -check_mask (op_t word, uintptr_t s_int) +check_mask (op_t word, uintptr_t s) { if (__BYTE_ORDER == __LITTLE_ENDIAN) - return word >> (CHAR_BIT * (s_int % sizeof (s_int))); + return word >> (CHAR_BIT * (s % sizeof (op_t))); else - return word << (CHAR_BIT * (s_int % sizeof (s_int))); + return word << (CHAR_BIT * (s % sizeof (op_t))); } /* Setup an word with each byte being c_in. For instance, on a 64 bits @@ -79,7 +78,7 @@ highbit_mask (op_t m) static __always_inline op_t * word_containing (char const *p) { - return (op_t *) ((uintptr_t) p & -sizeof(p)); + return (op_t *) ((op_t) p & -sizeof(p)); } #endif /* _STRING_MASKOFF_H */