From patchwork Wed Dec 6 19:43:27 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 120884 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp7468368qgn; Wed, 6 Dec 2017 11:44:00 -0800 (PST) X-Google-Smtp-Source: AGs4zMbojnWhUN44cmfDz44zoD+Z5/v2FBAqXzyNOPtQcgwWN//iKGarW7oCdZUwfEXToAloXfvZ X-Received: by 10.99.134.73 with SMTP id x70mr22617437pgd.130.1512589440576; Wed, 06 Dec 2017 11:44:00 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512589440; cv=none; d=google.com; s=arc-20160816; b=Tf3QcASAGlKZs4pyTtecbPARvNyrlL8hdWGq+QQ0SXeURWSlj5LRSWWP59UARa/C8o TTsePjnFn8yMWpE073OS4h60DIVos1qtOjKJPQURY13IAKUqmySsTX0g/Ypiqas6rfq9 mNvcS0cx1OCxtLZsurFSVBTva3rpgBLA1FvvqPp4ucVECda9GncX1K7+xQxlAKp/8fq1 mTyshM1luD+nn/xfA1EdRV8WaeoQ7l2zcnnroMyvauHzlGbbcYguLgr2JHZwUpqMxdyJ bak180ghi+U0V6uX63qJdsnqTzROJMj46w5+MLT3JTjPtuBRK+51BwMeehktnXiBIJSr Cj9g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=7+wQwKdaD7iP7HvwTiQ8RMauRJIxojqc+6drRtiScsg=; b=yQ3FO1w4xN7LOUGXwM5BQFodSPNd4xvTLqbGFWgtwJVNrKeOs/s5UpoZMls2jlZypp 1j8zOx+/cYiRazHjx7svAKQldpKza4mpACTFUJqzu35xgDpIwFObuKMo5PUx9NbdHPcM 2UK8MII4GgNnc0eYXtYJm+fVT2MN25NdfxV1OgNrk4X8FN70UIWjwsCoQNiCfbcEOfZI nphUn5xOM9ZsF2DVh47aMgK9xHKkq2pJ8grwGB6umH5UFt76RJI86cXy/6iuhNbW2PzJ WZzVNf8QCipO3w5b2xz5Rdp6pCP85bkxU7QhAb+eX8mxQYTHFFsEd5iAQ9oMa8lORVUo Z1aw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=PQG3okOK; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 31si2517485plz.160.2017.12.06.11.44.00; Wed, 06 Dec 2017 11:44:00 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=PQG3okOK; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752018AbdLFTn7 (ORCPT + 4 others); Wed, 6 Dec 2017 14:43:59 -0500 Received: from mail-wm0-f67.google.com ([74.125.82.67]:42267 "EHLO mail-wm0-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751208AbdLFTn5 (ORCPT ); Wed, 6 Dec 2017 14:43:57 -0500 Received: by mail-wm0-f67.google.com with SMTP id l141so9220151wmg.1 for ; Wed, 06 Dec 2017 11:43:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=+ZWLqI1QNuPJLTnEYdZnb8HCsE/psTZiCZMJQrLjBtI=; b=PQG3okOK561VwmAHj914T+IRbnBrTQK6kocg+H7s5pzfo1g9ET6ESji4NCFZXRu/If 7JZARlObU7uoyDNaU8vNdheqFc0covDndj3si6Yf551lvmZ9+29ZPN9R5XrRPj/oQxzz 3GpSPEhnEYIucgQDuNeKypEQIKLK/WXhBCDIc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=+ZWLqI1QNuPJLTnEYdZnb8HCsE/psTZiCZMJQrLjBtI=; b=F0+nbU6rN4R7FxUDa1rYZ96B7T2CsOYX27gl12vH5af5P9PjzylPgq9jIRuamNsWwo agDxCONqboPbXnO4ahvlepxII15Jh+BYQV9sk6L81ae/b8Bj2tYEKg2TfSCvGIWdZc9w Zc2FpBh7rHJg46VOK8lvJ/lEOEx3tyHdUQvs4SAC4/rccRkjJBLDdDvNZJN6wXDokJ8e aJrgf9KJJErQkzHJbvSsLBCVVZISZeSN0MjZqOip+YpAjo4J//II7352NvMK/qTABZVz t3MmB27fpjkIvzZmpfshJuhkNsaE/dW9licP3EWo0mOJZPvw4nxZcjhYLR5tGpps5tUY 2lDA== X-Gm-Message-State: AKGB3mJOANB+aD4SI9hRvDdR8iVijqRKND9ZpciXQEfCeEt/OSxXOSaB 3oRkWhd3yBclDmqBikbzZqJSzg== X-Received: by 10.28.193.66 with SMTP id r63mr12261173wmf.18.1512589435877; Wed, 06 Dec 2017 11:43:55 -0800 (PST) Received: from localhost.localdomain ([105.150.171.234]) by smtp.gmail.com with ESMTPSA id b66sm3596594wmh.32.2017.12.06.11.43.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 06 Dec 2017 11:43:55 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v3 01/20] crypto: testmgr - add a new test case for CRC-T10DIF Date: Wed, 6 Dec 2017 19:43:27 +0000 Message-Id: <20171206194346.24393-2-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171206194346.24393-1-ard.biesheuvel@linaro.org> References: <20171206194346.24393-1-ard.biesheuvel@linaro.org> Sender: linux-rt-users-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rt-users@vger.kernel.org In order to be able to test yield support under preempt, add a test vector for CRC-T10DIF that is long enough to take multiple iterations (and thus possible preemption between them) of the primary loop of the accelerated x86 and arm64 implementations. Signed-off-by: Ard Biesheuvel --- crypto/testmgr.h | 259 ++++++++++++++++++++ 1 file changed, 259 insertions(+) -- 2.11.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/crypto/testmgr.h b/crypto/testmgr.h index a714b6293959..0c849aec161d 100644 --- a/crypto/testmgr.h +++ b/crypto/testmgr.h @@ -1494,6 +1494,265 @@ static const struct hash_testvec crct10dif_tv_template[] = { .digest = (u8 *)(u16 []){ 0x44c6 }, .np = 4, .tap = { 1, 255, 57, 6 }, + }, { + .plaintext = "\x6e\x05\x79\x10\xa7\x1b\xb2\x49" + "\xe0\x54\xeb\x82\x19\x8d\x24\xbb" + "\x2f\xc6\x5d\xf4\x68\xff\x96\x0a" + "\xa1\x38\xcf\x43\xda\x71\x08\x7c" + "\x13\xaa\x1e\xb5\x4c\xe3\x57\xee" + "\x85\x1c\x90\x27\xbe\x32\xc9\x60" + "\xf7\x6b\x02\x99\x0d\xa4\x3b\xd2" + "\x46\xdd\x74\x0b\x7f\x16\xad\x21" + "\xb8\x4f\xe6\x5a\xf1\x88\x1f\x93" + "\x2a\xc1\x35\xcc\x63\xfa\x6e\x05" + "\x9c\x10\xa7\x3e\xd5\x49\xe0\x77" + "\x0e\x82\x19\xb0\x24\xbb\x52\xe9" + "\x5d\xf4\x8b\x22\x96\x2d\xc4\x38" + "\xcf\x66\xfd\x71\x08\x9f\x13\xaa" + "\x41\xd8\x4c\xe3\x7a\x11\x85\x1c" + "\xb3\x27\xbe\x55\xec\x60\xf7\x8e" + "\x02\x99\x30\xc7\x3b\xd2\x69\x00" + "\x74\x0b\xa2\x16\xad\x44\xdb\x4f" + "\xe6\x7d\x14\x88\x1f\xb6\x2a\xc1" + "\x58\xef\x63\xfa\x91\x05\x9c\x33" + "\xca\x3e\xd5\x6c\x03\x77\x0e\xa5" + "\x19\xb0\x47\xde\x52\xe9\x80\x17" + "\x8b\x22\xb9\x2d\xc4\x5b\xf2\x66" + "\xfd\x94\x08\x9f\x36\xcd\x41\xd8" + "\x6f\x06\x7a\x11\xa8\x1c\xb3\x4a" + "\xe1\x55\xec\x83\x1a\x8e\x25\xbc" + "\x30\xc7\x5e\xf5\x69\x00\x97\x0b" + "\xa2\x39\xd0\x44\xdb\x72\x09\x7d" + "\x14\xab\x1f\xb6\x4d\xe4\x58\xef" + "\x86\x1d\x91\x28\xbf\x33\xca\x61" + "\xf8\x6c\x03\x9a\x0e\xa5\x3c\xd3" + "\x47\xde\x75\x0c\x80\x17\xae\x22" + "\xb9\x50\xe7\x5b\xf2\x89\x20\x94" + "\x2b\xc2\x36\xcd\x64\xfb\x6f\x06" + "\x9d\x11\xa8\x3f\xd6\x4a\xe1\x78" + "\x0f\x83\x1a\xb1\x25\xbc\x53\xea" + "\x5e\xf5\x8c\x00\x97\x2e\xc5\x39" + "\xd0\x67\xfe\x72\x09\xa0\x14\xab" + "\x42\xd9\x4d\xe4\x7b\x12\x86\x1d" + "\xb4\x28\xbf\x56\xed\x61\xf8\x8f" + "\x03\x9a\x31\xc8\x3c\xd3\x6a\x01" + "\x75\x0c\xa3\x17\xae\x45\xdc\x50" + "\xe7\x7e\x15\x89\x20\xb7\x2b\xc2" + "\x59\xf0\x64\xfb\x92\x06\x9d\x34" + "\xcb\x3f\xd6\x6d\x04\x78\x0f\xa6" + "\x1a\xb1\x48\xdf\x53\xea\x81\x18" + "\x8c\x23\xba\x2e\xc5\x5c\xf3\x67" + "\xfe\x95\x09\xa0\x37\xce\x42\xd9" + "\x70\x07\x7b\x12\xa9\x1d\xb4\x4b" + "\xe2\x56\xed\x84\x1b\x8f\x26\xbd" + "\x31\xc8\x5f\xf6\x6a\x01\x98\x0c" + "\xa3\x3a\xd1\x45\xdc\x73\x0a\x7e" + "\x15\xac\x20\xb7\x4e\xe5\x59\xf0" + "\x87\x1e\x92\x29\xc0\x34\xcb\x62" + "\xf9\x6d\x04\x9b\x0f\xa6\x3d\xd4" + "\x48\xdf\x76\x0d\x81\x18\xaf\x23" + "\xba\x51\xe8\x5c\xf3\x8a\x21\x95" + "\x2c\xc3\x37\xce\x65\xfc\x70\x07" + "\x9e\x12\xa9\x40\xd7\x4b\xe2\x79" + "\x10\x84\x1b\xb2\x26\xbd\x54\xeb" + "\x5f\xf6\x8d\x01\x98\x2f\xc6\x3a" + "\xd1\x68\xff\x73\x0a\xa1\x15\xac" + "\x43\xda\x4e\xe5\x7c\x13\x87\x1e" + "\xb5\x29\xc0\x57\xee\x62\xf9\x90" + "\x04\x9b\x32\xc9\x3d\xd4\x6b\x02" + "\x76\x0d\xa4\x18\xaf\x46\xdd\x51" + "\xe8\x7f\x16\x8a\x21\xb8\x2c\xc3" + "\x5a\xf1\x65\xfc\x93\x07\x9e\x35" + "\xcc\x40\xd7\x6e\x05\x79\x10\xa7" + "\x1b\xb2\x49\xe0\x54\xeb\x82\x19" + "\x8d\x24\xbb\x2f\xc6\x5d\xf4\x68" + "\xff\x96\x0a\xa1\x38\xcf\x43\xda" + "\x71\x08\x7c\x13\xaa\x1e\xb5\x4c" + "\xe3\x57\xee\x85\x1c\x90\x27\xbe" + "\x32\xc9\x60\xf7\x6b\x02\x99\x0d" + "\xa4\x3b\xd2\x46\xdd\x74\x0b\x7f" + "\x16\xad\x21\xb8\x4f\xe6\x5a\xf1" + "\x88\x1f\x93\x2a\xc1\x35\xcc\x63" + "\xfa\x6e\x05\x9c\x10\xa7\x3e\xd5" + "\x49\xe0\x77\x0e\x82\x19\xb0\x24" + "\xbb\x52\xe9\x5d\xf4\x8b\x22\x96" + "\x2d\xc4\x38\xcf\x66\xfd\x71\x08" + "\x9f\x13\xaa\x41\xd8\x4c\xe3\x7a" + "\x11\x85\x1c\xb3\x27\xbe\x55\xec" + "\x60\xf7\x8e\x02\x99\x30\xc7\x3b" + "\xd2\x69\x00\x74\x0b\xa2\x16\xad" + "\x44\xdb\x4f\xe6\x7d\x14\x88\x1f" + "\xb6\x2a\xc1\x58\xef\x63\xfa\x91" + "\x05\x9c\x33\xca\x3e\xd5\x6c\x03" + "\x77\x0e\xa5\x19\xb0\x47\xde\x52" + "\xe9\x80\x17\x8b\x22\xb9\x2d\xc4" + "\x5b\xf2\x66\xfd\x94\x08\x9f\x36" + "\xcd\x41\xd8\x6f\x06\x7a\x11\xa8" + "\x1c\xb3\x4a\xe1\x55\xec\x83\x1a" + "\x8e\x25\xbc\x30\xc7\x5e\xf5\x69" + "\x00\x97\x0b\xa2\x39\xd0\x44\xdb" + "\x72\x09\x7d\x14\xab\x1f\xb6\x4d" + "\xe4\x58\xef\x86\x1d\x91\x28\xbf" + "\x33\xca\x61\xf8\x6c\x03\x9a\x0e" + "\xa5\x3c\xd3\x47\xde\x75\x0c\x80" + "\x17\xae\x22\xb9\x50\xe7\x5b\xf2" + "\x89\x20\x94\x2b\xc2\x36\xcd\x64" + "\xfb\x6f\x06\x9d\x11\xa8\x3f\xd6" + "\x4a\xe1\x78\x0f\x83\x1a\xb1\x25" + "\xbc\x53\xea\x5e\xf5\x8c\x00\x97" + "\x2e\xc5\x39\xd0\x67\xfe\x72\x09" + "\xa0\x14\xab\x42\xd9\x4d\xe4\x7b" + "\x12\x86\x1d\xb4\x28\xbf\x56\xed" + "\x61\xf8\x8f\x03\x9a\x31\xc8\x3c" + "\xd3\x6a\x01\x75\x0c\xa3\x17\xae" + "\x45\xdc\x50\xe7\x7e\x15\x89\x20" + "\xb7\x2b\xc2\x59\xf0\x64\xfb\x92" + "\x06\x9d\x34\xcb\x3f\xd6\x6d\x04" + "\x78\x0f\xa6\x1a\xb1\x48\xdf\x53" + "\xea\x81\x18\x8c\x23\xba\x2e\xc5" + "\x5c\xf3\x67\xfe\x95\x09\xa0\x37" + "\xce\x42\xd9\x70\x07\x7b\x12\xa9" + "\x1d\xb4\x4b\xe2\x56\xed\x84\x1b" + "\x8f\x26\xbd\x31\xc8\x5f\xf6\x6a" + "\x01\x98\x0c\xa3\x3a\xd1\x45\xdc" + "\x73\x0a\x7e\x15\xac\x20\xb7\x4e" + "\xe5\x59\xf0\x87\x1e\x92\x29\xc0" + "\x34\xcb\x62\xf9\x6d\x04\x9b\x0f" + "\xa6\x3d\xd4\x48\xdf\x76\x0d\x81" + "\x18\xaf\x23\xba\x51\xe8\x5c\xf3" + "\x8a\x21\x95\x2c\xc3\x37\xce\x65" + "\xfc\x70\x07\x9e\x12\xa9\x40\xd7" + "\x4b\xe2\x79\x10\x84\x1b\xb2\x26" + "\xbd\x54\xeb\x5f\xf6\x8d\x01\x98" + "\x2f\xc6\x3a\xd1\x68\xff\x73\x0a" + "\xa1\x15\xac\x43\xda\x4e\xe5\x7c" + "\x13\x87\x1e\xb5\x29\xc0\x57\xee" + "\x62\xf9\x90\x04\x9b\x32\xc9\x3d" + "\xd4\x6b\x02\x76\x0d\xa4\x18\xaf" + "\x46\xdd\x51\xe8\x7f\x16\x8a\x21" + "\xb8\x2c\xc3\x5a\xf1\x65\xfc\x93" + "\x07\x9e\x35\xcc\x40\xd7\x6e\x05" + "\x79\x10\xa7\x1b\xb2\x49\xe0\x54" + "\xeb\x82\x19\x8d\x24\xbb\x2f\xc6" + "\x5d\xf4\x68\xff\x96\x0a\xa1\x38" + "\xcf\x43\xda\x71\x08\x7c\x13\xaa" + "\x1e\xb5\x4c\xe3\x57\xee\x85\x1c" + "\x90\x27\xbe\x32\xc9\x60\xf7\x6b" + "\x02\x99\x0d\xa4\x3b\xd2\x46\xdd" + "\x74\x0b\x7f\x16\xad\x21\xb8\x4f" + "\xe6\x5a\xf1\x88\x1f\x93\x2a\xc1" + "\x35\xcc\x63\xfa\x6e\x05\x9c\x10" + "\xa7\x3e\xd5\x49\xe0\x77\x0e\x82" + "\x19\xb0\x24\xbb\x52\xe9\x5d\xf4" + "\x8b\x22\x96\x2d\xc4\x38\xcf\x66" + "\xfd\x71\x08\x9f\x13\xaa\x41\xd8" + "\x4c\xe3\x7a\x11\x85\x1c\xb3\x27" + "\xbe\x55\xec\x60\xf7\x8e\x02\x99" + "\x30\xc7\x3b\xd2\x69\x00\x74\x0b" + "\xa2\x16\xad\x44\xdb\x4f\xe6\x7d" + "\x14\x88\x1f\xb6\x2a\xc1\x58\xef" + "\x63\xfa\x91\x05\x9c\x33\xca\x3e" + "\xd5\x6c\x03\x77\x0e\xa5\x19\xb0" + "\x47\xde\x52\xe9\x80\x17\x8b\x22" + "\xb9\x2d\xc4\x5b\xf2\x66\xfd\x94" + "\x08\x9f\x36\xcd\x41\xd8\x6f\x06" + "\x7a\x11\xa8\x1c\xb3\x4a\xe1\x55" + "\xec\x83\x1a\x8e\x25\xbc\x30\xc7" + "\x5e\xf5\x69\x00\x97\x0b\xa2\x39" + "\xd0\x44\xdb\x72\x09\x7d\x14\xab" + "\x1f\xb6\x4d\xe4\x58\xef\x86\x1d" + "\x91\x28\xbf\x33\xca\x61\xf8\x6c" + "\x03\x9a\x0e\xa5\x3c\xd3\x47\xde" + "\x75\x0c\x80\x17\xae\x22\xb9\x50" + "\xe7\x5b\xf2\x89\x20\x94\x2b\xc2" + "\x36\xcd\x64\xfb\x6f\x06\x9d\x11" + "\xa8\x3f\xd6\x4a\xe1\x78\x0f\x83" + "\x1a\xb1\x25\xbc\x53\xea\x5e\xf5" + "\x8c\x00\x97\x2e\xc5\x39\xd0\x67" + "\xfe\x72\x09\xa0\x14\xab\x42\xd9" + "\x4d\xe4\x7b\x12\x86\x1d\xb4\x28" + "\xbf\x56\xed\x61\xf8\x8f\x03\x9a" + "\x31\xc8\x3c\xd3\x6a\x01\x75\x0c" + "\xa3\x17\xae\x45\xdc\x50\xe7\x7e" + "\x15\x89\x20\xb7\x2b\xc2\x59\xf0" + "\x64\xfb\x92\x06\x9d\x34\xcb\x3f" + "\xd6\x6d\x04\x78\x0f\xa6\x1a\xb1" + "\x48\xdf\x53\xea\x81\x18\x8c\x23" + "\xba\x2e\xc5\x5c\xf3\x67\xfe\x95" + "\x09\xa0\x37\xce\x42\xd9\x70\x07" + "\x7b\x12\xa9\x1d\xb4\x4b\xe2\x56" + "\xed\x84\x1b\x8f\x26\xbd\x31\xc8" + "\x5f\xf6\x6a\x01\x98\x0c\xa3\x3a" + "\xd1\x45\xdc\x73\x0a\x7e\x15\xac" + "\x20\xb7\x4e\xe5\x59\xf0\x87\x1e" + "\x92\x29\xc0\x34\xcb\x62\xf9\x6d" + "\x04\x9b\x0f\xa6\x3d\xd4\x48\xdf" + "\x76\x0d\x81\x18\xaf\x23\xba\x51" + "\xe8\x5c\xf3\x8a\x21\x95\x2c\xc3" + "\x37\xce\x65\xfc\x70\x07\x9e\x12" + "\xa9\x40\xd7\x4b\xe2\x79\x10\x84" + "\x1b\xb2\x26\xbd\x54\xeb\x5f\xf6" + "\x8d\x01\x98\x2f\xc6\x3a\xd1\x68" + "\xff\x73\x0a\xa1\x15\xac\x43\xda" + "\x4e\xe5\x7c\x13\x87\x1e\xb5\x29" + "\xc0\x57\xee\x62\xf9\x90\x04\x9b" + "\x32\xc9\x3d\xd4\x6b\x02\x76\x0d" + "\xa4\x18\xaf\x46\xdd\x51\xe8\x7f" + "\x16\x8a\x21\xb8\x2c\xc3\x5a\xf1" + "\x65\xfc\x93\x07\x9e\x35\xcc\x40" + "\xd7\x6e\x05\x79\x10\xa7\x1b\xb2" + "\x49\xe0\x54\xeb\x82\x19\x8d\x24" + "\xbb\x2f\xc6\x5d\xf4\x68\xff\x96" + "\x0a\xa1\x38\xcf\x43\xda\x71\x08" + "\x7c\x13\xaa\x1e\xb5\x4c\xe3\x57" + "\xee\x85\x1c\x90\x27\xbe\x32\xc9" + "\x60\xf7\x6b\x02\x99\x0d\xa4\x3b" + "\xd2\x46\xdd\x74\x0b\x7f\x16\xad" + "\x21\xb8\x4f\xe6\x5a\xf1\x88\x1f" + "\x93\x2a\xc1\x35\xcc\x63\xfa\x6e" + "\x05\x9c\x10\xa7\x3e\xd5\x49\xe0" + "\x77\x0e\x82\x19\xb0\x24\xbb\x52" + "\xe9\x5d\xf4\x8b\x22\x96\x2d\xc4" + "\x38\xcf\x66\xfd\x71\x08\x9f\x13" + "\xaa\x41\xd8\x4c\xe3\x7a\x11\x85" + "\x1c\xb3\x27\xbe\x55\xec\x60\xf7" + "\x8e\x02\x99\x30\xc7\x3b\xd2\x69" + "\x00\x74\x0b\xa2\x16\xad\x44\xdb" + "\x4f\xe6\x7d\x14\x88\x1f\xb6\x2a" + "\xc1\x58\xef\x63\xfa\x91\x05\x9c" + "\x33\xca\x3e\xd5\x6c\x03\x77\x0e" + "\xa5\x19\xb0\x47\xde\x52\xe9\x80" + "\x17\x8b\x22\xb9\x2d\xc4\x5b\xf2" + "\x66\xfd\x94\x08\x9f\x36\xcd\x41" + "\xd8\x6f\x06\x7a\x11\xa8\x1c\xb3" + "\x4a\xe1\x55\xec\x83\x1a\x8e\x25" + "\xbc\x30\xc7\x5e\xf5\x69\x00\x97" + "\x0b\xa2\x39\xd0\x44\xdb\x72\x09" + "\x7d\x14\xab\x1f\xb6\x4d\xe4\x58" + "\xef\x86\x1d\x91\x28\xbf\x33\xca" + "\x61\xf8\x6c\x03\x9a\x0e\xa5\x3c" + "\xd3\x47\xde\x75\x0c\x80\x17\xae" + "\x22\xb9\x50\xe7\x5b\xf2\x89\x20" + "\x94\x2b\xc2\x36\xcd\x64\xfb\x6f" + "\x06\x9d\x11\xa8\x3f\xd6\x4a\xe1" + "\x78\x0f\x83\x1a\xb1\x25\xbc\x53" + "\xea\x5e\xf5\x8c\x00\x97\x2e\xc5" + "\x39\xd0\x67\xfe\x72\x09\xa0\x14" + "\xab\x42\xd9\x4d\xe4\x7b\x12\x86" + "\x1d\xb4\x28\xbf\x56\xed\x61\xf8" + "\x8f\x03\x9a\x31\xc8\x3c\xd3\x6a" + "\x01\x75\x0c\xa3\x17\xae\x45\xdc" + "\x50\xe7\x7e\x15\x89\x20\xb7\x2b" + "\xc2\x59\xf0\x64\xfb\x92\x06\x9d" + "\x34\xcb\x3f\xd6\x6d\x04\x78\x0f" + "\xa6\x1a\xb1\x48\xdf\x53\xea\x81" + "\x18\x8c\x23\xba\x2e\xc5\x5c\xf3" + "\x67\xfe\x95\x09\xa0\x37\xce\x42" + "\xd9\x70\x07\x7b\x12\xa9\x1d\xb4" + "\x4b\xe2\x56\xed\x84\x1b\x8f\x26" + "\xbd\x31\xc8\x5f\xf6\x6a\x01\x98", + .psize = 2048, + .digest = (u8 *)(u16 []){ 0x23ca }, } }; From patchwork Wed Dec 6 19:43:32 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 120890 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp7468757qgn; Wed, 6 Dec 2017 11:44:27 -0800 (PST) X-Google-Smtp-Source: AGs4zMbmHnczmB6tSq1KkBPXb2nGtGjjjwMohy9RM9jNfRdDyL5ahLrGBMuj5NiBagIoPr7gFu9E X-Received: by 10.98.103.156 with SMTP id t28mr3775429pfj.234.1512589467132; Wed, 06 Dec 2017 11:44:27 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512589467; cv=none; d=google.com; s=arc-20160816; b=eKx1atNqM2pBSXSWInTpB2L+qvu1c3dVwo5DfZdnb99mrji7j58LHJ8+CqksRyRM8X Ect+KXGHm+HG8NJlnRIcRo1asdaYzCpKmg5HajebyDyzMkxVUt4A0O7X7TgKM12lAf1k 7NyQkNxcwp1luWf1CIxhBmX4vnCJTQ/VRc9ir6y0RR4zxNDhFaey/1l3S1Ea1x2Z2Rnx MitG0DcLxy/mokPjZhIpO0n/Rw1qCFvQpulUeohIJHOgQJdBv01f91fXzHMQRKt8wGYc 0fsYIxSzdRbOM2/SZ2zs4UdCAsBybOaR6Yrl+pLo+yMzGpt4jcqtOLNZ5a/FhO2QgwHa er0w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=0V94cTWdPJjrvAtc7N7YgjZLr0YNwq7XjJKr2QZYiB0=; b=SmXiAgziMmK+X99v+voiM4oyLBMobIJ3E24o7P8mdPwgSM/hUIeIxafMYbPIWmuTv/ Om2As1JeQ/zpiFsFMWIKU44Av0HTmnf/XIMmxKqtma5Wr904HWENwnWTbE/1ohw3xpyD NHvmQQq/mHP+xHwHLppaO9o7EAwnXRyJ/0jVQ/dsDgiXQNKUWxhXPvbBnzRLt56B5zCD hT2K4n5NawijahWDyQ+rt4LYgEiDAFzU5LItQeUchg0aT5ODkDPpBXPZcFeCjP2CEjQv CgDBQjPMIyKmVIf3fNq475V/bJ3oZ3A5hS/jkY/Vl80eQozDWTvF8U+pILHMwmbwpe3o J5yQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=dvQVG619; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id i16si2382524pgv.496.2017.12.06.11.44.26; Wed, 06 Dec 2017 11:44:27 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=dvQVG619; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752221AbdLFToT (ORCPT + 4 others); Wed, 6 Dec 2017 14:44:19 -0500 Received: from mail-wr0-f194.google.com ([209.85.128.194]:36245 "EHLO mail-wr0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752126AbdLFToM (ORCPT ); Wed, 6 Dec 2017 14:44:12 -0500 Received: by mail-wr0-f194.google.com with SMTP id v105so5120560wrc.3 for ; Wed, 06 Dec 2017 11:44:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=mPx5iqqZoQYjOqcGi3EBqUtnRcsEf9rWIzMkmnPsgJU=; b=dvQVG619pbWv5AVG7KCAkIzMot6oXpDeCgahZ2iSLXdFWCriN49JowzYFwVMhF+B/s KOvgY0d9q+xSXCCxFT+daGq2eZQaP+QefEXINtN3PLvdKHM4l5XP9MxihHm/LDaMUj/C aOebzxQ8ympZ/S3Uo8f/2yoLwSFPXW2ImJ/9o= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=mPx5iqqZoQYjOqcGi3EBqUtnRcsEf9rWIzMkmnPsgJU=; b=PiZxAOXrzxVGtxFKHTqhM55Aky5+U1GEYpFWNZfveccta5FI8UO3xxHGqeaKAqlrgW hC6WOamuZZ1D+KlOLy1c/gyjv4SWxxfePlwpt7TwAsw4pmhnOhNP/cqE+Wd4zbA8ugJR jX2pygz0psEEEm7EqwebSadd/DdtlQUU7UCEnNCRf4AHXZ/+qmnkyhmh1HnH9VIzmuam 1rEihnoxVAg3biA4pbi5zxArVf5UKpLK5QJWrnvW5yemTrJTFjXJ9mL1osaDjZhG4A+5 npifTNtiGV5btemBe+YL5H8THilboZMPoYBwwvqo+Z1cMUjuZvJn2fXofS0vV36/NSlA FB6w== X-Gm-Message-State: AJaThX5NMr+PvweX7rtyKpMPBDFb/clE0osLjwfeQHLMmHPMM7EJKo7B 6DXERk4GoDmkxySbLdqIICi+uQ== X-Received: by 10.223.152.234 with SMTP id w97mr20295519wrb.215.1512589451423; Wed, 06 Dec 2017 11:44:11 -0800 (PST) Received: from localhost.localdomain ([105.150.171.234]) by smtp.gmail.com with ESMTPSA id b66sm3596594wmh.32.2017.12.06.11.44.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 06 Dec 2017 11:44:10 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v3 06/20] crypto: arm64/aes-blk - remove configurable interleave Date: Wed, 6 Dec 2017 19:43:32 +0000 Message-Id: <20171206194346.24393-7-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171206194346.24393-1-ard.biesheuvel@linaro.org> References: <20171206194346.24393-1-ard.biesheuvel@linaro.org> Sender: linux-rt-users-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rt-users@vger.kernel.org The AES block mode implementation using Crypto Extensions or plain NEON was written before real hardware existed, and so its interleave factor was made build time configurable (as well as an option to instantiate all interleaved sequences inline rather than as subroutines) We ended up using INTERLEAVE=4 with inlining disabled for both flavors of the core AES routines, so let's stick with that, and remove the option to configure this at build time. This makes the code easier to modify, which is nice now that we're adding yield support. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/Makefile | 3 - arch/arm64/crypto/aes-modes.S | 237 ++++---------------- 2 files changed, 40 insertions(+), 200 deletions(-) -- 2.11.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/arch/arm64/crypto/Makefile b/arch/arm64/crypto/Makefile index b5edc5918c28..aaf4e9afd750 100644 --- a/arch/arm64/crypto/Makefile +++ b/arch/arm64/crypto/Makefile @@ -50,9 +50,6 @@ aes-arm64-y := aes-cipher-core.o aes-cipher-glue.o obj-$(CONFIG_CRYPTO_AES_ARM64_BS) += aes-neon-bs.o aes-neon-bs-y := aes-neonbs-core.o aes-neonbs-glue.o -AFLAGS_aes-ce.o := -DINTERLEAVE=4 -AFLAGS_aes-neon.o := -DINTERLEAVE=4 - CFLAGS_aes-glue-ce.o := -DUSE_V8_CRYPTO_EXTENSIONS $(obj)/aes-glue-%.o: $(src)/aes-glue.c FORCE diff --git a/arch/arm64/crypto/aes-modes.S b/arch/arm64/crypto/aes-modes.S index 65b273667b34..27a235b2ddee 100644 --- a/arch/arm64/crypto/aes-modes.S +++ b/arch/arm64/crypto/aes-modes.S @@ -13,44 +13,6 @@ .text .align 4 -/* - * There are several ways to instantiate this code: - * - no interleave, all inline - * - 2-way interleave, 2x calls out of line (-DINTERLEAVE=2) - * - 2-way interleave, all inline (-DINTERLEAVE=2 -DINTERLEAVE_INLINE) - * - 4-way interleave, 4x calls out of line (-DINTERLEAVE=4) - * - 4-way interleave, all inline (-DINTERLEAVE=4 -DINTERLEAVE_INLINE) - * - * Macros imported by this code: - * - enc_prepare - setup NEON registers for encryption - * - dec_prepare - setup NEON registers for decryption - * - enc_switch_key - change to new key after having prepared for encryption - * - encrypt_block - encrypt a single block - * - decrypt block - decrypt a single block - * - encrypt_block2x - encrypt 2 blocks in parallel (if INTERLEAVE == 2) - * - decrypt_block2x - decrypt 2 blocks in parallel (if INTERLEAVE == 2) - * - encrypt_block4x - encrypt 4 blocks in parallel (if INTERLEAVE == 4) - * - decrypt_block4x - decrypt 4 blocks in parallel (if INTERLEAVE == 4) - */ - -#if defined(INTERLEAVE) && !defined(INTERLEAVE_INLINE) -#define FRAME_PUSH stp x29, x30, [sp,#-16]! ; mov x29, sp -#define FRAME_POP ldp x29, x30, [sp],#16 - -#if INTERLEAVE == 2 - -aes_encrypt_block2x: - encrypt_block2x v0, v1, w3, x2, x8, w7 - ret -ENDPROC(aes_encrypt_block2x) - -aes_decrypt_block2x: - decrypt_block2x v0, v1, w3, x2, x8, w7 - ret -ENDPROC(aes_decrypt_block2x) - -#elif INTERLEAVE == 4 - aes_encrypt_block4x: encrypt_block4x v0, v1, v2, v3, w3, x2, x8, w7 ret @@ -61,48 +23,6 @@ aes_decrypt_block4x: ret ENDPROC(aes_decrypt_block4x) -#else -#error INTERLEAVE should equal 2 or 4 -#endif - - .macro do_encrypt_block2x - bl aes_encrypt_block2x - .endm - - .macro do_decrypt_block2x - bl aes_decrypt_block2x - .endm - - .macro do_encrypt_block4x - bl aes_encrypt_block4x - .endm - - .macro do_decrypt_block4x - bl aes_decrypt_block4x - .endm - -#else -#define FRAME_PUSH -#define FRAME_POP - - .macro do_encrypt_block2x - encrypt_block2x v0, v1, w3, x2, x8, w7 - .endm - - .macro do_decrypt_block2x - decrypt_block2x v0, v1, w3, x2, x8, w7 - .endm - - .macro do_encrypt_block4x - encrypt_block4x v0, v1, v2, v3, w3, x2, x8, w7 - .endm - - .macro do_decrypt_block4x - decrypt_block4x v0, v1, v2, v3, w3, x2, x8, w7 - .endm - -#endif - /* * aes_ecb_encrypt(u8 out[], u8 const in[], u8 const rk[], int rounds, * int blocks) @@ -111,28 +31,21 @@ ENDPROC(aes_decrypt_block4x) */ AES_ENTRY(aes_ecb_encrypt) - FRAME_PUSH + stp x29, x30, [sp, #-16]! + mov x29, sp enc_prepare w3, x2, x5 .LecbencloopNx: -#if INTERLEAVE >= 2 - subs w4, w4, #INTERLEAVE + subs w4, w4, #4 bmi .Lecbenc1x -#if INTERLEAVE == 2 - ld1 {v0.16b-v1.16b}, [x1], #32 /* get 2 pt blocks */ - do_encrypt_block2x - st1 {v0.16b-v1.16b}, [x0], #32 -#else ld1 {v0.16b-v3.16b}, [x1], #64 /* get 4 pt blocks */ - do_encrypt_block4x + bl aes_encrypt_block4x st1 {v0.16b-v3.16b}, [x0], #64 -#endif b .LecbencloopNx .Lecbenc1x: - adds w4, w4, #INTERLEAVE + adds w4, w4, #4 beq .Lecbencout -#endif .Lecbencloop: ld1 {v0.16b}, [x1], #16 /* get next pt block */ encrypt_block v0, w3, x2, x5, w6 @@ -140,34 +53,27 @@ AES_ENTRY(aes_ecb_encrypt) subs w4, w4, #1 bne .Lecbencloop .Lecbencout: - FRAME_POP + ldp x29, x30, [sp], #16 ret AES_ENDPROC(aes_ecb_encrypt) AES_ENTRY(aes_ecb_decrypt) - FRAME_PUSH + stp x29, x30, [sp, #-16]! + mov x29, sp dec_prepare w3, x2, x5 .LecbdecloopNx: -#if INTERLEAVE >= 2 - subs w4, w4, #INTERLEAVE + subs w4, w4, #4 bmi .Lecbdec1x -#if INTERLEAVE == 2 - ld1 {v0.16b-v1.16b}, [x1], #32 /* get 2 ct blocks */ - do_decrypt_block2x - st1 {v0.16b-v1.16b}, [x0], #32 -#else ld1 {v0.16b-v3.16b}, [x1], #64 /* get 4 ct blocks */ - do_decrypt_block4x + bl aes_decrypt_block4x st1 {v0.16b-v3.16b}, [x0], #64 -#endif b .LecbdecloopNx .Lecbdec1x: - adds w4, w4, #INTERLEAVE + adds w4, w4, #4 beq .Lecbdecout -#endif .Lecbdecloop: ld1 {v0.16b}, [x1], #16 /* get next ct block */ decrypt_block v0, w3, x2, x5, w6 @@ -175,7 +81,7 @@ AES_ENTRY(aes_ecb_decrypt) subs w4, w4, #1 bne .Lecbdecloop .Lecbdecout: - FRAME_POP + ldp x29, x30, [sp], #16 ret AES_ENDPROC(aes_ecb_decrypt) @@ -204,30 +110,20 @@ AES_ENDPROC(aes_cbc_encrypt) AES_ENTRY(aes_cbc_decrypt) - FRAME_PUSH + stp x29, x30, [sp, #-16]! + mov x29, sp ld1 {v7.16b}, [x5] /* get iv */ dec_prepare w3, x2, x6 .LcbcdecloopNx: -#if INTERLEAVE >= 2 - subs w4, w4, #INTERLEAVE + subs w4, w4, #4 bmi .Lcbcdec1x -#if INTERLEAVE == 2 - ld1 {v0.16b-v1.16b}, [x1], #32 /* get 2 ct blocks */ - mov v2.16b, v0.16b - mov v3.16b, v1.16b - do_decrypt_block2x - eor v0.16b, v0.16b, v7.16b - eor v1.16b, v1.16b, v2.16b - mov v7.16b, v3.16b - st1 {v0.16b-v1.16b}, [x0], #32 -#else ld1 {v0.16b-v3.16b}, [x1], #64 /* get 4 ct blocks */ mov v4.16b, v0.16b mov v5.16b, v1.16b mov v6.16b, v2.16b - do_decrypt_block4x + bl aes_decrypt_block4x sub x1, x1, #16 eor v0.16b, v0.16b, v7.16b eor v1.16b, v1.16b, v4.16b @@ -235,12 +131,10 @@ AES_ENTRY(aes_cbc_decrypt) eor v2.16b, v2.16b, v5.16b eor v3.16b, v3.16b, v6.16b st1 {v0.16b-v3.16b}, [x0], #64 -#endif b .LcbcdecloopNx .Lcbcdec1x: - adds w4, w4, #INTERLEAVE + adds w4, w4, #4 beq .Lcbcdecout -#endif .Lcbcdecloop: ld1 {v1.16b}, [x1], #16 /* get next ct block */ mov v0.16b, v1.16b /* ...and copy to v0 */ @@ -251,8 +145,8 @@ AES_ENTRY(aes_cbc_decrypt) subs w4, w4, #1 bne .Lcbcdecloop .Lcbcdecout: - FRAME_POP st1 {v7.16b}, [x5] /* return iv */ + ldp x29, x30, [sp], #16 ret AES_ENDPROC(aes_cbc_decrypt) @@ -263,34 +157,19 @@ AES_ENDPROC(aes_cbc_decrypt) */ AES_ENTRY(aes_ctr_encrypt) - FRAME_PUSH + stp x29, x30, [sp, #-16]! + mov x29, sp enc_prepare w3, x2, x6 ld1 {v4.16b}, [x5] umov x6, v4.d[1] /* keep swabbed ctr in reg */ rev x6, x6 -#if INTERLEAVE >= 2 cmn w6, w4 /* 32 bit overflow? */ bcs .Lctrloop .LctrloopNx: - subs w4, w4, #INTERLEAVE + subs w4, w4, #4 bmi .Lctr1x -#if INTERLEAVE == 2 - mov v0.8b, v4.8b - mov v1.8b, v4.8b - rev x7, x6 - add x6, x6, #1 - ins v0.d[1], x7 - rev x7, x6 - add x6, x6, #1 - ins v1.d[1], x7 - ld1 {v2.16b-v3.16b}, [x1], #32 /* get 2 input blocks */ - do_encrypt_block2x - eor v0.16b, v0.16b, v2.16b - eor v1.16b, v1.16b, v3.16b - st1 {v0.16b-v1.16b}, [x0], #32 -#else ldr q8, =0x30000000200000001 /* addends 1,2,3[,0] */ dup v7.4s, w6 mov v0.16b, v4.16b @@ -303,23 +182,21 @@ AES_ENTRY(aes_ctr_encrypt) mov v2.s[3], v8.s[1] mov v3.s[3], v8.s[2] ld1 {v5.16b-v7.16b}, [x1], #48 /* get 3 input blocks */ - do_encrypt_block4x + bl aes_encrypt_block4x eor v0.16b, v5.16b, v0.16b ld1 {v5.16b}, [x1], #16 /* get 1 input block */ eor v1.16b, v6.16b, v1.16b eor v2.16b, v7.16b, v2.16b eor v3.16b, v5.16b, v3.16b st1 {v0.16b-v3.16b}, [x0], #64 - add x6, x6, #INTERLEAVE -#endif + add x6, x6, #4 rev x7, x6 ins v4.d[1], x7 cbz w4, .Lctrout b .LctrloopNx .Lctr1x: - adds w4, w4, #INTERLEAVE + adds w4, w4, #4 beq .Lctrout -#endif .Lctrloop: mov v0.16b, v4.16b encrypt_block v0, w3, x2, x8, w7 @@ -339,12 +216,12 @@ AES_ENTRY(aes_ctr_encrypt) .Lctrout: st1 {v4.16b}, [x5] /* return next CTR value */ - FRAME_POP + ldp x29, x30, [sp], #16 ret .Lctrtailblock: st1 {v0.16b}, [x0] - FRAME_POP + ldp x29, x30, [sp], #16 ret .Lctrcarry: @@ -378,7 +255,9 @@ CPU_LE( .quad 1, 0x87 ) CPU_BE( .quad 0x87, 1 ) AES_ENTRY(aes_xts_encrypt) - FRAME_PUSH + stp x29, x30, [sp, #-16]! + mov x29, sp + ld1 {v4.16b}, [x6] cbz w7, .Lxtsencnotfirst @@ -394,25 +273,8 @@ AES_ENTRY(aes_xts_encrypt) ldr q7, .Lxts_mul_x next_tweak v4, v4, v7, v8 .LxtsencNx: -#if INTERLEAVE >= 2 - subs w4, w4, #INTERLEAVE + subs w4, w4, #4 bmi .Lxtsenc1x -#if INTERLEAVE == 2 - ld1 {v0.16b-v1.16b}, [x1], #32 /* get 2 pt blocks */ - next_tweak v5, v4, v7, v8 - eor v0.16b, v0.16b, v4.16b - eor v1.16b, v1.16b, v5.16b - do_encrypt_block2x - eor v0.16b, v0.16b, v4.16b - eor v1.16b, v1.16b, v5.16b - st1 {v0.16b-v1.16b}, [x0], #32 - cbz w4, .LxtsencoutNx - next_tweak v4, v5, v7, v8 - b .LxtsencNx -.LxtsencoutNx: - mov v4.16b, v5.16b - b .Lxtsencout -#else ld1 {v0.16b-v3.16b}, [x1], #64 /* get 4 pt blocks */ next_tweak v5, v4, v7, v8 eor v0.16b, v0.16b, v4.16b @@ -421,7 +283,7 @@ AES_ENTRY(aes_xts_encrypt) eor v2.16b, v2.16b, v6.16b next_tweak v7, v6, v7, v8 eor v3.16b, v3.16b, v7.16b - do_encrypt_block4x + bl aes_encrypt_block4x eor v3.16b, v3.16b, v7.16b eor v0.16b, v0.16b, v4.16b eor v1.16b, v1.16b, v5.16b @@ -430,11 +292,9 @@ AES_ENTRY(aes_xts_encrypt) mov v4.16b, v7.16b cbz w4, .Lxtsencout b .LxtsencloopNx -#endif .Lxtsenc1x: - adds w4, w4, #INTERLEAVE + adds w4, w4, #4 beq .Lxtsencout -#endif .Lxtsencloop: ld1 {v1.16b}, [x1], #16 eor v0.16b, v1.16b, v4.16b @@ -447,13 +307,15 @@ AES_ENTRY(aes_xts_encrypt) b .Lxtsencloop .Lxtsencout: st1 {v4.16b}, [x6] - FRAME_POP + ldp x29, x30, [sp], #16 ret AES_ENDPROC(aes_xts_encrypt) AES_ENTRY(aes_xts_decrypt) - FRAME_PUSH + stp x29, x30, [sp, #-16]! + mov x29, sp + ld1 {v4.16b}, [x6] cbz w7, .Lxtsdecnotfirst @@ -469,25 +331,8 @@ AES_ENTRY(aes_xts_decrypt) ldr q7, .Lxts_mul_x next_tweak v4, v4, v7, v8 .LxtsdecNx: -#if INTERLEAVE >= 2 - subs w4, w4, #INTERLEAVE + subs w4, w4, #4 bmi .Lxtsdec1x -#if INTERLEAVE == 2 - ld1 {v0.16b-v1.16b}, [x1], #32 /* get 2 ct blocks */ - next_tweak v5, v4, v7, v8 - eor v0.16b, v0.16b, v4.16b - eor v1.16b, v1.16b, v5.16b - do_decrypt_block2x - eor v0.16b, v0.16b, v4.16b - eor v1.16b, v1.16b, v5.16b - st1 {v0.16b-v1.16b}, [x0], #32 - cbz w4, .LxtsdecoutNx - next_tweak v4, v5, v7, v8 - b .LxtsdecNx -.LxtsdecoutNx: - mov v4.16b, v5.16b - b .Lxtsdecout -#else ld1 {v0.16b-v3.16b}, [x1], #64 /* get 4 ct blocks */ next_tweak v5, v4, v7, v8 eor v0.16b, v0.16b, v4.16b @@ -496,7 +341,7 @@ AES_ENTRY(aes_xts_decrypt) eor v2.16b, v2.16b, v6.16b next_tweak v7, v6, v7, v8 eor v3.16b, v3.16b, v7.16b - do_decrypt_block4x + bl aes_decrypt_block4x eor v3.16b, v3.16b, v7.16b eor v0.16b, v0.16b, v4.16b eor v1.16b, v1.16b, v5.16b @@ -505,11 +350,9 @@ AES_ENTRY(aes_xts_decrypt) mov v4.16b, v7.16b cbz w4, .Lxtsdecout b .LxtsdecloopNx -#endif .Lxtsdec1x: - adds w4, w4, #INTERLEAVE + adds w4, w4, #4 beq .Lxtsdecout -#endif .Lxtsdecloop: ld1 {v1.16b}, [x1], #16 eor v0.16b, v1.16b, v4.16b @@ -522,7 +365,7 @@ AES_ENTRY(aes_xts_decrypt) b .Lxtsdecloop .Lxtsdecout: st1 {v4.16b}, [x6] - FRAME_POP + ldp x29, x30, [sp], #16 ret AES_ENDPROC(aes_xts_decrypt) From patchwork Wed Dec 6 19:43:33 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 120889 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp7468699qgn; Wed, 6 Dec 2017 11:44:23 -0800 (PST) X-Google-Smtp-Source: AGs4zMbu96T7mcOWO4y7gzpPYgQjfJW0x+3qk/RNOEINfyK1YLmKQmaH8tAQH4tYB2e/ljfnQfz1 X-Received: by 10.84.168.98 with SMTP id e89mr22689727plb.417.1512589463086; Wed, 06 Dec 2017 11:44:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512589463; cv=none; d=google.com; s=arc-20160816; b=S6WvL0GKiHml7A3Bx0Be/UfTyyGU2dq4Hv+xcZF2FWuxXe9I1dWOrQXK2IExnMC66S wTcCAhUtM/eKuOgVkbQVgGVaacHqe9qkYrxlsiqwxI4VVyakGVt/dmxn4VXISLl3ryuv KOKDfJ/KdoyUMBFj2pQf2s2s80liuiGoXVKyRHCW2OCih9JtbZSarf/OuiLFvDLofuGa W56NDpF7rEAA+aXmbwLWPwTBqmQBio2A0w+mFZjOuMjFm+DnvpCn02G0Yyc8GUN2+vHz 37NYM5GG3VehKBp4tLdj3+4ASvhcpNNI3QD4GI4N6rw1BpRdcLdu/VSfOfqn1ezb04OD Ampg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=ByEfG8S6mziH6dHyDov6YH26KAHAKkbH9eW3AIOrYSU=; b=tHHnn2ebh0a8txYTt7idy4wmRlSPDnQ6tDiboNaMMKiOf/0Lg3CVxx4m+gjWq4x9u9 VkI+3YAi/ndYAtKbFhqv/aiLuCCwrb5laShYyd5vqXd4KYg0REn2SPM9egKwqDFOBGWI /SV1jsG418rG+tPt7bTcDV+1LR+VJvGH543rdfINJVBmDwhUdbvptX5lmpIsrkNn4LMG leuK5vZaUzWG06/+3iysos9HWXh/f6n6Chb+284SAXLCVmTZ/WXphaRViX5GfKZTZAR0 rK13oPT5E89xYPdvCWWSDAmqBYpE+YKZd3HH6yDUafOj1rbvc+7n9Dx6Uo+h0SsBkUuT 7vvw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=gPnc/IaL; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id i16si2382524pgv.496.2017.12.06.11.44.22; Wed, 06 Dec 2017 11:44:23 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=gPnc/IaL; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752282AbdLFToV (ORCPT + 4 others); Wed, 6 Dec 2017 14:44:21 -0500 Received: from mail-wr0-f194.google.com ([209.85.128.194]:41417 "EHLO mail-wr0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752119AbdLFToQ (ORCPT ); Wed, 6 Dec 2017 14:44:16 -0500 Received: by mail-wr0-f194.google.com with SMTP id z18so5104136wrb.8 for ; Wed, 06 Dec 2017 11:44:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=I6Gnvs3Ig+XMQwO7jGoODHjYQvJyRj6RsM7VfBN0Rns=; b=gPnc/IaLPhCdaYOcGr4J+2RoP5OwvmoAXTKq9ICW09VxYkYF54b42E1vjjqzmKGLoM T7GLu3KWgqOXwI0iRN8boNafiqAXMmo+78Twwjs6gs3A3YmjVaKpmmOMOmWT0+WjGoUb R0uCMHgmXAQVpei9kx6u1t9YmKaE1qwLbasfo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=I6Gnvs3Ig+XMQwO7jGoODHjYQvJyRj6RsM7VfBN0Rns=; b=erCvdS1hCRkFn5sYiAmNNV3Cq5kYcJhug4TSarCw5pxWVVhlYIc190Hk5mPkd1rwHi UYyPFGBysUSegmrRjAQm8RHFv2a+ZIfxSSXnaJrKRn8IfVewXBdGsA0jPda3Tyz+yvBT I6UR0Irr/8/0TdvUWy1JX3VYElVkXUwX8TnEhu0RE3WXpRcSgWfcvlaFuaUw4pV1nN25 xGHc0uYVdn20A6/qHB+t/k/g3A6vxR2/9JyehH+SjWK/DP6Ha41WMzKSUWiI6u58y7gK h4/Un1G+OY9DCPM97LgAWiurMwqu1aLFR7Lzrq2PV6Zy/uMOMcziHyioRDgvBTiEH0RU Wr+Q== X-Gm-Message-State: AJaThX6yvHCWNPUl+Y0wgZtves/F8rm6alnVqerKwS+Huk4RkkokcNsf 4HgPq5Istrri+6+e/BYqARfiRw== X-Received: by 10.223.195.103 with SMTP id e36mr21193552wrg.10.1512589454918; Wed, 06 Dec 2017 11:44:14 -0800 (PST) Received: from localhost.localdomain ([105.150.171.234]) by smtp.gmail.com with ESMTPSA id b66sm3596594wmh.32.2017.12.06.11.44.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 06 Dec 2017 11:44:13 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v3 07/20] crypto: arm64/aes-blk - add 4 way interleave to CBC encrypt path Date: Wed, 6 Dec 2017 19:43:33 +0000 Message-Id: <20171206194346.24393-8-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171206194346.24393-1-ard.biesheuvel@linaro.org> References: <20171206194346.24393-1-ard.biesheuvel@linaro.org> Sender: linux-rt-users-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rt-users@vger.kernel.org CBC encryption is strictly sequential, and so the current AES code simply processes the input one block at a time. However, we are about to add yield support, which adds a bit of overhead, and which we prefer to align with other modes in terms of granularity (i.e., it is better to have all routines yield every 64 bytes and not have an exception for CBC encrypt which yields every 16 bytes) So unroll the loop by 4. We still cannot perform the AES algorithm in parallel, but we can at least merge the loads and stores. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-modes.S | 31 ++++++++++++++++---- 1 file changed, 25 insertions(+), 6 deletions(-) -- 2.11.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/arch/arm64/crypto/aes-modes.S b/arch/arm64/crypto/aes-modes.S index 27a235b2ddee..e86535a1329d 100644 --- a/arch/arm64/crypto/aes-modes.S +++ b/arch/arm64/crypto/aes-modes.S @@ -94,17 +94,36 @@ AES_ENDPROC(aes_ecb_decrypt) */ AES_ENTRY(aes_cbc_encrypt) - ld1 {v0.16b}, [x5] /* get iv */ + ld1 {v4.16b}, [x5] /* get iv */ enc_prepare w3, x2, x6 -.Lcbcencloop: - ld1 {v1.16b}, [x1], #16 /* get next pt block */ - eor v0.16b, v0.16b, v1.16b /* ..and xor with iv */ +.Lcbcencloop4x: + subs w4, w4, #4 + bmi .Lcbcenc1x + ld1 {v0.16b-v3.16b}, [x1], #64 /* get 4 pt blocks */ + eor v0.16b, v0.16b, v4.16b /* ..and xor with iv */ encrypt_block v0, w3, x2, x6, w7 - st1 {v0.16b}, [x0], #16 + eor v1.16b, v1.16b, v0.16b + encrypt_block v1, w3, x2, x6, w7 + eor v2.16b, v2.16b, v1.16b + encrypt_block v2, w3, x2, x6, w7 + eor v3.16b, v3.16b, v2.16b + encrypt_block v3, w3, x2, x6, w7 + st1 {v0.16b-v3.16b}, [x0], #64 + mov v4.16b, v3.16b + b .Lcbcencloop4x +.Lcbcenc1x: + adds w4, w4, #4 + beq .Lcbcencout +.Lcbcencloop: + ld1 {v0.16b}, [x1], #16 /* get next pt block */ + eor v4.16b, v4.16b, v0.16b /* ..and xor with iv */ + encrypt_block v4, w3, x2, x6, w7 + st1 {v4.16b}, [x0], #16 subs w4, w4, #1 bne .Lcbcencloop - st1 {v0.16b}, [x5] /* return iv */ +.Lcbcencout: + st1 {v4.16b}, [x5] /* return iv */ ret AES_ENDPROC(aes_cbc_encrypt) From patchwork Wed Dec 6 19:43:34 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 120891 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp7468832qgn; Wed, 6 Dec 2017 11:44:31 -0800 (PST) X-Google-Smtp-Source: AGs4zMZJhY1miJe/m/rJART59ELOUC3eedlcP9BW7afoumF96m2I/VQ3/P+lFNGPaBkKAzkgirrM X-Received: by 10.98.21.17 with SMTP id 17mr4028097pfv.120.1512589471589; Wed, 06 Dec 2017 11:44:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512589471; cv=none; d=google.com; s=arc-20160816; b=bF/j8VY8hVUMOXSUnsy8fKqX8j3cYnm1MnmZy93zQpAt4tiiLkE/xkd/i5yvUkFtGy ICJVgYH3FswYuePyfRZ7+TwRJFeYDZBkx+2rdzModNuNk8p7yZQm2e1vbnAzF81WK/Y3 eRaM9StyGFAGu6fgTIHOvfRhpI+y8CpS4H8pze72k8x3Pe5Dct0RiR+afYFph83CnHkN kMhCf0B+uO5LAf4ArIuVbqZRoKEDcsWl8jMO5we5OwKYMx/RyMhMfWn6KWPSQd3GkAbg 0GP40tOLhJVs0uG7OWNZyjgKkGYY2Beg8/SZkeuCfs4n2rWMyRBdZOov7ncTuvCajPYG pRrA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=4L8VBFJ8/vUZayezPEZFRvTau06CygKiThORT60NjPo=; b=VF3xQ7vtXM2EJ9Hhi+9yf9sLjbfuaWNRBr6NeD4GElabvbwdzSUvcvcnRRztGqSA7h 4FRJ4LMKRV/AgPsxPTf6YZ9U+tdlYGNklE3C95l41Of/U4SziBk3Y2/xeyAa1Ce82bz8 23IbNh5sASV0nVKMMKOskJZhkW86qdxNb6/7+mkpUlljTxHDx0YO4hFM8nwBZydgPN+7 7MbIkuc5t1gBnv0vcwqYBz+tilGnd+ZVShRWoC6czauQIkK+YHqKxkeN/nSuUzzYlZft gbEHTNsAdkwEZBf+ORW2+t31HFhYjRhfK9xaw7WONsKkULv7VQF1XhQ6NRuBZ9rSQ8bW j0kQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=BfeaCIe3; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id i16si2382524pgv.496.2017.12.06.11.44.31; Wed, 06 Dec 2017 11:44:31 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=BfeaCIe3; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752344AbdLFTo3 (ORCPT + 4 others); Wed, 6 Dec 2017 14:44:29 -0500 Received: from mail-wr0-f196.google.com ([209.85.128.196]:35633 "EHLO mail-wr0-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752132AbdLFToT (ORCPT ); Wed, 6 Dec 2017 14:44:19 -0500 Received: by mail-wr0-f196.google.com with SMTP id g53so5128980wra.2 for ; Wed, 06 Dec 2017 11:44:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=AVGr1XiAAq1ruDzOgmfaJd5iqwO5GoptNYnkdzAZJA8=; b=BfeaCIe3bP4WigQt68E0qH/qrUXmozDFVhJ9mVfSpVq8sWTT3H1/xKTP0S5ULBRNeE i+UVvst52Nr+FIHkQbwrd1cASaskFmYHscpH0XkAXRAKmXEWcmpRgCZY8D9nFRxbQ+JJ KYB59CdIaJwWuiJVhT3r6HljuCYh8nsmJ7Maw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=AVGr1XiAAq1ruDzOgmfaJd5iqwO5GoptNYnkdzAZJA8=; b=KeffHrOJLfaTOLFIgpqxBfqy3Er3RKEgzyox8GccWy2m5BwQKJMX188QXKws6+9AqY hcd+vs2xHnkiUkiuUsLE5LyFgOS0pUP8HqjIUZLKcA/CGWR6K0STbt6RkSYStjzi5iCE Rpqkg25f3KxVzNYphG1XSfsdbRK/VJQOY9kH2cYeeeXtohFPw4spTaxoyPGbtb+fVYCx PFh5Z5hBUD50uGoOsmoezpe5AA1qAjY+SDcqQfMCAxcvIhRGvioavfinB3gTO1O8pPk/ 5twPub4UK2fu8igicJIUjGcJXpvXFuW/Qd5jbh//1TWBmiHgjWA+qrgQ8Lz2ZzcoTvM4 IeGA== X-Gm-Message-State: AKGB3mJYS5DLJjRGqScdZLOhb7R8SBYNynnfDIaUQyUOhbE09qTu9YK6 WJx6osX9YYSESSW5c8jFMssIRg== X-Received: by 10.223.134.184 with SMTP id 53mr509613wrx.17.1512589458265; Wed, 06 Dec 2017 11:44:18 -0800 (PST) Received: from localhost.localdomain ([105.150.171.234]) by smtp.gmail.com with ESMTPSA id b66sm3596594wmh.32.2017.12.06.11.44.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 06 Dec 2017 11:44:17 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v3 08/20] crypto: arm64/aes-blk - add 4 way interleave to CBC-MAC encrypt path Date: Wed, 6 Dec 2017 19:43:34 +0000 Message-Id: <20171206194346.24393-9-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171206194346.24393-1-ard.biesheuvel@linaro.org> References: <20171206194346.24393-1-ard.biesheuvel@linaro.org> Sender: linux-rt-users-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rt-users@vger.kernel.org CBC MAC is strictly sequential, and so the current AES code simply processes the input one block at a time. However, we are about to add yield support, which adds a bit of overhead, and which we prefer to align with other modes in terms of granularity (i.e., it is better to have all routines yield every 64 bytes and not have an exception for CBC MAC which yields every 16 bytes) So unroll the loop by 4. We still cannot perform the AES algorithm in parallel, but we can at least merge the loads and stores. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-modes.S | 23 ++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) -- 2.11.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/arch/arm64/crypto/aes-modes.S b/arch/arm64/crypto/aes-modes.S index e86535a1329d..a68412e1e3a4 100644 --- a/arch/arm64/crypto/aes-modes.S +++ b/arch/arm64/crypto/aes-modes.S @@ -395,8 +395,28 @@ AES_ENDPROC(aes_xts_decrypt) AES_ENTRY(aes_mac_update) ld1 {v0.16b}, [x4] /* get dg */ enc_prepare w2, x1, x7 - cbnz w5, .Lmacenc + cbz w5, .Lmacloop4x + encrypt_block v0, w2, x1, x7, w8 + +.Lmacloop4x: + subs w3, w3, #4 + bmi .Lmac1x + ld1 {v1.16b-v4.16b}, [x0], #64 /* get next pt block */ + eor v0.16b, v0.16b, v1.16b /* ..and xor with dg */ + encrypt_block v0, w2, x1, x7, w8 + eor v0.16b, v0.16b, v2.16b + encrypt_block v0, w2, x1, x7, w8 + eor v0.16b, v0.16b, v3.16b + encrypt_block v0, w2, x1, x7, w8 + eor v0.16b, v0.16b, v4.16b + cmp w3, wzr + csinv x5, x6, xzr, eq + cbz w5, .Lmacout + encrypt_block v0, w2, x1, x7, w8 + b .Lmacloop4x +.Lmac1x: + add w3, w3, #4 .Lmacloop: cbz w3, .Lmacout ld1 {v1.16b}, [x0], #16 /* get next pt block */ @@ -406,7 +426,6 @@ AES_ENTRY(aes_mac_update) csinv x5, x6, xzr, eq cbz w5, .Lmacout -.Lmacenc: encrypt_block v0, w2, x1, x7, w8 b .Lmacloop From patchwork Wed Dec 6 19:43:38 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 120896 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp7469021qgn; Wed, 6 Dec 2017 11:44:43 -0800 (PST) X-Google-Smtp-Source: AGs4zMbTUwTvYKAKDHyXpxYjFZrt1bLUZnJl13bOUX5niD1CgGefhuK60UuQ6nbbNGK2dr9AZE2V X-Received: by 10.99.154.66 with SMTP id e2mr22442287pgo.8.1512589483706; Wed, 06 Dec 2017 11:44:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512589483; cv=none; d=google.com; s=arc-20160816; b=POOeG4huSs/bjzPnsQ+ESptjXa4CT8YfvX56kwt/gvWpsq+NaHncbMXvQd4OuOOSoP bGpp8CDNu+Wr/TvrN6IamN09SkgrUOjklFFVv/0gr1uf/+PnZXZtMPLvYm6y0Q7MO+sG S2afqWFMYeCofxeZ0U6rusmi8XZVqia4iRLT89z741vRRGvAnydz1PLZiMD5c7leRGNl wA3AkSwyfOt96IMiuql7Xb1SaQHWInH+T+XrJlUK7ngcSHkzWmSh4EFGfwKtmQokrbwN AYuvjXGweBUmHcfB/irnvZn11LAvEKdOKy9pHzBekWa/nx8ePnWK/4OodKzzAyCPHoRG Icrw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=nfy5zys5sGPk+2nXssI9aGBPiaq2uVy9hW55bCXgA9g=; b=QRQNfzC5vRkDfNxH4SoodTDL9+h4tysyNZw//TnXn8aWWenT5MFgEU+yvlHkKKLkpS qwLDGZz8g2Jl6m7M/ZU5CKelLJBTPEy8D/InVtngCfhb02R8FmgDwBU/8ZvYHRa9I/yu Dij1DLI7kY0gxf6haFEzu0KAt5eKQ7aMuSJrQWdkAk+I3G5Gis58InV2wjWB11MgbVxA lgszDOdaafsrSSJPbszTEuH8O7sOcQpzAbIGItt/aHCAVD4ZLmpSSAWS9jZcWzhXXeeQ TtsSb8vHNm7fESdJnih59xDgQY6KItPYXzK2Q6UOVoKBoAxCaCsvFbZPNmMgw1ZPnlhF 6o5g== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=OxEpuixC; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id i16si2382524pgv.496.2017.12.06.11.44.43; Wed, 06 Dec 2017 11:44:43 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=OxEpuixC; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752430AbdLFTom (ORCPT + 4 others); Wed, 6 Dec 2017 14:44:42 -0500 Received: from mail-wr0-f196.google.com ([209.85.128.196]:34532 "EHLO mail-wr0-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752132AbdLFTob (ORCPT ); Wed, 6 Dec 2017 14:44:31 -0500 Received: by mail-wr0-f196.google.com with SMTP id y21so5132801wrc.1 for ; Wed, 06 Dec 2017 11:44:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Rz5PcTd8nEXxwHJxDvYFukQ1E4CKqQgN4lX4bhAfRbM=; b=OxEpuixCWaYpY1LYPWVoryEs43gWtjVNdTEFKnnrraCYg+r8FaP/hHoLght5TJVGZQ yBEa7FSzJ99yaD8Mp/wdVJkfALB6IUwqWQ+rBV+6/G1RU9p6jrgF9JyaFLMA7bGAewpq 9Ri6FHIokNo6QLBiCK6cAZYPjtOPdtSF6beRA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Rz5PcTd8nEXxwHJxDvYFukQ1E4CKqQgN4lX4bhAfRbM=; b=AroMKow332qyCvaiSvjyDoQHfvCNpKrkVCjIdSYTE6Y71s9mnhcBfhzrPoBEJeoFwO yHV1tQ7CrFxUZftEEQMD27fYIFwwuo66Jm/NOhPwzeuA3AeUfZzL+1zQ7bWQ7Xv6c+5u 9xAlGir/Ma+GFxJ0HzYuGMqgMz9mMQZtpr7Oy8g4m3d+6RRVAqjrz3+ziJQtcScjyZJL UIiCpsfmEA0/2QfZqchSk9Q3G2iZSg8acbZ84nDl6GZQxJkYsLHfneMW1xti07U+jyWf ExfTkGGuZaB00Sf/u9hTDryFG153lRhKEPOc32jjfM0FEbkYFAcIM2md4AGwn9SarOGt 4Dpw== X-Gm-Message-State: AJaThX73SwYx36G5fZuRkivFgKmtMfW68dtOBXghaa0sxNKEkFTIFY4l esDuj3w7qcvEH1M9kABorfz2tw== X-Received: by 10.223.186.197 with SMTP id w5mr19426807wrg.201.1512589470200; Wed, 06 Dec 2017 11:44:30 -0800 (PST) Received: from localhost.localdomain ([105.150.171.234]) by smtp.gmail.com with ESMTPSA id b66sm3596594wmh.32.2017.12.06.11.44.27 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 06 Dec 2017 11:44:29 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v3 12/20] crypto: arm64/sha1-ce - yield NEON after every block of input Date: Wed, 6 Dec 2017 19:43:38 +0000 Message-Id: <20171206194346.24393-13-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171206194346.24393-1-ard.biesheuvel@linaro.org> References: <20171206194346.24393-1-ard.biesheuvel@linaro.org> Sender: linux-rt-users-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rt-users@vger.kernel.org Avoid excessive scheduling delays under a preemptible kernel by yielding the NEON after every block of input. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/sha1-ce-core.S | 42 ++++++++++++++------ 1 file changed, 29 insertions(+), 13 deletions(-) -- 2.11.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/arch/arm64/crypto/sha1-ce-core.S b/arch/arm64/crypto/sha1-ce-core.S index 8550408735a0..3139206e8787 100644 --- a/arch/arm64/crypto/sha1-ce-core.S +++ b/arch/arm64/crypto/sha1-ce-core.S @@ -70,31 +70,37 @@ * int blocks) */ ENTRY(sha1_ce_transform) + frame_push 3 + + mov x19, x0 + mov x20, x1 + mov x21, x2 + /* load round constants */ - adr x6, .Lsha1_rcon +0: adr x6, .Lsha1_rcon ld1r {k0.4s}, [x6], #4 ld1r {k1.4s}, [x6], #4 ld1r {k2.4s}, [x6], #4 ld1r {k3.4s}, [x6] /* load state */ - ld1 {dgav.4s}, [x0] - ldr dgb, [x0, #16] + ld1 {dgav.4s}, [x19] + ldr dgb, [x19, #16] /* load sha1_ce_state::finalize */ ldr_l w4, sha1_ce_offsetof_finalize, x4 - ldr w4, [x0, x4] + ldr w4, [x19, x4] /* load input */ -0: ld1 {v8.4s-v11.4s}, [x1], #64 - sub w2, w2, #1 +1: ld1 {v8.4s-v11.4s}, [x20], #64 + sub w21, w21, #1 CPU_LE( rev32 v8.16b, v8.16b ) CPU_LE( rev32 v9.16b, v9.16b ) CPU_LE( rev32 v10.16b, v10.16b ) CPU_LE( rev32 v11.16b, v11.16b ) -1: add t0.4s, v8.4s, k0.4s +2: add t0.4s, v8.4s, k0.4s mov dg0v.16b, dgav.16b add_update c, ev, k0, 8, 9, 10, 11, dgb @@ -125,16 +131,25 @@ CPU_LE( rev32 v11.16b, v11.16b ) add dgbv.2s, dgbv.2s, dg1v.2s add dgav.4s, dgav.4s, dg0v.4s - cbnz w2, 0b + cbz w21, 3f + + if_will_cond_yield_neon + st1 {dgav.4s}, [x19] + str dgb, [x19, #16] + do_cond_yield_neon + b 0b + endif_yield_neon + + b 1b /* * Final block: add padding and total bit count. * Skip if the input size was not a round multiple of the block size, * the padding is handled by the C code in that case. */ - cbz x4, 3f +3: cbz x4, 4f ldr_l w4, sha1_ce_offsetof_count, x4 - ldr x4, [x0, x4] + ldr x4, [x19, x4] movi v9.2d, #0 mov x8, #0x80000000 movi v10.2d, #0 @@ -143,10 +158,11 @@ CPU_LE( rev32 v11.16b, v11.16b ) mov x4, #0 mov v11.d[0], xzr mov v11.d[1], x7 - b 1b + b 2b /* store new state */ -3: st1 {dgav.4s}, [x0] - str dgb, [x0, #16] +4: st1 {dgav.4s}, [x19] + str dgb, [x19, #16] + frame_pop 3 ret ENDPROC(sha1_ce_transform) From patchwork Wed Dec 6 19:43:39 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 120897 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp7469079qgn; Wed, 6 Dec 2017 11:44:47 -0800 (PST) X-Google-Smtp-Source: AGs4zMbuoKOiWI/FAyUwtGf2UaeK58KHu7B2/Vpf0DlupBQXwn4D7dXvMHzJfKu3HxOuzeF+Aa9J X-Received: by 10.159.206.200 with SMTP id x8mr23334305plo.273.1512589487307; Wed, 06 Dec 2017 11:44:47 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512589487; cv=none; d=google.com; s=arc-20160816; b=rNbuEQNpsdg0p/8kELlUft+VeU4A47xEFI3clmVh4sB56C9NU5E4Pu03VS4SBrzvR0 ukZSnIUG9QTpnJo1e3x0fcfQQLzKXfd7Td72Ut17pDKFEozc6HZpulRrvxTAIVsTugNN 2QGKbCq/D/WI3WBiEXYQ5IvKwBKE643T68SU6ieaEmgNquerUOuA6J+LEi4l3ZOZWOle oGfVYfasQO6411f7PMoQHqW7eowzi51PuRaS/sUAgIpz4pX+yIfkV29YZdnkXZ36nQtP QUennZNz/SKFBcU44ikQFUczzIe5hG3zG/1d8ASF04gr4PpIYfBwQPlk/pTm6SWMN4ac v1aw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=5bs8bmz/xGK0vd/p/LPYi8SYYVEzH8WkohdDzp8Esr0=; b=OTty4fEdfzJOkrfJHOf1lHIYKUgY089AADAAOnJsX6BTyu5dz/DNC1JLz/62gJYnTB EzdLzqQAzZROPLSAySDQF+Vr0TlwlBtmB9vXs5HRVlO4/Xg9+UDoNEM1gYr6MRK4UcyY AzWF0/KHvMuGGgXCm5CDFd3LkXZv+S1e47Bll/eAuJfuSxU7UVgZrR0tYaf0R3Kp/4ex K4Go4sjRUPlZCXouq/UJU/9W9ecphJoDNDMJsXJ4a+xVdUXmwwK537t8eC6UzJ4X1VgL knN8xyD25mcm4k69E9P6dGeUmRJaHgwKT474+qs/zcPXG7Uun7qtXRZ9E3ul8kNv1IkB ptxA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=Ro2PH8dR; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id i16si2382524pgv.496.2017.12.06.11.44.47; Wed, 06 Dec 2017 11:44:47 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=Ro2PH8dR; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751208AbdLFTop (ORCPT + 4 others); Wed, 6 Dec 2017 14:44:45 -0500 Received: from mail-wm0-f65.google.com ([74.125.82.65]:43048 "EHLO mail-wm0-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752362AbdLFToe (ORCPT ); Wed, 6 Dec 2017 14:44:34 -0500 Received: by mail-wm0-f65.google.com with SMTP id n138so9219048wmg.2 for ; Wed, 06 Dec 2017 11:44:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=lA1WyRfGamNxDChcuAimYdrPatx8IosIFMhWUn71St8=; b=Ro2PH8dRZJ0y+ZQp+UYPvExjl2FvuKovCaX74xBJeqGaNQD8CGA35Wl6LpwXn212tg f4TMJHGi+bYyfEw5hXnsQReHH814blH7o4FHX7u0YBeBp3o4qKYwCKYFwF6qC24SSKfj /VtWh6D9LyEEKLZiACg6DL876IHPBdN2YEVoY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=lA1WyRfGamNxDChcuAimYdrPatx8IosIFMhWUn71St8=; b=T0xXXkwvLLxy+LzJRIGgO7CQLbOdUkq02chbxBjGCULZT7DGi5eU+QUaiQ3GoKn73b +v0c1nlIgcmNZgd2LRtj6rVQ+bw2nNUouwMiru+iKTghmaz3fPC1vk6z0edLSmeCvPsN +q6s3ve55fPx/kwg+A02gHOgqNyjcst46zj2pGk7MjtV9jO3HM5Smb2tqt0PeM2mMWja 2rEllcBo7iFGrHZWikNo2Kls6lK96DRa3ZTv+FfixuGqoYmEGcXb8EJTBbEtVpqrl59f n+jNKM5S0dfXmFcTeqLtorOo5oGnDqWxdgxfoYOsZy4XFpFKEmv8n2NbI1mVXmly9tyd D3Dg== X-Gm-Message-State: AKGB3mLvWwRGzKx5gZ9zJ+57W23yuJF1zndcg54djrWaDLxlJrtzQfoi sah6hVcQodUHLRDSCV8g+VmPUQ== X-Received: by 10.28.193.66 with SMTP id r63mr12262059wmf.18.1512589473602; Wed, 06 Dec 2017 11:44:33 -0800 (PST) Received: from localhost.localdomain ([105.150.171.234]) by smtp.gmail.com with ESMTPSA id b66sm3596594wmh.32.2017.12.06.11.44.30 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 06 Dec 2017 11:44:32 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v3 13/20] crypto: arm64/sha2-ce - yield NEON after every block of input Date: Wed, 6 Dec 2017 19:43:39 +0000 Message-Id: <20171206194346.24393-14-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171206194346.24393-1-ard.biesheuvel@linaro.org> References: <20171206194346.24393-1-ard.biesheuvel@linaro.org> Sender: linux-rt-users-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rt-users@vger.kernel.org Avoid excessive scheduling delays under a preemptible kernel by yielding the NEON after every block of input. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/sha2-ce-core.S | 37 ++++++++++++++------ 1 file changed, 26 insertions(+), 11 deletions(-) -- 2.11.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/arch/arm64/crypto/sha2-ce-core.S b/arch/arm64/crypto/sha2-ce-core.S index 679c6c002f4f..7709455dae92 100644 --- a/arch/arm64/crypto/sha2-ce-core.S +++ b/arch/arm64/crypto/sha2-ce-core.S @@ -77,30 +77,36 @@ * int blocks) */ ENTRY(sha2_ce_transform) + frame_push 3 + + mov x19, x0 + mov x20, x1 + mov x21, x2 + /* load round constants */ - adr x8, .Lsha2_rcon +0: adr x8, .Lsha2_rcon ld1 { v0.4s- v3.4s}, [x8], #64 ld1 { v4.4s- v7.4s}, [x8], #64 ld1 { v8.4s-v11.4s}, [x8], #64 ld1 {v12.4s-v15.4s}, [x8] /* load state */ - ld1 {dgav.4s, dgbv.4s}, [x0] + ld1 {dgav.4s, dgbv.4s}, [x19] /* load sha256_ce_state::finalize */ ldr_l w4, sha256_ce_offsetof_finalize, x4 - ldr w4, [x0, x4] + ldr w4, [x19, x4] /* load input */ -0: ld1 {v16.4s-v19.4s}, [x1], #64 - sub w2, w2, #1 +1: ld1 {v16.4s-v19.4s}, [x20], #64 + sub w21, w21, #1 CPU_LE( rev32 v16.16b, v16.16b ) CPU_LE( rev32 v17.16b, v17.16b ) CPU_LE( rev32 v18.16b, v18.16b ) CPU_LE( rev32 v19.16b, v19.16b ) -1: add t0.4s, v16.4s, v0.4s +2: add t0.4s, v16.4s, v0.4s mov dg0v.16b, dgav.16b mov dg1v.16b, dgbv.16b @@ -129,16 +135,24 @@ CPU_LE( rev32 v19.16b, v19.16b ) add dgbv.4s, dgbv.4s, dg1v.4s /* handled all input blocks? */ - cbnz w2, 0b + cbz w21, 3f + + if_will_cond_yield_neon + st1 {dgav.4s, dgbv.4s}, [x19] + do_cond_yield_neon + b 0b + endif_yield_neon + + b 1b /* * Final block: add padding and total bit count. * Skip if the input size was not a round multiple of the block size, * the padding is handled by the C code in that case. */ - cbz x4, 3f +3: cbz x4, 4f ldr_l w4, sha256_ce_offsetof_count, x4 - ldr x4, [x0, x4] + ldr x4, [x19, x4] movi v17.2d, #0 mov x8, #0x80000000 movi v18.2d, #0 @@ -147,9 +161,10 @@ CPU_LE( rev32 v19.16b, v19.16b ) mov x4, #0 mov v19.d[0], xzr mov v19.d[1], x7 - b 1b + b 2b /* store new state */ -3: st1 {dgav.4s, dgbv.4s}, [x0] +4: st1 {dgav.4s, dgbv.4s}, [x19] + frame_pop 3 ret ENDPROC(sha2_ce_transform) From patchwork Wed Dec 6 19:43:44 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 120902 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp7469298qgn; Wed, 6 Dec 2017 11:45:06 -0800 (PST) X-Google-Smtp-Source: AGs4zMYaxRV9V/S0T7F5aK8ECdVky//YpnWYGHa2CAPwBJYwrhpEzzlyNBlGUIAG0ZHR8q62NGaz X-Received: by 10.101.83.133 with SMTP id x5mr19264664pgq.175.1512589506109; Wed, 06 Dec 2017 11:45:06 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512589506; cv=none; d=google.com; s=arc-20160816; b=qvTtkPFgzyga2Zt5GGE8H9eYW/IMqAnkNx1IOr93Nh/QpZlpgFy0LhbUFyoXY52DEi P3M4VQazn5hV0Q3PVB+ikpQaH0rUZiDLmmb8YBUJq9EY+f8OySgO6AXHZjYKuwnTj6cg 5259Ec3W8kqenx2H9y1t+wisigrNr929AObyj5TphnMF50AL5apx5zpHFzlaCSUGUPIj S7el8HRW3v+zrYRQ5XggC1nJ6eQLmlGuPcPilKpeC9vux1YQUq2q3/FsLEvQ8Dno/IPH PUaI+oH+zjtQ6JhF3ffVf27C5zsZQwr7tWGoFF7sVBYwTScToc9CnH0MxhqFuyWD+5O/ R+BA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=3FDiYcVSZv5dJULVyfC0zAYFsVUn8WCwvOjYNWycuFA=; b=G0mco+laY/d36luhyS7JmXBE/2xNJ7UGGHPzyqlfUijHCY1IrQxfU4O/DdlesOLpLC /lhLvSNVdixHbru2CFkkrRge+K6sVmWLkpaMnPxSB+P1kMFe9ohAoUOdAHOG7XKz6VZX KycA5N5ekxDkRr29H1pWvzucaNiriQxxN2qLC0kgokWefSdz0MRxQ5KktyDAWdRW8sAm MuZ6zYEoRlKNVuukEYf8SecZdLBvnH+FM3Ysr3uOkEfIvMtpPP8ke86KLCZhDWk7nWYM HhfzDorBzO7tWvcljGSiECYoJkGmb0v/GblVY6zii4//NGjHpx9m8MTr4jWo9YHgeErL 3+iw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=DoP3Ra9J; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id i16si2382524pgv.496.2017.12.06.11.45.05; Wed, 06 Dec 2017 11:45:06 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=DoP3Ra9J; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752312AbdLFTpE (ORCPT + 4 others); Wed, 6 Dec 2017 14:45:04 -0500 Received: from mail-wr0-f196.google.com ([209.85.128.196]:44378 "EHLO mail-wr0-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752362AbdLFTov (ORCPT ); Wed, 6 Dec 2017 14:44:51 -0500 Received: by mail-wr0-f196.google.com with SMTP id l22so5101661wrc.11 for ; Wed, 06 Dec 2017 11:44:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=BL7MNGsFJHzh9iuhaPLwnIKlO2L0viHJHw1NaMe/+d0=; b=DoP3Ra9JcaGQyn2jVGzl+u90f/z3fxxRBRqkoe1OQIliGygn6N47xYs4qLmqEiDqui lp4ITXWEI3skiSyHBVRvKZaIfDokk83Bw/rB4mBsQPxU1gJ62DiBVscsZN+MjhslndE1 7IDzIKzp/lMN+ZuSy3Q3FQ1bGNoIO+f6mYwkc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=BL7MNGsFJHzh9iuhaPLwnIKlO2L0viHJHw1NaMe/+d0=; b=NYqv0yWokAKwY7hU+ahpcG1/g8HeT+E1ztc4jbVGmzYeo3RId8XlyXm6/gtjnr8tiP 4vhOfLImew+VuX7hZr1Vl5n/KDmMyJuuWK4R4bhLUnwf5Zd+iAw9yMRNw6qU1rZO527G dvVbImIBegmcyUf9OULRFq2V0Y5KeT+JzO+ezPMMk1CAmyAR3YeLUzJmeSXi2aAbHRhT 1SnoEeuyNCr29BadKyX7593GXv2KwdWpkryC7VR5SX8DLW3CJ8U2v189UdLHZVSUrdx7 dZmogyB8YptHeQogm5t66PrahyPCRmGxX7W/47OoVU/LIpQxULFp3vvZdE2zwQGrheXQ hj7A== X-Gm-Message-State: AJaThX69XE3OUmibMyneaCOOSFCki2E80BViNThPqT45F5AW7GZFvl0A 2PG8aYbwRfSicPxFQNwX4dOLYQ== X-Received: by 10.223.176.27 with SMTP id f27mr21352453wra.105.1512589490251; Wed, 06 Dec 2017 11:44:50 -0800 (PST) Received: from localhost.localdomain ([105.150.171.234]) by smtp.gmail.com with ESMTPSA id b66sm3596594wmh.32.2017.12.06.11.44.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 06 Dec 2017 11:44:49 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v3 18/20] crypto: arm64/crc32-ce - yield NEON after every block of input Date: Wed, 6 Dec 2017 19:43:44 +0000 Message-Id: <20171206194346.24393-19-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171206194346.24393-1-ard.biesheuvel@linaro.org> References: <20171206194346.24393-1-ard.biesheuvel@linaro.org> Sender: linux-rt-users-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rt-users@vger.kernel.org Avoid excessive scheduling delays under a preemptible kernel by yielding the NEON after every block of input. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/crc32-ce-core.S | 44 ++++++++++++++------ 1 file changed, 32 insertions(+), 12 deletions(-) -- 2.11.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/arch/arm64/crypto/crc32-ce-core.S b/arch/arm64/crypto/crc32-ce-core.S index 18f5a8442276..b4ddbb2027e5 100644 --- a/arch/arm64/crypto/crc32-ce-core.S +++ b/arch/arm64/crypto/crc32-ce-core.S @@ -100,9 +100,9 @@ dCONSTANT .req d0 qCONSTANT .req q0 - BUF .req x0 - LEN .req x1 - CRC .req x2 + BUF .req x19 + LEN .req x20 + CRC .req x21 vzr .req v9 @@ -116,13 +116,21 @@ * size_t len, uint crc32) */ ENTRY(crc32_pmull_le) - adr x3, .Lcrc32_constants + frame_push 4, 64 + + adr x22, .Lcrc32_constants b 0f ENTRY(crc32c_pmull_le) - adr x3, .Lcrc32c_constants + frame_push 4, 64 + + adr x22, .Lcrc32c_constants + +0: mov BUF, x0 + mov LEN, x1 + mov CRC, x2 -0: bic LEN, LEN, #15 + bic LEN, LEN, #15 ld1 {v1.16b-v4.16b}, [BUF], #0x40 movi vzr.16b, #0 fmov dCONSTANT, CRC @@ -131,7 +139,7 @@ ENTRY(crc32c_pmull_le) cmp LEN, #0x40 b.lt less_64 - ldr qCONSTANT, [x3] + ldr qCONSTANT, [x22] loop_64: /* 64 bytes Full cache line folding */ sub LEN, LEN, #0x40 @@ -161,10 +169,21 @@ loop_64: /* 64 bytes Full cache line folding */ eor v4.16b, v4.16b, v8.16b cmp LEN, #0x40 - b.ge loop_64 + b.lt less_64 + + if_will_cond_yield_neon + stp q1, q2, [sp, #48] + stp q3, q4, [sp, #80] + do_cond_yield_neon + ldp q1, q2, [sp, #48] + ldp q3, q4, [sp, #80] + ldr qCONSTANT, [x22] + movi vzr.16b, #0 + endif_yield_neon + b loop_64 less_64: /* Folding cache line into 128bit */ - ldr qCONSTANT, [x3, #16] + ldr qCONSTANT, [x22, #16] pmull2 v5.1q, v1.2d, vCONSTANT.2d pmull v1.1q, v1.1d, vCONSTANT.1d @@ -203,8 +222,8 @@ fold_64: eor v1.16b, v1.16b, v2.16b /* final 32-bit fold */ - ldr dCONSTANT, [x3, #32] - ldr d3, [x3, #40] + ldr dCONSTANT, [x22, #32] + ldr d3, [x22, #40] ext v2.16b, v1.16b, vzr.16b, #4 and v1.16b, v1.16b, v3.16b @@ -212,7 +231,7 @@ fold_64: eor v1.16b, v1.16b, v2.16b /* Finish up with the bit-reversed barrett reduction 64 ==> 32 bits */ - ldr qCONSTANT, [x3, #48] + ldr qCONSTANT, [x22, #48] and v2.16b, v1.16b, v3.16b ext v2.16b, vzr.16b, v2.16b, #8 @@ -222,6 +241,7 @@ fold_64: eor v1.16b, v1.16b, v2.16b mov w0, v1.s[1] + frame_pop 4, 64 ret ENDPROC(crc32_pmull_le) ENDPROC(crc32c_pmull_le) From patchwork Wed Dec 6 19:43:45 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 120903 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp7469368qgn; Wed, 6 Dec 2017 11:45:12 -0800 (PST) X-Google-Smtp-Source: AGs4zMYi+joX9nEkCwc2/tM7X4jEUeOUjeuDFHFameYpBj6kMdMGJSc9GPHtUvOY4CjSO5GJuKqG X-Received: by 10.99.4.11 with SMTP id 11mr22743813pge.123.1512589512213; Wed, 06 Dec 2017 11:45:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512589512; cv=none; d=google.com; s=arc-20160816; b=BpjDA9CnNXKHgkAHDOr+O75kCPsC/jthgJd40Stml4HzcMeBCKlzocdbFpGBX8zYAZ 6orWGrGozRqOch4iurkT2Z8rgnrwAdInk79okGw/qJMjpl+Tt2bZINKkasd/jfJk0TUy eCe52g+vwCtwLyloW/uXTZf/wPITgd5MSjM8xWXkNQNQio6DvzR2EOLC++g642D8EkoC 6UDppXyzBw3KcHJ+DQEnF63xw38bBPBdP7ciQJfbPcqAbF8p/GqeAcZdqVEbliwVP4gB qZcMXIoNEjwx4UEoWxAN/EoLfQJnl1gFUZtChKGamNxRmofiEyL2K4dDkWr5oKhZWovn z4+w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=92+Ph+DNoBuCn/oigy/C0SMOQGiZnU6P9vOtjvn4gmA=; b=eFVaM7r16xEnjpyTPGxx+0iXJ0SgSvuau7GlxPbo8GJJPfrU3qDwzgMYEqWWreo3Iz arTHNOXjj+jHs9mo+hGQBBO+Db1hVjVeFd3bzCdYsFGYuGjP1kqeAbUDYPKzZbOJooN+ puZAREfCO4OxBRUewmTjuNfdyNP1FqD01fsNlURx853DcQ6aabHngEKoE+KAUKbn76Ne QDbdGs88d7Dk/KP3G5K8QmrWcC2+iFWGKLof8GTANNjtb6lOQhQzkArtrFWWMYivyS5J Zz5G6R3gGFHT0UiR/3KVFZYTzYPmLlEENhn4Im/tw1na/gVZrPbVNy92yMHo5jC3Bcdb tjUg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=JjbszcL1; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id i16si2382524pgv.496.2017.12.06.11.45.11; Wed, 06 Dec 2017 11:45:12 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=JjbszcL1; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751463AbdLFTpK (ORCPT + 4 others); Wed, 6 Dec 2017 14:45:10 -0500 Received: from mail-wm0-f68.google.com ([74.125.82.68]:46944 "EHLO mail-wm0-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752501AbdLFToy (ORCPT ); Wed, 6 Dec 2017 14:44:54 -0500 Received: by mail-wm0-f68.google.com with SMTP id r78so9291454wme.5 for ; Wed, 06 Dec 2017 11:44:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=DEQY0/LoXnU9pE4NqHBI/5S5x1EkKDsFAJsw6tFA+2s=; b=JjbszcL1QLYfb9KIS8xhZQOQn2eIBxu0r6aj47MK99HUcf03NTf7obEUcPOmqNcyVt oB6igxGd/aX/rpzZqECbXOGi4RSATmwFJ1jylkBq7MZA6Mfv2sS6sz0jEQMtQy5Fys5T l6C+uoUNcfXE3UGQzP4kEzaKvkznd23ZRG2PE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=DEQY0/LoXnU9pE4NqHBI/5S5x1EkKDsFAJsw6tFA+2s=; b=ZF0QIL1hPN/p4fiAEFhk6YS/sGtqdiFEB5oT4FDAHWXFDSv5cYXs/kLwrr57S6a3Jr cKAX1nfP3szDqw02iP76kmM2cNn5O0W8FBMwlrAxFJW7d6LyPEb7oNpdOqzDEdqLUzs2 Rh2OFdo2HBkG98ljx8yWaojnZCxzisR39Ogb6uGksBIW7Y/MJnPujnbty8C3X9Oqg+jk /BLgNUWW7/aN7r9frkoZvS/drqCzMdmqB0yxa8dYJ+cPB3pfw180tFtxRKgKG2t9daqQ Rf5t6xLDtV5Cjqvutn1iY8o2BhTaeRYfZe7+E6lTibpynStTY8PqlwdZ4CJtDKMj39Zw 3YdA== X-Gm-Message-State: AKGB3mI21Ppn5Jnzbm59WxQ2kCEfGboRrhfBZjZfGZyS2YIjmG7pIaXO rmv28wValhDR3qVNjUJlbX9lpA== X-Received: by 10.28.24.210 with SMTP id 201mr8391378wmy.120.1512589493581; Wed, 06 Dec 2017 11:44:53 -0800 (PST) Received: from localhost.localdomain ([105.150.171.234]) by smtp.gmail.com with ESMTPSA id b66sm3596594wmh.32.2017.12.06.11.44.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 06 Dec 2017 11:44:52 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v3 19/20] crypto: arm64/crct10dif-ce - yield NEON after every block of input Date: Wed, 6 Dec 2017 19:43:45 +0000 Message-Id: <20171206194346.24393-20-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171206194346.24393-1-ard.biesheuvel@linaro.org> References: <20171206194346.24393-1-ard.biesheuvel@linaro.org> Sender: linux-rt-users-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rt-users@vger.kernel.org Avoid excessive scheduling delays under a preemptible kernel by yielding the NEON after every block of input. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/crct10dif-ce-core.S | 32 +++++++++++++++++--- 1 file changed, 28 insertions(+), 4 deletions(-) -- 2.11.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/arch/arm64/crypto/crct10dif-ce-core.S b/arch/arm64/crypto/crct10dif-ce-core.S index d5b5a8c038c8..111675f7bad5 100644 --- a/arch/arm64/crypto/crct10dif-ce-core.S +++ b/arch/arm64/crypto/crct10dif-ce-core.S @@ -74,13 +74,19 @@ .text .cpu generic+crypto - arg1_low32 .req w0 - arg2 .req x1 - arg3 .req x2 + arg1_low32 .req w19 + arg2 .req x20 + arg3 .req x21 vzr .req v13 ENTRY(crc_t10dif_pmull) + frame_push 3, 128 + + mov arg1_low32, w0 + mov arg2, x1 + mov arg3, x2 + movi vzr.16b, #0 // init zero register // adjust the 16-bit initial_crc value, scale it to 32 bits @@ -175,8 +181,25 @@ CPU_LE( ext v12.16b, v12.16b, v12.16b, #8 ) subs arg3, arg3, #128 // check if there is another 64B in the buffer to be able to fold - b.ge _fold_64_B_loop + b.lt _fold_64_B_end + + if_will_cond_yield_neon + stp q0, q1, [sp, #48] + stp q2, q3, [sp, #80] + stp q4, q5, [sp, #112] + stp q6, q7, [sp, #144] + do_cond_yield_neon + ldp q0, q1, [sp, #48] + ldp q2, q3, [sp, #80] + ldp q4, q5, [sp, #112] + ldp q6, q7, [sp, #144] + ldr q10, rk3 + movi vzr.16b, #0 // init zero register + endif_yield_neon + + b _fold_64_B_loop +_fold_64_B_end: // at this point, the buffer pointer is pointing at the last y Bytes // of the buffer the 64B of folded data is in 4 of the vector // registers: v0, v1, v2, v3 @@ -304,6 +327,7 @@ _barrett: _cleanup: // scale the result back to 16 bits lsr x0, x0, #16 + frame_pop 3, 128 ret _less_than_128: From patchwork Wed Dec 6 19:43:46 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 120904 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp7469409qgn; Wed, 6 Dec 2017 11:45:15 -0800 (PST) X-Google-Smtp-Source: AGs4zMZPpJMlLaA41crmXMAYonX2IwinhqF6Ogwz+kQHGKFANDfXgGeVRB4fmlQ4PAc36edwlQd8 X-Received: by 10.99.101.1 with SMTP id z1mr12990739pgb.139.1512589515063; Wed, 06 Dec 2017 11:45:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512589515; cv=none; d=google.com; s=arc-20160816; b=fmuc++6hTGz1rATPXKmGXkIywe4v6cQSinDwa8AK1johC/SMWS8mueYdQA7TsyvlGI KMnzKbXYrfHeXSwgL6dCVclIF5pQkbeSDxVxQ8l3u4ig/ZzVm3uqPKEop+eed2UqPgWx pE2P/ykqHQqRUmLgGBkrU3W/pSKLrvgFlOR0E7qQYTX0CkWQNDSJu5aDjEE1Lja8MOn2 vg2lxAWOrJBlVXDCgG4HaMtSc9K9m3UIdvk4WkrmvilgGsmJ9MbyhudVlNkSWy87V8dI BxdipHvaNNSu59uferyWdReV5Iui1vMv1RYm0/oxvZ5RUKORAImdgEU5ZvaHlRmiFlOu x5uQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=yOHq5F0sQldrOscMLdLFwmbREIDm0gUDga/IPFaHAwU=; b=U3YyzcgYB9iRvRd3FEyKLkaX0mQ3O5bHIAOmxGqlDu2mlE1Od5SvDm8eWaR/JocCK4 97bntFWk5pPH84QxsGz4d514yOELcJy5o9jrD95yGKLS1qP4YZ7Eh6OyUF6RsJZgCPQs ieNIl8mM5wJG0kobz81apm2o/g/6/8vFH6CzV4vM3y5DQhCvZBfnDDfOSQJtsFJoo0l/ pG9rVGjta5nl/BN04Y2NTHLcx+f+St4/7gVMKIuaN43lUCbqHsOm7hNNWO+EcxW7rxf4 vyb1QDIx3LeuFnzOdIPcbsX2k929YWn8++84odCVQDjyRFlG9VJiHGiyonCHcjoz9bR1 ZZUQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=M8oOMDIB; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id i16si2382524pgv.496.2017.12.06.11.45.14; Wed, 06 Dec 2017 11:45:15 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@linaro.org header.s=google header.b=M8oOMDIB; spf=pass (google.com: best guess record for domain of linux-rt-users-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-rt-users-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752127AbdLFTpO (ORCPT + 4 others); Wed, 6 Dec 2017 14:45:14 -0500 Received: from mail-wm0-f66.google.com ([74.125.82.66]:35672 "EHLO mail-wm0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752034AbdLFTo6 (ORCPT ); Wed, 6 Dec 2017 14:44:58 -0500 Received: by mail-wm0-f66.google.com with SMTP id f9so9193473wmh.0 for ; Wed, 06 Dec 2017 11:44:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Il4jJORApUVG57eL6L3+t7rXYy7wi3t835FS6Fs0UwA=; b=M8oOMDIB17slUq7gJsRk8VSsW4N5oeUHYntY9ZYq8NQiAXCv0IKwSfwWEBM2MP5mKA 6412scHYrTvFBfq62bKwtPsFeV8jRIFJUe+Fq3+0PeBv+qAjFtJQ3i3bRhy1QlZMrWbc L1nZktK95zYyv8jZ4CKFtovI+Z8lr3xgGVHko= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Il4jJORApUVG57eL6L3+t7rXYy7wi3t835FS6Fs0UwA=; b=hY6ZlgMpR/SpHapuwpA+D3GrCEQPYJdlxPCe9f103mYHIPFWOEP8RXLbLwVGoKzHzN z2FXKvvD2nRvmS7NtP8zPYOP+MPOKWByrEpVBdaIlIJ1czrmxerajbIZF/Z7erBh+dPb N/VvL26bSAE1OZr4rz4pm9aO0uQnrqXPxTcFsFc1yPyAfLO+MdgzUooGSNOqz+4RvZPx DRraCsHFKCfiu/hCSOIr6KLCDq6xRd5+1IQQQwDywp0iR/lWKjeGLQhSgy97Xbfv7nFs uWby15x6aRx2qvhbssqranJnn4iUg8wqZrs2fVnk2/p/obue9XYZMyor6TJG3LRMbjvg pZkg== X-Gm-Message-State: AKGB3mJ7A1aOW/kmWmZ4/eVbJ4cRaZeWEtZEqANKiJDThvzMw5TinpOK FpBkWxPgLrUj4xQmCGZW3a/WTg== X-Received: by 10.28.107.5 with SMTP id g5mr10082921wmc.125.1512589497585; Wed, 06 Dec 2017 11:44:57 -0800 (PST) Received: from localhost.localdomain ([105.150.171.234]) by smtp.gmail.com with ESMTPSA id b66sm3596594wmh.32.2017.12.06.11.44.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 06 Dec 2017 11:44:55 -0800 (PST) From: Ard Biesheuvel To: linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Dave Martin , Russell King - ARM Linux , Sebastian Andrzej Siewior , Mark Rutland , linux-rt-users@vger.kernel.org, Peter Zijlstra , Catalin Marinas , Will Deacon , Steven Rostedt , Thomas Gleixner Subject: [PATCH v3 20/20] DO NOT MERGE Date: Wed, 6 Dec 2017 19:43:46 +0000 Message-Id: <20171206194346.24393-21-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171206194346.24393-1-ard.biesheuvel@linaro.org> References: <20171206194346.24393-1-ard.biesheuvel@linaro.org> Sender: linux-rt-users-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rt-users@vger.kernel.org Test code to force a kernel_neon_end+begin sequence at every yield point, and wipe the entire NEON state before resuming the algorithm. --- arch/arm64/include/asm/assembler.h | 33 ++++++++++++++++++++ 1 file changed, 33 insertions(+) -- 2.11.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index c54e408fd5a7..7072c29b4e83 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -607,6 +607,7 @@ alternative_else_nop_endif cmp w1, #1 // == PREEMPT_OFFSET csel x0, x0, xzr, eq tbnz x0, #TIF_NEED_RESCHED, 5555f // needs rescheduling? + b 5555f #endif .subsection 1 5555: @@ -615,6 +616,38 @@ alternative_else_nop_endif .macro do_cond_yield_neon bl kernel_neon_end bl kernel_neon_begin + movi v0.16b, #0x55 + movi v1.16b, #0x55 + movi v2.16b, #0x55 + movi v3.16b, #0x55 + movi v4.16b, #0x55 + movi v5.16b, #0x55 + movi v6.16b, #0x55 + movi v7.16b, #0x55 + movi v8.16b, #0x55 + movi v9.16b, #0x55 + movi v10.16b, #0x55 + movi v11.16b, #0x55 + movi v12.16b, #0x55 + movi v13.16b, #0x55 + movi v14.16b, #0x55 + movi v15.16b, #0x55 + movi v16.16b, #0x55 + movi v17.16b, #0x55 + movi v18.16b, #0x55 + movi v19.16b, #0x55 + movi v20.16b, #0x55 + movi v21.16b, #0x55 + movi v22.16b, #0x55 + movi v23.16b, #0x55 + movi v24.16b, #0x55 + movi v25.16b, #0x55 + movi v26.16b, #0x55 + movi v27.16b, #0x55 + movi v28.16b, #0x55 + movi v29.16b, #0x55 + movi v30.16b, #0x55 + movi v31.16b, #0x55 .endm .macro endif_yield_neon, lbl=6666f