From patchwork Tue Jan 9 01:00:14 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 123800 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp3403168qgn; Mon, 8 Jan 2018 17:01:13 -0800 (PST) X-Google-Smtp-Source: ACJfBotE92WVViHDkx9X7FZfR9gMRabP2nv7xx0Iu5gE8RKqvoYi6EGzCsuxdkBSHrb9lWnOblN3 X-Received: by 10.200.35.78 with SMTP id b14mr20036975qtb.260.1515459673799; Mon, 08 Jan 2018 17:01:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515459673; cv=none; d=google.com; s=arc-20160816; b=ak/eqL78lOGug/kHBNIdQQUZ4hlHVjpypHM1WY7ab8IGkLq+of0vMGpM+9P0pmuveI 8qgaVGG0su+fCEmi3grpYH73WA0CpjeyuKILo0L8y9vii5Z+Nzfa2OjqLUR/Xb9iOUHq 1vnAPHvCtopFVgeWU76UfuF78Z9eNXwqXRPqY1PFYjkpoBEJsmdQnhFwsplK9ONIeDLH GckKgpEFFz+sbs17BBYBsxXGJAxwWqLJHHOToqPTkmM1ywnwEOscp6IKUFzzj3VJhEMQ hIZER6rBUBr9JkR17KfXC/ol1MOVMY4+Ro7w7+Vjsas/mO2n4GUKV21rhnLcCLqCnROJ 4DFA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=2OGkD0AY85ZteDPdpt1nb68ABskNfnHSAk6v2pPM/h4=; b=oCY/vWDTJEgG8fbSGUeL1udmhzqVPhu5PAsqMr2p4Jg84X1PtZNEk/zMy8bEwkXF1F OVujTsxgq8nsDunyxOOMPW3ufDLrdbU/Sr3lsz2T7Iv4/YfgM++cenrkScrCKJNUt2OF lqVtgN5NAyR+Hy/q51QVy7O4GvZ+DJd5Kc0V8XSm792LtFBjWK0Tz3AFSSCoYgQ6vHTz Z1XtT/2dCZRv3h4t/1tmnErm0MpAX+0uY3M3XIyZK4k2l5vZdDwNibhzAqGg8K8Et2Jb be15+RhcwoBwyaZ6OEZHmX12aZiaBERfHmSMMtC7aQsa+GUHZTl82KV5A7rtL90Lk6Ht I5BQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (ec2-54-197-127-237.compute-1.amazonaws.com. [54.197.127.237]) by mx.google.com with ESMTP id s58si673299qtc.123.2018.01.08.17.01.13; Mon, 08 Jan 2018 17:01:13 -0800 (PST) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) client-ip=54.197.127.237; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 7D1AB61700; Tue, 9 Jan 2018 01:01:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H2, URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 299B76151B; Tue, 9 Jan 2018 01:00:32 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id E3A9A616FF; Tue, 9 Jan 2018 01:00:23 +0000 (UTC) Received: from forward101p.mail.yandex.net (forward101p.mail.yandex.net [77.88.28.101]) by lists.linaro.org (Postfix) with ESMTPS id 092D06151B for ; Tue, 9 Jan 2018 01:00:22 +0000 (UTC) Received: from mxback2g.mail.yandex.net (mxback2g.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:163]) by forward101p.mail.yandex.net (Yandex) with ESMTP id 405E96A8130D for ; Tue, 9 Jan 2018 04:00:19 +0300 (MSK) Received: from smtp2o.mail.yandex.net (smtp2o.mail.yandex.net [2a02:6b8:0:1a2d::26]) by mxback2g.mail.yandex.net (nwsmtp/Yandex) with ESMTP id jESp4h3GHI-0JNKGSL4; Tue, 09 Jan 2018 04:00:19 +0300 Received: by smtp2o.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id giv4jfpZGA-0J58xVe1; Tue, 09 Jan 2018 04:00:19 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Tue, 9 Jan 2018 04:00:14 +0300 Message-Id: <1515459617-23415-2-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1515459617-23415-1-git-send-email-odpbot@yandex.ru> References: <1515459617-23415-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 342 Subject: [lng-odp] [PATCH v14 1/4] performance: add AES tests to crypto performance tests X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Dmitry Eremin-Solenikov Signed-off-by: Dmitry Eremin-Solenikov --- /** Email created from pull request 342 (lumag:openssl-ctx) ** https://github.com/Linaro/odp/pull/342 ** Patch: https://github.com/Linaro/odp/pull/342.patch ** Base sha: 49ebafae0edebbc750742d8874ad0a7588286dea ** Merge commit sha: 558f84453cf3b6b85daf9fe8a2786fc7797f4931 **/ test/performance/odp_crypto.c | 71 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 70 insertions(+), 1 deletion(-) diff --git a/test/performance/odp_crypto.c b/test/performance/odp_crypto.c index 0cbc2754e..b1c1b79fe 100644 --- a/test/performance/odp_crypto.c +++ b/test/performance/odp_crypto.c @@ -30,7 +30,7 @@ */ #define POOL_NUM_PKT 64 -static uint8_t test_iv[8] = "01234567"; +static uint8_t test_iv[16] = "0123456789abcdef"; static uint8_t test_key16[16] = { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0a, @@ -38,6 +38,12 @@ static uint8_t test_key16[16] = { 0x01, 0x02, 0x03, 0x04, 0x05, 0x10, }; +static uint8_t test_key20[20] = { 0x01, 0x02, 0x03, 0x04, 0x05, + 0x06, 0x07, 0x08, 0x09, 0x0a, + 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, + 0x10, 0x11, 0x12, 0x13, 0x14, +}; + static uint8_t test_key24[24] = { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, @@ -226,6 +232,69 @@ static crypto_alg_config_t algs_config[] = { .auth_digest_len = 12, }, }, + { + .name = "aes-cbc-null", + .session = { + .cipher_alg = ODP_CIPHER_ALG_AES_CBC, + .cipher_key = { + .data = test_key16, + .length = sizeof(test_key16) + }, + .iv = { + .data = test_iv, + .length = 16, + }, + .auth_alg = ODP_AUTH_ALG_NULL + }, + }, + { + .name = "aes-cbc-hmac-sha1-96", + .session = { + .cipher_alg = ODP_CIPHER_ALG_AES_CBC, + .cipher_key = { + .data = test_key16, + .length = sizeof(test_key16) + }, + .iv = { + .data = test_iv, + .length = 16, + }, + .auth_alg = ODP_AUTH_ALG_SHA1_HMAC, + .auth_key = { + .data = test_key20, + .length = sizeof(test_key20) + }, + .auth_digest_len = 12, + }, + }, + { + .name = "null-hmac-sha1-96", + .session = { + .cipher_alg = ODP_CIPHER_ALG_NULL, + .auth_alg = ODP_AUTH_ALG_SHA1_HMAC, + .auth_key = { + .data = test_key20, + .length = sizeof(test_key20) + }, + .auth_digest_len = 12, + }, + }, + { + .name = "aes-gcm", + .session = { + .cipher_alg = ODP_CIPHER_ALG_AES_GCM, + .cipher_key = { + .data = test_key16, + .length = sizeof(test_key16) + }, + .iv = { + .data = test_iv, + .length = 12, + }, + .auth_alg = ODP_AUTH_ALG_AES_GCM, + .auth_digest_len = 16, + }, + }, }; /** From patchwork Tue Jan 9 01:00:15 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 123801 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp3403837qgn; Mon, 8 Jan 2018 17:01:57 -0800 (PST) X-Google-Smtp-Source: ACJfBovT5r8o2hfNg13AO9ofTFBAmp+EqBzsrLYO77sPpwEIOHS87zKEwUFezsV8naQjLCW2hcRU X-Received: by 10.237.43.5 with SMTP id p5mr19522288qtd.92.1515459717620; Mon, 08 Jan 2018 17:01:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515459717; cv=none; d=google.com; s=arc-20160816; b=qGqq9wCGdR7HH3LYpfxR9Tht6J87lTmebngsEY+11tE33uW655cY/ohpTdtt0Voja5 kqFSXfkxPQCEUMMsccp/xkjvwovKOOWrsKBBSgvv9GWHxVH/TvefwXY0VRD/xcMgoQgd PYlBk4Gzhc9Csev1iFqqxlyvXb/fPl+pEQ/TJlFiK9Y68SPkH5+i4kjJL+3KQZhVbXxr pU2VtCmPCP5Zs1FFc0ofjr0Ffyqc7NgU+7t67B+oEgl4X8tMrnwC6du6SXyiJN7c+U9T WNs+Fi0LcbGPEPEmAQxeTD286fT53tF/I+HvUWHOV+fc/STJKnoGRIh1DJDeNHEEJ88L 9Ngg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=Lxtf1LQh9ne66e2XpM4RdaSlq/ogyfLlYi6zVTZrCF0=; b=ZoOE9ngg8PqqwkhNRF5qqHwgzmmxQahgudQm6xArPIB25hhRIBXXfmmD/xQedC5L4p roXn8NtvEC0xF8qOkKZAFoqMyLDMgnhR0KsvX/qgzn0NN2qS37L0+0tigd4UDd8v7Bf2 6XcGmPSgz9hfYa+L4ML29bnVph/db1bIFvwlXTvBOJedHTHVFi4JENC7FKgo2gKzSSKP sdAIwyKs2S84ienYDg1vjn5+mK+OviRuf4FQYsiHw7z01BgDpZHnCHkJLMXMjLDbugXg JYCA1CtmO0I6twiLK9fwKsftj3fQjZDuLL+ya0z01EGnEh9NWyNk8JJ0qHwpdiGKqxWR /mDw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (ec2-54-197-127-237.compute-1.amazonaws.com. [54.197.127.237]) by mx.google.com with ESMTP id y28si2717541qkj.450.2018.01.08.17.01.57; Mon, 08 Jan 2018 17:01:57 -0800 (PST) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) client-ip=54.197.127.237; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 301576171E; Tue, 9 Jan 2018 01:01:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H2, URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 7F7E661722; Tue, 9 Jan 2018 01:00:37 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 277A46171E; Tue, 9 Jan 2018 01:00:28 +0000 (UTC) Received: from forward100o.mail.yandex.net (forward100o.mail.yandex.net [37.140.190.180]) by lists.linaro.org (Postfix) with ESMTPS id 8E43B616F5 for ; Tue, 9 Jan 2018 01:00:22 +0000 (UTC) Received: from mxback6g.mail.yandex.net (mxback6g.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:167]) by forward100o.mail.yandex.net (Yandex) with ESMTP id 5247E2A213E7 for ; Tue, 9 Jan 2018 04:00:21 +0300 (MSK) Received: from smtp2o.mail.yandex.net (smtp2o.mail.yandex.net [2a02:6b8:0:1a2d::26]) by mxback6g.mail.yandex.net (nwsmtp/Yandex) with ESMTP id nV2QO6heNa-0L4WgQUJ; Tue, 09 Jan 2018 04:00:21 +0300 Received: by smtp2o.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id giv4jfpZGA-0K58mCfo; Tue, 09 Jan 2018 04:00:20 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Tue, 9 Jan 2018 04:00:15 +0300 Message-Id: <1515459617-23415-3-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1515459617-23415-1-git-send-email-odpbot@yandex.ru> References: <1515459617-23415-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 342 Subject: [lng-odp] [PATCH v14 2/4] linux-gen: crypto: move session type to odp_crypto module X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Dmitry Eremin-Solenikov There is no point in having odp_crypto_generic_session_t definition in global include file. Move it to odp_crypto module. Signed-off-by: Dmitry Eremin-Solenikov --- /** Email created from pull request 342 (lumag:openssl-ctx) ** https://github.com/Linaro/odp/pull/342 ** Patch: https://github.com/Linaro/odp/pull/342.patch ** Base sha: 49ebafae0edebbc750742d8874ad0a7588286dea ** Merge commit sha: 558f84453cf3b6b85daf9fe8a2786fc7797f4931 **/ platform/linux-generic/Makefile.am | 1 - .../linux-generic/include/odp_crypto_internal.h | 73 ---------------------- platform/linux-generic/odp_crypto.c | 46 +++++++++++++- 3 files changed, 44 insertions(+), 76 deletions(-) delete mode 100644 platform/linux-generic/include/odp_crypto_internal.h diff --git a/platform/linux-generic/Makefile.am b/platform/linux-generic/Makefile.am index 7e40448bd..e25970b0c 100644 --- a/platform/linux-generic/Makefile.am +++ b/platform/linux-generic/Makefile.am @@ -138,7 +138,6 @@ noinst_HEADERS = \ include/odp_classification_inlines.h \ include/odp_classification_internal.h \ include/odp_config_internal.h \ - include/odp_crypto_internal.h \ include/odp_debug_internal.h \ include/odp_errno_define.h \ include/odp_forward_typedefs_internal.h \ diff --git a/platform/linux-generic/include/odp_crypto_internal.h b/platform/linux-generic/include/odp_crypto_internal.h deleted file mode 100644 index 32178d9de..000000000 --- a/platform/linux-generic/include/odp_crypto_internal.h +++ /dev/null @@ -1,73 +0,0 @@ -/* Copyright (c) 2014, Linaro Limited - * All rights reserved. - * - * SPDX-License-Identifier: BSD-3-Clause - */ - -#ifndef ODP_CRYPTO_INTERNAL_H_ -#define ODP_CRYPTO_INTERNAL_H_ - -#ifdef __cplusplus -extern "C" { -#endif - -#include - -#define MAX_IV_LEN 64 -#define OP_RESULT_MAGIC 0x91919191 - -/** Forward declaration of session structure */ -typedef struct odp_crypto_generic_session odp_crypto_generic_session_t; - -/** - * Algorithm handler function prototype - */ -typedef -odp_crypto_alg_err_t (*crypto_func_t)(odp_packet_t pkt, - const odp_crypto_packet_op_param_t *param, - odp_crypto_generic_session_t *session); - -/** - * Per crypto session data structure - */ -struct odp_crypto_generic_session { - struct odp_crypto_generic_session *next; - - /* Session creation parameters */ - odp_crypto_session_param_t p; - - odp_bool_t do_cipher_first; - - struct { - /* Copy of session IV data */ - uint8_t iv_data[MAX_IV_LEN]; - uint8_t key_data[EVP_MAX_KEY_LENGTH]; - - const EVP_CIPHER *evp_cipher; - crypto_func_t func; - } cipher; - - struct { - uint8_t key[EVP_MAX_KEY_LENGTH]; - uint32_t key_length; - union { - const EVP_MD *evp_md; - const EVP_CIPHER *evp_cipher; - }; - crypto_func_t func; - } auth; -}; - -/** - * Per session creation operation result - */ -typedef struct odp_crypto_generic_session_result { - odp_crypto_ses_create_err_t rc; - odp_crypto_session_t session; -} odp_crypto_generic_session_result_t; - -#ifdef __cplusplus -} -#endif - -#endif diff --git a/platform/linux-generic/odp_crypto.c b/platform/linux-generic/odp_crypto.c index 2038aa46b..3b110c191 100644 --- a/platform/linux-generic/odp_crypto.c +++ b/platform/linux-generic/odp_crypto.c @@ -15,7 +15,6 @@ #include #include #include -#include #include #include #include @@ -89,6 +88,49 @@ static const odp_crypto_auth_capability_t auth_capa_aes_gcm[] = { static const odp_crypto_auth_capability_t auth_capa_aes_gmac[] = { {.digest_len = 16, .key_len = 16, .aad_len = {.min = 0, .max = 0, .inc = 0} } }; +/** Forward declaration of session structure */ +typedef struct odp_crypto_generic_session_t odp_crypto_generic_session_t; + +/** + * Algorithm handler function prototype + */ +typedef +odp_crypto_alg_err_t (*crypto_func_t)(odp_packet_t pkt, + const odp_crypto_packet_op_param_t *param, + odp_crypto_generic_session_t *session); + +/** + * Per crypto session data structure + */ +struct odp_crypto_generic_session_t { + odp_crypto_generic_session_t *next; + + /* Session creation parameters */ + odp_crypto_session_param_t p; + + odp_bool_t do_cipher_first; + + struct { + /* Copy of session IV data */ + uint8_t iv_data[EVP_MAX_IV_LENGTH]; + uint8_t key_data[EVP_MAX_KEY_LENGTH]; + + const EVP_CIPHER *evp_cipher; + crypto_func_t func; + } cipher; + + struct { + uint8_t key[EVP_MAX_KEY_LENGTH]; + uint32_t key_length; + uint32_t bytes; + union { + const EVP_MD *evp_md; + const EVP_CIPHER *evp_cipher; + }; + crypto_func_t func; + } auth; +}; + typedef struct odp_crypto_global_s odp_crypto_global_t; struct odp_crypto_global_s { @@ -860,7 +902,7 @@ odp_crypto_session_create(odp_crypto_session_param_t *param, /* Copy parameters */ session->p = *param; - if (session->p.iv.length > MAX_IV_LEN) { + if (session->p.iv.length > EVP_MAX_IV_LENGTH) { ODP_DBG("Maximum IV length exceeded\n"); *status = ODP_CRYPTO_SES_CREATE_ERR_INV_CIPHER; goto err; From patchwork Tue Jan 9 01:00:16 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 123802 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp3404793qgn; Mon, 8 Jan 2018 17:02:55 -0800 (PST) X-Google-Smtp-Source: ACJfBot3KLrVVsyq6U2TLvfwfZ2HS5du7LeAWMuDuVfqnxRzvl0hGpaX7QrLKnveWHJ+IrZYPkBp X-Received: by 10.237.33.1 with SMTP id 1mr19111146qtc.106.1515459775419; Mon, 08 Jan 2018 17:02:55 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515459775; cv=none; d=google.com; s=arc-20160816; b=whmHJrnzYx8vgZAdNUtSE5KNHEknDetZjJCUYofIYFn2EOxH7XoVMIr8Ja2ODjEEal kInGAwMFLSz8lFsqbKvfqJ0YNwA+8nEimhNrs5k2FeGzqE7eMhfTgEjI/vB2lKdpZPa4 foRTe0yVFtM61OQG2IumG/zRODSK/xarkghAt5T5catWfpA053QBXh1wBZAWss1han7i zHvUg4VnCF1DFqOMvkNrhDTlhAFDCtrPHqurjM3xrif9cg4Mf27DlPz3y5QzhEG3uRkg E6ixdBOXFUS1b16BUXBfV7u0I8qJGdi9lyUG0RoO1OU2RD8NFdzYTGyjAAR1raXRB8Kv ulJw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=7LXvyJB5rjsbiawGdSkZk3oc2Zng+jVqTQZkrz6C6eU=; b=ouc2djZ/g/mrkCtoh5tYoy2rfE9jbdry9tVFc3+93ZAyJmL3TqhA1gPemERmjofOnP kmnvF4zS8LVEqFWOO3liLiG1yI2nEVDi+Hnzz8sdgqfbg4PvM1U+3133xpbnrNNhY0UY evZ1TmTJ0Q1eS1EtIHd3lmR+GqUYJ0VUkMr4jrIpZYY8O4OLU7mC+hV7rLrb0aTgXRRh wakPWZFWicpaKBRbvZ8THovAnES2CaYxp++DirFidaRxEYC3JD/wjouqzPYJZ1DEhku9 64XMDKs7JQsGNlHGnjtbysuWSzK6w79XQf7iW+bXwwJZRgLPLl9QrjPXg+r9J13ZhXtg hQiA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (ec2-54-197-127-237.compute-1.amazonaws.com. [54.197.127.237]) by mx.google.com with ESMTP id k29si1445090qkl.398.2018.01.08.17.02.55; Mon, 08 Jan 2018 17:02:55 -0800 (PST) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) client-ip=54.197.127.237; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 14DE76173A; Tue, 9 Jan 2018 01:02:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id C4A816174F; Tue, 9 Jan 2018 01:00:44 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id DBC6F60949; Tue, 9 Jan 2018 01:00:29 +0000 (UTC) Received: from forward103j.mail.yandex.net (forward103j.mail.yandex.net [5.45.198.246]) by lists.linaro.org (Postfix) with ESMTPS id E9D4C60949 for ; Tue, 9 Jan 2018 01:00:23 +0000 (UTC) Received: from mxback4o.mail.yandex.net (mxback4o.mail.yandex.net [IPv6:2a02:6b8:0:1a2d::1e]) by forward103j.mail.yandex.net (Yandex) with ESMTP id 2000134C4606 for ; Tue, 9 Jan 2018 04:00:22 +0300 (MSK) Received: from smtp2o.mail.yandex.net (smtp2o.mail.yandex.net [2a02:6b8:0:1a2d::26]) by mxback4o.mail.yandex.net (nwsmtp/Yandex) with ESMTP id tdlSw8lsxN-0MXSC2kl; Tue, 09 Jan 2018 04:00:22 +0300 Received: by smtp2o.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id giv4jfpZGA-0L5uxYwu; Tue, 09 Jan 2018 04:00:21 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Tue, 9 Jan 2018 04:00:16 +0300 Message-Id: <1515459617-23415-4-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1515459617-23415-1-git-send-email-odpbot@yandex.ru> References: <1515459617-23415-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 342 Subject: [lng-odp] [PATCH v14 3/4] linux-gen: crypto: significant speedup of all operations X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Dmitry Eremin-Solenikov Per idea of Janne Peltonen, do not allocate/free crypto contexts for each operation, providing significant speed increase. Each thread on startup allocates hmac+cipher contexts pair for each crypto session. Then they are initialized on demand, when session is first executed using this thread. Signed-off-by: Dmitry Eremin-Solenikov --- /** Email created from pull request 342 (lumag:openssl-ctx) ** https://github.com/Linaro/odp/pull/342 ** Patch: https://github.com/Linaro/odp/pull/342.patch ** Base sha: 49ebafae0edebbc750742d8874ad0a7588286dea ** Merge commit sha: 558f84453cf3b6b85daf9fe8a2786fc7797f4931 **/ platform/linux-generic/include/odp_internal.h | 2 + platform/linux-generic/odp_crypto.c | 285 ++++++++++++++++++-------- platform/linux-generic/odp_init.c | 13 ++ 3 files changed, 209 insertions(+), 91 deletions(-) diff --git a/platform/linux-generic/include/odp_internal.h b/platform/linux-generic/include/odp_internal.h index 54f95b8b3..4fd721449 100644 --- a/platform/linux-generic/include/odp_internal.h +++ b/platform/linux-generic/include/odp_internal.h @@ -110,6 +110,8 @@ int odp_queue_term_global(void); int odp_crypto_init_global(void); int odp_crypto_term_global(void); +int _odp_crypto_init_local(void); +int _odp_crypto_term_local(void); int odp_timer_init_global(const odp_init_t *params); int odp_timer_term_global(void); diff --git a/platform/linux-generic/odp_crypto.c b/platform/linux-generic/odp_crypto.c index 3b110c191..486d62aad 100644 --- a/platform/linux-generic/odp_crypto.c +++ b/platform/linux-generic/odp_crypto.c @@ -9,7 +9,6 @@ #include #include #include -#include #include #include #include @@ -20,6 +19,7 @@ #include #include #include +#include #include #include @@ -129,6 +129,8 @@ struct odp_crypto_generic_session_t { }; crypto_func_t func; } auth; + + unsigned idx; }; typedef struct odp_crypto_global_s odp_crypto_global_t; @@ -137,15 +139,56 @@ struct odp_crypto_global_s { odp_spinlock_t lock; odp_crypto_generic_session_t *free; odp_crypto_generic_session_t sessions[MAX_SESSIONS]; + + /* These bitfields are cleared at alloc_session() + * together with the rest of data */ + bitset_t hmac_valid[ODP_THREAD_COUNT_MAX] + [(MAX_SESSIONS + ATOM_BITSET_SIZE - 1) / ATOM_BITSET_SIZE]; + bitset_t cipher_valid[ODP_THREAD_COUNT_MAX] + [(MAX_SESSIONS + ATOM_BITSET_SIZE - 1) / ATOM_BITSET_SIZE]; + bitset_t mac_cipher_valid[ODP_THREAD_COUNT_MAX] + [(MAX_SESSIONS + ATOM_BITSET_SIZE - 1) / ATOM_BITSET_SIZE]; + odp_ticketlock_t openssl_lock[0]; }; static odp_crypto_global_t *global; +typedef struct crypto_local_t { + HMAC_CTX *hmac_ctx[MAX_SESSIONS]; + EVP_CIPHER_CTX *cipher_ctx[MAX_SESSIONS]; + EVP_CIPHER_CTX *mac_cipher_ctx[MAX_SESSIONS]; + bitset_t *hmac_valid; + bitset_t *cipher_valid; + bitset_t *mac_cipher_valid; +} crypto_local_t; + +static __thread crypto_local_t local; + +typedef enum { KIND_HMAC, KIND_CIPHER, KIND_MAC_CIPHER } crypto_kind_t; + +static inline int crypto_should_init(odp_crypto_generic_session_t *session, + crypto_kind_t kind) +{ + bitset_t *ptr = kind == KIND_HMAC ? local.hmac_valid : + kind == KIND_CIPHER ? local.cipher_valid : + local.mac_cipher_valid; + bitset_t *cv = ptr + (session->idx / ATOM_BITSET_SIZE); + bitset_t cur = atom_bitset_load(cv, __ATOMIC_ACQUIRE); + + if (bitset_is_set(cur, session->idx % ATOM_BITSET_SIZE)) + return false; + + atom_bitset_set(cv, session->idx % ATOM_BITSET_SIZE, __ATOMIC_RELEASE); + + return true; +} + static odp_crypto_generic_session_t *alloc_session(void) { odp_crypto_generic_session_t *session = NULL; + unsigned i; odp_spinlock_lock(&global->lock); session = global->free; @@ -155,6 +198,27 @@ odp_crypto_generic_session_t *alloc_session(void) } odp_spinlock_unlock(&global->lock); + session->idx = session - global->sessions; + + for (i = 0; i < ODP_THREAD_COUNT_MAX; i++) { + bitset_t *cv; + + cv = global->hmac_valid[i] + + (session->idx / ATOM_BITSET_SIZE); + atom_bitset_clr(cv, session->idx % ATOM_BITSET_SIZE, + __ATOMIC_ACQ_REL); + + cv = global->cipher_valid[i] + + (session->idx / ATOM_BITSET_SIZE); + atom_bitset_clr(cv, session->idx % ATOM_BITSET_SIZE, + __ATOMIC_ACQ_REL); + + cv = global->mac_cipher_valid[i] + + (session->idx / ATOM_BITSET_SIZE); + atom_bitset_clr(cv, session->idx % ATOM_BITSET_SIZE, + __ATOMIC_ACQ_REL); + } + return session; } @@ -175,24 +239,51 @@ null_crypto_routine(odp_packet_t pkt ODP_UNUSED, return ODP_CRYPTO_ALG_ERR_NONE; } +/* Mimic new OpenSSL 1.1.y API */ +#if OPENSSL_VERSION_NUMBER < 0x10100000L +static HMAC_CTX *HMAC_CTX_new(void) +{ + HMAC_CTX *ctx = malloc(sizeof(*ctx)); + + HMAC_CTX_init(ctx); + return ctx; +} + +static void HMAC_CTX_free(HMAC_CTX *ctx) +{ + HMAC_CTX_cleanup(ctx); + free(ctx); +} +#endif + static -void packet_hmac_calculate(HMAC_CTX *ctx, - odp_packet_t pkt, - const odp_crypto_packet_op_param_t *param, - odp_crypto_generic_session_t *session, - uint8_t *hash) +void packet_hmac(odp_packet_t pkt, + const odp_crypto_packet_op_param_t *param, + odp_crypto_generic_session_t *session, + uint8_t *hash) { + HMAC_CTX *ctx = local.hmac_ctx[session->idx]; uint32_t offset = param->auth_range.offset; uint32_t len = param->auth_range.length; ODP_ASSERT(offset + len <= odp_packet_len(pkt)); - HMAC_Init_ex(ctx, - session->auth.key, - session->auth.key_length, - session->auth.evp_md, - NULL); + if (crypto_should_init(session, KIND_HMAC)) { + HMAC_Init_ex(ctx, + session->auth.key, + session->auth.key_length, + session->auth.evp_md, + NULL); + } else { + /* Reinitialize HMAC calculation without resetting the key */ + HMAC_Init_ex(ctx, + NULL, + 0, + NULL, + NULL); + } + /* Hash it */ while (len > 0) { uint32_t seglen = 0; /* GCC */ void *mapaddr = odp_packet_offset(pkt, offset, &seglen, NULL); @@ -206,36 +297,6 @@ void packet_hmac_calculate(HMAC_CTX *ctx, HMAC_Final(ctx, hash, NULL); } -#if OPENSSL_VERSION_NUMBER < 0x10100000L -static -void packet_hmac(odp_packet_t pkt, - const odp_crypto_packet_op_param_t *param, - odp_crypto_generic_session_t *session, - uint8_t *hash) -{ - HMAC_CTX ctx; - - /* Hash it */ - HMAC_CTX_init(&ctx); - packet_hmac_calculate(&ctx, pkt, param, session, hash); - HMAC_CTX_cleanup(&ctx); -} -#else -static -void packet_hmac(odp_packet_t pkt, - const odp_crypto_packet_op_param_t *param, - odp_crypto_generic_session_t *session, - uint8_t *hash) -{ - HMAC_CTX *ctx; - - /* Hash it */ - ctx = HMAC_CTX_new(); - packet_hmac_calculate(ctx, pkt, param, session, hash); - HMAC_CTX_free(ctx); -} -#endif - static odp_crypto_alg_err_t auth_gen(odp_packet_t pkt, const odp_crypto_packet_op_param_t *param, @@ -430,7 +491,7 @@ odp_crypto_alg_err_t cipher_encrypt(odp_packet_t pkt, const odp_crypto_packet_op_param_t *param, odp_crypto_generic_session_t *session) { - EVP_CIPHER_CTX *ctx; + EVP_CIPHER_CTX *ctx = local.cipher_ctx[session->idx]; void *iv_ptr; int ret; @@ -442,16 +503,15 @@ odp_crypto_alg_err_t cipher_encrypt(odp_packet_t pkt, return ODP_CRYPTO_ALG_ERR_IV_INVALID; /* Encrypt it */ - ctx = EVP_CIPHER_CTX_new(); - EVP_EncryptInit_ex(ctx, session->cipher.evp_cipher, NULL, - session->cipher.key_data, NULL); + if (crypto_should_init(session, KIND_CIPHER)) { + EVP_EncryptInit_ex(ctx, session->cipher.evp_cipher, NULL, + session->cipher.key_data, NULL); + EVP_CIPHER_CTX_set_padding(ctx, 0); + } EVP_EncryptInit_ex(ctx, NULL, NULL, NULL, iv_ptr); - EVP_CIPHER_CTX_set_padding(ctx, 0); ret = internal_encrypt(ctx, pkt, param); - EVP_CIPHER_CTX_free(ctx); - return ret <= 0 ? ODP_CRYPTO_ALG_ERR_DATA_SIZE : ODP_CRYPTO_ALG_ERR_NONE; } @@ -461,7 +521,7 @@ odp_crypto_alg_err_t cipher_decrypt(odp_packet_t pkt, const odp_crypto_packet_op_param_t *param, odp_crypto_generic_session_t *session) { - EVP_CIPHER_CTX *ctx; + EVP_CIPHER_CTX *ctx = local.cipher_ctx[session->idx]; void *iv_ptr; int ret; @@ -473,16 +533,15 @@ odp_crypto_alg_err_t cipher_decrypt(odp_packet_t pkt, return ODP_CRYPTO_ALG_ERR_IV_INVALID; /* Decrypt it */ - ctx = EVP_CIPHER_CTX_new(); - EVP_DecryptInit_ex(ctx, session->cipher.evp_cipher, NULL, - session->cipher.key_data, NULL); + if (crypto_should_init(session, KIND_CIPHER)) { + EVP_DecryptInit_ex(ctx, session->cipher.evp_cipher, NULL, + session->cipher.key_data, NULL); + EVP_CIPHER_CTX_set_padding(ctx, 0); + } EVP_DecryptInit_ex(ctx, NULL, NULL, NULL, iv_ptr); - EVP_CIPHER_CTX_set_padding(ctx, 0); ret = internal_decrypt(ctx, pkt, param); - EVP_CIPHER_CTX_free(ctx); - return ret <= 0 ? ODP_CRYPTO_ALG_ERR_DATA_SIZE : ODP_CRYPTO_ALG_ERR_NONE; } @@ -519,7 +578,7 @@ odp_crypto_alg_err_t aes_gcm_encrypt(odp_packet_t pkt, const odp_crypto_packet_op_param_t *param, odp_crypto_generic_session_t *session) { - EVP_CIPHER_CTX *ctx; + EVP_CIPHER_CTX *ctx = local.cipher_ctx[session->idx]; const uint8_t *aad_head = param->aad_ptr; uint32_t aad_len = session->p.auth_aad_len; void *iv_ptr; @@ -535,13 +594,14 @@ odp_crypto_alg_err_t aes_gcm_encrypt(odp_packet_t pkt, return ODP_CRYPTO_ALG_ERR_IV_INVALID; /* Encrypt it */ - ctx = EVP_CIPHER_CTX_new(); - EVP_EncryptInit_ex(ctx, session->cipher.evp_cipher, NULL, - session->cipher.key_data, NULL); - EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, - session->p.iv.length, NULL); + if (crypto_should_init(session, KIND_CIPHER)) { + EVP_EncryptInit_ex(ctx, session->cipher.evp_cipher, NULL, + session->cipher.key_data, NULL); + EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, + session->p.iv.length, NULL); + EVP_CIPHER_CTX_set_padding(ctx, 0); + } EVP_EncryptInit_ex(ctx, NULL, NULL, NULL, iv_ptr); - EVP_CIPHER_CTX_set_padding(ctx, 0); /* Authenticate header data (if any) without encrypting them */ if (aad_len > 0) @@ -555,8 +615,6 @@ odp_crypto_alg_err_t aes_gcm_encrypt(odp_packet_t pkt, odp_packet_copy_from_mem(pkt, param->hash_result_offset, session->p.auth_digest_len, block); - EVP_CIPHER_CTX_free(ctx); - return ret <= 0 ? ODP_CRYPTO_ALG_ERR_DATA_SIZE : ODP_CRYPTO_ALG_ERR_NONE; } @@ -566,7 +624,7 @@ odp_crypto_alg_err_t aes_gcm_decrypt(odp_packet_t pkt, const odp_crypto_packet_op_param_t *param, odp_crypto_generic_session_t *session) { - EVP_CIPHER_CTX *ctx; + EVP_CIPHER_CTX *ctx = local.cipher_ctx[session->idx]; const uint8_t *aad_head = param->aad_ptr; uint32_t aad_len = session->p.auth_aad_len; int dummy_len = 0; @@ -582,13 +640,14 @@ odp_crypto_alg_err_t aes_gcm_decrypt(odp_packet_t pkt, return ODP_CRYPTO_ALG_ERR_IV_INVALID; /* Decrypt it */ - ctx = EVP_CIPHER_CTX_new(); - EVP_DecryptInit_ex(ctx, session->cipher.evp_cipher, NULL, - session->cipher.key_data, NULL); - EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, - session->p.iv.length, NULL); + if (crypto_should_init(session, KIND_CIPHER)) { + EVP_DecryptInit_ex(ctx, session->cipher.evp_cipher, NULL, + session->cipher.key_data, NULL); + EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, + session->p.iv.length, NULL); + EVP_CIPHER_CTX_set_padding(ctx, 0); + } EVP_DecryptInit_ex(ctx, NULL, NULL, NULL, iv_ptr); - EVP_CIPHER_CTX_set_padding(ctx, 0); odp_packet_copy_to_mem(pkt, param->hash_result_offset, session->p.auth_digest_len, block); @@ -602,8 +661,6 @@ odp_crypto_alg_err_t aes_gcm_decrypt(odp_packet_t pkt, ret = internal_decrypt(ctx, pkt, param); - EVP_CIPHER_CTX_free(ctx); - return ret <= 0 ? ODP_CRYPTO_ALG_ERR_ICV_CHECK : ODP_CRYPTO_ALG_ERR_NONE; } @@ -635,7 +692,7 @@ odp_crypto_alg_err_t aes_gmac_gen(odp_packet_t pkt, const odp_crypto_packet_op_param_t *param, odp_crypto_generic_session_t *session) { - EVP_CIPHER_CTX *ctx; + EVP_CIPHER_CTX *ctx = local.mac_cipher_ctx[session->idx]; void *iv_ptr; uint8_t block[EVP_MAX_MD_SIZE]; int ret; @@ -648,13 +705,14 @@ odp_crypto_alg_err_t aes_gmac_gen(odp_packet_t pkt, return ODP_CRYPTO_ALG_ERR_IV_INVALID; /* Encrypt it */ - ctx = EVP_CIPHER_CTX_new(); - EVP_EncryptInit_ex(ctx, session->auth.evp_cipher, NULL, - session->auth.key, NULL); - EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, - session->p.iv.length, NULL); + if (crypto_should_init(session, KIND_MAC_CIPHER)) { + EVP_EncryptInit_ex(ctx, session->auth.evp_cipher, NULL, + session->auth.key, NULL); + EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, + session->p.iv.length, NULL); + EVP_CIPHER_CTX_set_padding(ctx, 0); + } EVP_EncryptInit_ex(ctx, NULL, NULL, NULL, iv_ptr); - EVP_CIPHER_CTX_set_padding(ctx, 0); ret = internal_aad(ctx, pkt, param); @@ -663,8 +721,6 @@ odp_crypto_alg_err_t aes_gmac_gen(odp_packet_t pkt, odp_packet_copy_from_mem(pkt, param->hash_result_offset, session->p.auth_digest_len, block); - EVP_CIPHER_CTX_free(ctx); - return ret <= 0 ? ODP_CRYPTO_ALG_ERR_DATA_SIZE : ODP_CRYPTO_ALG_ERR_NONE; } @@ -674,7 +730,7 @@ odp_crypto_alg_err_t aes_gmac_check(odp_packet_t pkt, const odp_crypto_packet_op_param_t *param, odp_crypto_generic_session_t *session) { - EVP_CIPHER_CTX *ctx; + EVP_CIPHER_CTX *ctx = local.mac_cipher_ctx[session->idx]; void *iv_ptr; uint8_t block[EVP_MAX_MD_SIZE]; int ret; @@ -687,13 +743,14 @@ odp_crypto_alg_err_t aes_gmac_check(odp_packet_t pkt, return ODP_CRYPTO_ALG_ERR_IV_INVALID; /* Decrypt it */ - ctx = EVP_CIPHER_CTX_new(); - EVP_DecryptInit_ex(ctx, session->auth.evp_cipher, NULL, - session->auth.key, NULL); - EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, - session->p.iv.length, NULL); + if (crypto_should_init(session, KIND_MAC_CIPHER)) { + EVP_DecryptInit_ex(ctx, session->auth.evp_cipher, NULL, + session->auth.key, NULL); + EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, + session->p.iv.length, NULL); + EVP_CIPHER_CTX_set_padding(ctx, 0); + } EVP_DecryptInit_ex(ctx, NULL, NULL, NULL, iv_ptr); - EVP_CIPHER_CTX_set_padding(ctx, 0); odp_packet_copy_to_mem(pkt, param->hash_result_offset, session->p.auth_digest_len, block); @@ -704,8 +761,6 @@ odp_crypto_alg_err_t aes_gmac_check(odp_packet_t pkt, ret = internal_aad(ctx, pkt, param); - EVP_CIPHER_CTX_free(ctx); - return ret <= 0 ? ODP_CRYPTO_ALG_ERR_ICV_CHECK : ODP_CRYPTO_ALG_ERR_NONE; } @@ -1216,6 +1271,54 @@ int odp_crypto_term_global(void) return rc; } +int _odp_crypto_init_local(void) +{ + unsigned i; + int id; + + memset(&local, 0, sizeof(local)); + + for (i = 0; i < MAX_SESSIONS; i++) { + local.hmac_ctx[i] = HMAC_CTX_new(); + local.cipher_ctx[i] = EVP_CIPHER_CTX_new(); + local.mac_cipher_ctx[i] = EVP_CIPHER_CTX_new(); + + if (local.hmac_ctx[i] == NULL || + local.cipher_ctx[i] == NULL || + local.mac_cipher_ctx[i] == NULL) { + _odp_crypto_term_local(); + return -1; + } + } + + /* Clear local valid bits */ + id = odp_thread_id(); + local.hmac_valid = global->hmac_valid[id]; + local.cipher_valid = global->cipher_valid[id]; + local.mac_cipher_valid = global->mac_cipher_valid[id]; + memset(local.hmac_valid, 0, sizeof(global->hmac_valid[0])); + memset(local.cipher_valid, 0, sizeof(global->cipher_valid[0])); + memset(local.mac_cipher_valid, 0, sizeof(global->mac_cipher_valid[0])); + + return 0; +} + +int _odp_crypto_term_local(void) +{ + unsigned i; + + for (i = 0; i < MAX_SESSIONS; i++) { + if (local.hmac_ctx[i] != NULL) + HMAC_CTX_free(local.hmac_ctx[i]); + if (local.cipher_ctx[i] != NULL) + EVP_CIPHER_CTX_free(local.cipher_ctx[i]); + if (local.mac_cipher_ctx[i] != NULL) + EVP_CIPHER_CTX_free(local.mac_cipher_ctx[i]); + } + + return 0; +} + odp_random_kind_t odp_random_max_kind(void) { return ODP_RANDOM_CRYPTO; diff --git a/platform/linux-generic/odp_init.c b/platform/linux-generic/odp_init.c index bff8303ee..bbcbd8dd3 100644 --- a/platform/linux-generic/odp_init.c +++ b/platform/linux-generic/odp_init.c @@ -327,6 +327,12 @@ int odp_init_local(odp_instance_t instance, odp_thread_type_t thr_type) } stage = PKTIO_INIT; + if (_odp_crypto_init_local()) { + ODP_ERR("ODP crypto local init failed.\n"); + goto init_fail; + } + stage = CRYPTO_INIT; + if (odp_pool_init_local()) { ODP_ERR("ODP pool local init failed.\n"); goto init_fail; @@ -379,6 +385,13 @@ int _odp_term_local(enum init_stage stage) } /* Fall through */ + case CRYPTO_INIT: + if (_odp_crypto_term_local()) { + ODP_ERR("ODP crypto local term failed.\n"); + rc = -1; + } + /* Fall through */ + case POOL_INIT: if (odp_pool_term_local()) { ODP_ERR("ODP buffer pool local term failed.\n"); From patchwork Tue Jan 9 01:00:17 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 123803 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp3406111qgn; Mon, 8 Jan 2018 17:04:24 -0800 (PST) X-Google-Smtp-Source: ACJfBovwe8cATrrWL+s+masCnjvmWxBdJDXqIqTPhnWmBQE4iGot3O/hhDadXiOgBocqoyJgzbIl X-Received: by 10.200.23.18 with SMTP id w18mr19847286qtj.131.1515459864660; Mon, 08 Jan 2018 17:04:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515459864; cv=none; d=google.com; s=arc-20160816; b=ZyHTqeuhjKlW2v/xdnFPvntErsmMX8gMjD00PjrWS3VE01AuU8asJAyK3TxbPMZv3a Zdza2OYjY/OSHtUVdvn5z/STzGDYXN7YDilVP5B7NvgdVVzLNUVp0MRNubGqqA3396Qt sCJpLZzygDHDPWlWhKLI5LLReuEA2e3SSVH2DAcKPFpgdsG560MI78PbDogIYyNg4DNY 9DDWZqMPUG99BUgbSRw13TR7alXUFHsIxsL6T1uejYVxZhpezyHzRXH5Y5aKOtNohW7Y Rra4eOzNLvxlZf5OuTxSHuwQkkm4dpnj4jZ+BZOAB5TkHSLK+XkTB9OLfuIj2X8k8l0i eUvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=LZAz6so49DH4rpIZm+uyiu1r15qxnMQgFLSonKLHGjs=; b=jSkzAmZIhFtNMP0w+ROnG4AnL6V1DDXdg5wpRBFpywSmq5JU30HGhBG/cGMYpJcenx m7FJCpOpGBK4njAtvnjoMi61x8msd2EAhx+7Gh/sM4OGmFVNwnsAuhLhglatqD6yZNuk aIHSOX39ay0bneqFpp7BrCN9m0UqSWhN6rHpyqNtYt3aMFsNAnGKlNZmoTVmT39sM+XC 47owqPeA3bMRU4erEZbtJpIDxI9xkgZUHQCApk0NFT40Ivz0r9z8+OgLYbozg6Rco0T7 65lPWPaERGXS79+67pySx0HsvXVOeMMg/YvqlMYBbpP8CvSPAsu2/N9mL4TOmAhsT71G PzVA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (ec2-54-197-127-237.compute-1.amazonaws.com. [54.197.127.237]) by mx.google.com with ESMTP id h184si235489qkc.340.2018.01.08.17.04.24; Mon, 08 Jan 2018 17:04:24 -0800 (PST) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) client-ip=54.197.127.237; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id 4EA486174F; Tue, 9 Jan 2018 01:04:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 493D761755; Tue, 9 Jan 2018 01:00:51 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id B269B6151B; Tue, 9 Jan 2018 01:00:31 +0000 (UTC) Received: from forward106o.mail.yandex.net (forward106o.mail.yandex.net [37.140.190.187]) by lists.linaro.org (Postfix) with ESMTPS id 1A2356151B for ; Tue, 9 Jan 2018 01:00:25 +0000 (UTC) Received: from mxback11g.mail.yandex.net (mxback11g.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:90]) by forward106o.mail.yandex.net (Yandex) with ESMTP id D6BB1781326 for ; Tue, 9 Jan 2018 04:00:22 +0300 (MSK) Received: from smtp2o.mail.yandex.net (smtp2o.mail.yandex.net [2a02:6b8:0:1a2d::26]) by mxback11g.mail.yandex.net (nwsmtp/Yandex) with ESMTP id m2lMhUxyLU-0MZqZ1ux; Tue, 09 Jan 2018 04:00:22 +0300 Received: by smtp2o.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id giv4jfpZGA-0M5KmTwU; Tue, 09 Jan 2018 04:00:22 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Tue, 9 Jan 2018 04:00:17 +0300 Message-Id: <1515459617-23415-5-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1515459617-23415-1-git-send-email-odpbot@yandex.ru> References: <1515459617-23415-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 342 Subject: [lng-odp] [PATCH v14 4/4] linux-gen: crypto: simplify local ctx init X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Dmitry Eremin-Solenikov Replace several bitfields with single bitfield. Use callbacks to initialize context data. Signed-off-by: Dmitry Eremin-Solenikov --- /** Email created from pull request 342 (lumag:openssl-ctx) ** https://github.com/Linaro/odp/pull/342 ** Patch: https://github.com/Linaro/odp/pull/342.patch ** Base sha: 49ebafae0edebbc750742d8874ad0a7588286dea ** Merge commit sha: 558f84453cf3b6b85daf9fe8a2786fc7797f4931 **/ platform/linux-generic/odp_crypto.c | 222 ++++++++++++++++++++---------------- 1 file changed, 124 insertions(+), 98 deletions(-) diff --git a/platform/linux-generic/odp_crypto.c b/platform/linux-generic/odp_crypto.c index 486d62aad..0017b2d31 100644 --- a/platform/linux-generic/odp_crypto.c +++ b/platform/linux-generic/odp_crypto.c @@ -98,6 +98,7 @@ typedef odp_crypto_alg_err_t (*crypto_func_t)(odp_packet_t pkt, const odp_crypto_packet_op_param_t *param, odp_crypto_generic_session_t *session); +typedef void (*crypto_init_func_t)(odp_crypto_generic_session_t *session); /** * Per crypto session data structure @@ -117,6 +118,7 @@ struct odp_crypto_generic_session_t { const EVP_CIPHER *evp_cipher; crypto_func_t func; + crypto_init_func_t init; } cipher; struct { @@ -128,6 +130,7 @@ struct odp_crypto_generic_session_t { const EVP_CIPHER *evp_cipher; }; crypto_func_t func; + crypto_init_func_t init; } auth; unsigned idx; @@ -140,13 +143,9 @@ struct odp_crypto_global_s { odp_crypto_generic_session_t *free; odp_crypto_generic_session_t sessions[MAX_SESSIONS]; - /* These bitfields are cleared at alloc_session() + /* This bitfield is cleared at alloc_session() * together with the rest of data */ - bitset_t hmac_valid[ODP_THREAD_COUNT_MAX] - [(MAX_SESSIONS + ATOM_BITSET_SIZE - 1) / ATOM_BITSET_SIZE]; - bitset_t cipher_valid[ODP_THREAD_COUNT_MAX] - [(MAX_SESSIONS + ATOM_BITSET_SIZE - 1) / ATOM_BITSET_SIZE]; - bitset_t mac_cipher_valid[ODP_THREAD_COUNT_MAX] + bitset_t ctx_valid[ODP_THREAD_COUNT_MAX] [(MAX_SESSIONS + ATOM_BITSET_SIZE - 1) / ATOM_BITSET_SIZE]; odp_ticketlock_t openssl_lock[0]; @@ -158,30 +157,26 @@ typedef struct crypto_local_t { HMAC_CTX *hmac_ctx[MAX_SESSIONS]; EVP_CIPHER_CTX *cipher_ctx[MAX_SESSIONS]; EVP_CIPHER_CTX *mac_cipher_ctx[MAX_SESSIONS]; - bitset_t *hmac_valid; - bitset_t *cipher_valid; - bitset_t *mac_cipher_valid; + bitset_t *ctx_valid; } crypto_local_t; static __thread crypto_local_t local; typedef enum { KIND_HMAC, KIND_CIPHER, KIND_MAC_CIPHER } crypto_kind_t; -static inline int crypto_should_init(odp_crypto_generic_session_t *session, - crypto_kind_t kind) +static inline void crypto_init(odp_crypto_generic_session_t *session) { - bitset_t *ptr = kind == KIND_HMAC ? local.hmac_valid : - kind == KIND_CIPHER ? local.cipher_valid : - local.mac_cipher_valid; + bitset_t *ptr = local.ctx_valid; bitset_t *cv = ptr + (session->idx / ATOM_BITSET_SIZE); bitset_t cur = atom_bitset_load(cv, __ATOMIC_ACQUIRE); if (bitset_is_set(cur, session->idx % ATOM_BITSET_SIZE)) - return false; + return; - atom_bitset_set(cv, session->idx % ATOM_BITSET_SIZE, __ATOMIC_RELEASE); + session->cipher.init(session); + session->auth.init(session); - return true; + atom_bitset_set(cv, session->idx % ATOM_BITSET_SIZE, __ATOMIC_RELEASE); } static @@ -203,17 +198,7 @@ odp_crypto_generic_session_t *alloc_session(void) for (i = 0; i < ODP_THREAD_COUNT_MAX; i++) { bitset_t *cv; - cv = global->hmac_valid[i] + - (session->idx / ATOM_BITSET_SIZE); - atom_bitset_clr(cv, session->idx % ATOM_BITSET_SIZE, - __ATOMIC_ACQ_REL); - - cv = global->cipher_valid[i] + - (session->idx / ATOM_BITSET_SIZE); - atom_bitset_clr(cv, session->idx % ATOM_BITSET_SIZE, - __ATOMIC_ACQ_REL); - - cv = global->mac_cipher_valid[i] + + cv = global->ctx_valid[i] + (session->idx / ATOM_BITSET_SIZE); atom_bitset_clr(cv, session->idx % ATOM_BITSET_SIZE, __ATOMIC_ACQ_REL); @@ -239,6 +224,12 @@ null_crypto_routine(odp_packet_t pkt ODP_UNUSED, return ODP_CRYPTO_ALG_ERR_NONE; } +static void +null_crypto_init_routine(odp_crypto_generic_session_t *session ODP_UNUSED) +{ + return; +} + /* Mimic new OpenSSL 1.1.y API */ #if OPENSSL_VERSION_NUMBER < 0x10100000L static HMAC_CTX *HMAC_CTX_new(void) @@ -256,6 +247,18 @@ static void HMAC_CTX_free(HMAC_CTX *ctx) } #endif +static void +auth_init(odp_crypto_generic_session_t *session) +{ + HMAC_CTX *ctx = local.hmac_ctx[session->idx]; + + HMAC_Init_ex(ctx, + session->auth.key, + session->auth.key_length, + session->auth.evp_md, + NULL); +} + static void packet_hmac(odp_packet_t pkt, const odp_crypto_packet_op_param_t *param, @@ -268,20 +271,8 @@ void packet_hmac(odp_packet_t pkt, ODP_ASSERT(offset + len <= odp_packet_len(pkt)); - if (crypto_should_init(session, KIND_HMAC)) { - HMAC_Init_ex(ctx, - session->auth.key, - session->auth.key_length, - session->auth.evp_md, - NULL); - } else { - /* Reinitialize HMAC calculation without resetting the key */ - HMAC_Init_ex(ctx, - NULL, - 0, - NULL, - NULL); - } + /* Reinitialize HMAC calculation without resetting the key */ + HMAC_Init_ex(ctx, NULL, 0, NULL, NULL); /* Hash it */ while (len > 0) { @@ -486,6 +477,16 @@ int internal_decrypt(EVP_CIPHER_CTX *ctx, return ret; } +static void +cipher_encrypt_init(odp_crypto_generic_session_t *session) +{ + EVP_CIPHER_CTX *ctx = local.cipher_ctx[session->idx]; + + EVP_EncryptInit_ex(ctx, session->cipher.evp_cipher, NULL, + session->cipher.key_data, NULL); + EVP_CIPHER_CTX_set_padding(ctx, 0); +} + static odp_crypto_alg_err_t cipher_encrypt(odp_packet_t pkt, const odp_crypto_packet_op_param_t *param, @@ -502,12 +503,6 @@ odp_crypto_alg_err_t cipher_encrypt(odp_packet_t pkt, else return ODP_CRYPTO_ALG_ERR_IV_INVALID; - /* Encrypt it */ - if (crypto_should_init(session, KIND_CIPHER)) { - EVP_EncryptInit_ex(ctx, session->cipher.evp_cipher, NULL, - session->cipher.key_data, NULL); - EVP_CIPHER_CTX_set_padding(ctx, 0); - } EVP_EncryptInit_ex(ctx, NULL, NULL, NULL, iv_ptr); ret = internal_encrypt(ctx, pkt, param); @@ -516,6 +511,16 @@ odp_crypto_alg_err_t cipher_encrypt(odp_packet_t pkt, ODP_CRYPTO_ALG_ERR_NONE; } +static void +cipher_decrypt_init(odp_crypto_generic_session_t *session) +{ + EVP_CIPHER_CTX *ctx = local.cipher_ctx[session->idx]; + + EVP_DecryptInit_ex(ctx, session->cipher.evp_cipher, NULL, + session->cipher.key_data, NULL); + EVP_CIPHER_CTX_set_padding(ctx, 0); +} + static odp_crypto_alg_err_t cipher_decrypt(odp_packet_t pkt, const odp_crypto_packet_op_param_t *param, @@ -532,12 +537,6 @@ odp_crypto_alg_err_t cipher_decrypt(odp_packet_t pkt, else return ODP_CRYPTO_ALG_ERR_IV_INVALID; - /* Decrypt it */ - if (crypto_should_init(session, KIND_CIPHER)) { - EVP_DecryptInit_ex(ctx, session->cipher.evp_cipher, NULL, - session->cipher.key_data, NULL); - EVP_CIPHER_CTX_set_padding(ctx, 0); - } EVP_DecryptInit_ex(ctx, NULL, NULL, NULL, iv_ptr); ret = internal_decrypt(ctx, pkt, param); @@ -565,14 +564,29 @@ static int process_cipher_param(odp_crypto_generic_session_t *session, session->p.cipher_key.length); /* Set function */ - if (ODP_CRYPTO_OP_ENCODE == session->p.op) + if (ODP_CRYPTO_OP_ENCODE == session->p.op) { session->cipher.func = cipher_encrypt; - else + session->cipher.init = cipher_encrypt_init; + } else { session->cipher.func = cipher_decrypt; + session->cipher.init = cipher_decrypt_init; + } return 0; } +static void +aes_gcm_encrypt_init(odp_crypto_generic_session_t *session) +{ + EVP_CIPHER_CTX *ctx = local.cipher_ctx[session->idx]; + + EVP_EncryptInit_ex(ctx, session->cipher.evp_cipher, NULL, + session->cipher.key_data, NULL); + EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, + session->p.iv.length, NULL); + EVP_CIPHER_CTX_set_padding(ctx, 0); +} + static odp_crypto_alg_err_t aes_gcm_encrypt(odp_packet_t pkt, const odp_crypto_packet_op_param_t *param, @@ -593,14 +607,6 @@ odp_crypto_alg_err_t aes_gcm_encrypt(odp_packet_t pkt, else return ODP_CRYPTO_ALG_ERR_IV_INVALID; - /* Encrypt it */ - if (crypto_should_init(session, KIND_CIPHER)) { - EVP_EncryptInit_ex(ctx, session->cipher.evp_cipher, NULL, - session->cipher.key_data, NULL); - EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, - session->p.iv.length, NULL); - EVP_CIPHER_CTX_set_padding(ctx, 0); - } EVP_EncryptInit_ex(ctx, NULL, NULL, NULL, iv_ptr); /* Authenticate header data (if any) without encrypting them */ @@ -619,6 +625,18 @@ odp_crypto_alg_err_t aes_gcm_encrypt(odp_packet_t pkt, ODP_CRYPTO_ALG_ERR_NONE; } +static void +aes_gcm_decrypt_init(odp_crypto_generic_session_t *session) +{ + EVP_CIPHER_CTX *ctx = local.cipher_ctx[session->idx]; + + EVP_DecryptInit_ex(ctx, session->cipher.evp_cipher, NULL, + session->cipher.key_data, NULL); + EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, + session->p.iv.length, NULL); + EVP_CIPHER_CTX_set_padding(ctx, 0); +} + static odp_crypto_alg_err_t aes_gcm_decrypt(odp_packet_t pkt, const odp_crypto_packet_op_param_t *param, @@ -639,14 +657,6 @@ odp_crypto_alg_err_t aes_gcm_decrypt(odp_packet_t pkt, else return ODP_CRYPTO_ALG_ERR_IV_INVALID; - /* Decrypt it */ - if (crypto_should_init(session, KIND_CIPHER)) { - EVP_DecryptInit_ex(ctx, session->cipher.evp_cipher, NULL, - session->cipher.key_data, NULL); - EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, - session->p.iv.length, NULL); - EVP_CIPHER_CTX_set_padding(ctx, 0); - } EVP_DecryptInit_ex(ctx, NULL, NULL, NULL, iv_ptr); odp_packet_copy_to_mem(pkt, param->hash_result_offset, @@ -679,14 +689,29 @@ static int process_aes_gcm_param(odp_crypto_generic_session_t *session, session->cipher.evp_cipher = cipher; /* Set function */ - if (ODP_CRYPTO_OP_ENCODE == session->p.op) + if (ODP_CRYPTO_OP_ENCODE == session->p.op) { session->cipher.func = aes_gcm_encrypt; - else + session->cipher.init = aes_gcm_encrypt_init; + } else { session->cipher.func = aes_gcm_decrypt; + session->cipher.init = aes_gcm_decrypt_init; + } return 0; } +static void +aes_gmac_gen_init(odp_crypto_generic_session_t *session) +{ + EVP_CIPHER_CTX *ctx = local.mac_cipher_ctx[session->idx]; + + EVP_EncryptInit_ex(ctx, session->auth.evp_cipher, NULL, + session->auth.key, NULL); + EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, + session->p.iv.length, NULL); + EVP_CIPHER_CTX_set_padding(ctx, 0); +} + static odp_crypto_alg_err_t aes_gmac_gen(odp_packet_t pkt, const odp_crypto_packet_op_param_t *param, @@ -704,14 +729,6 @@ odp_crypto_alg_err_t aes_gmac_gen(odp_packet_t pkt, else return ODP_CRYPTO_ALG_ERR_IV_INVALID; - /* Encrypt it */ - if (crypto_should_init(session, KIND_MAC_CIPHER)) { - EVP_EncryptInit_ex(ctx, session->auth.evp_cipher, NULL, - session->auth.key, NULL); - EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, - session->p.iv.length, NULL); - EVP_CIPHER_CTX_set_padding(ctx, 0); - } EVP_EncryptInit_ex(ctx, NULL, NULL, NULL, iv_ptr); ret = internal_aad(ctx, pkt, param); @@ -725,6 +742,18 @@ odp_crypto_alg_err_t aes_gmac_gen(odp_packet_t pkt, ODP_CRYPTO_ALG_ERR_NONE; } +static void +aes_gmac_check_init(odp_crypto_generic_session_t *session) +{ + EVP_CIPHER_CTX *ctx = local.mac_cipher_ctx[session->idx]; + + EVP_DecryptInit_ex(ctx, session->auth.evp_cipher, NULL, + session->auth.key, NULL); + EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, + session->p.iv.length, NULL); + EVP_CIPHER_CTX_set_padding(ctx, 0); +} + static odp_crypto_alg_err_t aes_gmac_check(odp_packet_t pkt, const odp_crypto_packet_op_param_t *param, @@ -742,14 +771,6 @@ odp_crypto_alg_err_t aes_gmac_check(odp_packet_t pkt, else return ODP_CRYPTO_ALG_ERR_IV_INVALID; - /* Decrypt it */ - if (crypto_should_init(session, KIND_MAC_CIPHER)) { - EVP_DecryptInit_ex(ctx, session->auth.evp_cipher, NULL, - session->auth.key, NULL); - EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, - session->p.iv.length, NULL); - EVP_CIPHER_CTX_set_padding(ctx, 0); - } EVP_DecryptInit_ex(ctx, NULL, NULL, NULL, iv_ptr); odp_packet_copy_to_mem(pkt, param->hash_result_offset, @@ -779,10 +800,13 @@ static int process_aes_gmac_param(odp_crypto_generic_session_t *session, session->auth.evp_cipher = cipher; /* Set function */ - if (ODP_CRYPTO_OP_ENCODE == session->p.op) + if (ODP_CRYPTO_OP_ENCODE == session->p.op) { session->auth.func = aes_gmac_gen; - else + session->auth.init = aes_gmac_gen_init; + } else { session->auth.func = aes_gmac_check; + session->auth.init = aes_gmac_check_init; + } return 0; } @@ -796,6 +820,7 @@ static int process_auth_param(odp_crypto_generic_session_t *session, session->auth.func = auth_gen; else session->auth.func = auth_check; + session->auth.init = auth_init; session->auth.evp_md = evp_md; @@ -978,6 +1003,7 @@ odp_crypto_session_create(odp_crypto_session_param_t *param, switch (param->cipher_alg) { case ODP_CIPHER_ALG_NULL: session->cipher.func = null_crypto_routine; + session->cipher.init = null_crypto_init_routine; rc = 0; break; case ODP_CIPHER_ALG_3DES_CBC: @@ -1053,6 +1079,7 @@ odp_crypto_session_create(odp_crypto_session_param_t *param, switch (param->auth_alg) { case ODP_AUTH_ALG_NULL: session->auth.func = null_crypto_routine; + session->auth.init = null_crypto_init_routine; rc = 0; break; #if ODP_DEPRECATED_API @@ -1092,6 +1119,7 @@ odp_crypto_session_create(odp_crypto_session_param_t *param, * cipher at the same time */ if (param->cipher_alg == ODP_CIPHER_ALG_AES_GCM || aes_gcm) { session->auth.func = null_crypto_routine; + session->auth.init = null_crypto_init_routine; rc = 0; } else { rc = -1; @@ -1293,12 +1321,8 @@ int _odp_crypto_init_local(void) /* Clear local valid bits */ id = odp_thread_id(); - local.hmac_valid = global->hmac_valid[id]; - local.cipher_valid = global->cipher_valid[id]; - local.mac_cipher_valid = global->mac_cipher_valid[id]; - memset(local.hmac_valid, 0, sizeof(global->hmac_valid[0])); - memset(local.cipher_valid, 0, sizeof(global->cipher_valid[0])); - memset(local.mac_cipher_valid, 0, sizeof(global->mac_cipher_valid[0])); + local.ctx_valid = global->ctx_valid[id]; + memset(local.ctx_valid, 0, sizeof(global->ctx_valid[0])); return 0; } @@ -1488,6 +1512,8 @@ int odp_crypto_int(odp_packet_t pkt_in, pkt_in = ODP_PACKET_INVALID; } + crypto_init(session); + /* Invoke the functions */ if (session->do_cipher_first) { rc_cipher = session->cipher.func(out_pkt, param, session);