From patchwork Fri Jul 18 12:11:38 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Slaby X-Patchwork-Id: 33853 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-pd0-f199.google.com (mail-pd0-f199.google.com [209.85.192.199]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id CA5D420CA0 for ; Fri, 18 Jul 2014 13:56:31 +0000 (UTC) Received: by mail-pd0-f199.google.com with SMTP id ft15sf25398327pdb.2 for ; Fri, 18 Jul 2014 06:56:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:delivered-to:from:to:cc:subject :date:message-id:in-reply-to:references:in-reply-to:references :sender:precedence:list-id:x-original-sender :x-original-authentication-results:mailing-list:list-post:list-help :list-archive:list-unsubscribe; bh=zJG2RLLe7EHQ1WqWdHZuzdLJI46U0qXKFo7Q77oECgo=; b=KP0TsQklyQyq6Nkc4421m0rvfusiKAbC3MF6mkd3PB8cEYZvhY+uP13YE89y7ARM9u XtikIrVx0N0rr05g0MXu/ujy1X67LV4T3zVryWAmA/RtGAHbchPFMXYejUWugqRPIpPq 01gNJYb588d+GEJIesFTBQQmdAzksJ3AO+e/CG/Cqv8QHB0NSIQOek9d8znxfelq0hO/ 9uYncf/MmEP/zi8dX29dlMtdVOcq6tenBA7pXqJisV1CWlsjU20tqYTzympTliHVxeqz NqPH/VNpS5tUET3xLfVdE9Y8usCzu7iZ4u1TMdAKLU6g159NrgmX+uZrLBnGzybZvR6q uJDA== X-Gm-Message-State: ALoCoQlU+cyvi3p/NgOCGI8MHNdjBN0yznbTfebY0fEhreKG22wDNTaHIMmcCHRBRZK3UNuKqxb/ X-Received: by 10.66.102.68 with SMTP id fm4mr2379051pab.27.1405691790462; Fri, 18 Jul 2014 06:56:30 -0700 (PDT) MIME-Version: 1.0 X-BeenThere: patchwork-forward@linaro.org Received: by 10.140.89.116 with SMTP id u107ls961696qgd.46.gmail; Fri, 18 Jul 2014 06:56:30 -0700 (PDT) X-Received: by 10.52.129.232 with SMTP id nz8mr3022240vdb.94.1405691790343; Fri, 18 Jul 2014 06:56:30 -0700 (PDT) Received: from mail-vc0-f180.google.com (mail-vc0-f180.google.com [209.85.220.180]) by mx.google.com with ESMTPS id aq17si5757071vdc.13.2014.07.18.06.56.30 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 18 Jul 2014 06:56:30 -0700 (PDT) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.220.180 as permitted sender) client-ip=209.85.220.180; Received: by mail-vc0-f180.google.com with SMTP id ij19so7376186vcb.11 for ; Fri, 18 Jul 2014 06:56:30 -0700 (PDT) X-Received: by 10.220.44.141 with SMTP id a13mr3931614vcf.71.1405691790230; Fri, 18 Jul 2014 06:56:30 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.221.37.5 with SMTP id tc5csp13272vcb; Fri, 18 Jul 2014 06:56:29 -0700 (PDT) X-Received: by 10.68.135.67 with SMTP id pq3mr3013968pbb.165.1405691789378; Fri, 18 Jul 2014 06:56:29 -0700 (PDT) Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id jx3si5886387pbc.150.2014.07.18.06.56.22; Fri, 18 Jul 2014 06:56:22 -0700 (PDT) Received-SPF: none (google.com: linux-kernel-owner@vger.kernel.org does not designate permitted sender hosts) client-ip=209.132.180.67; Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1030831AbaGRNyx (ORCPT + 25 others); Fri, 18 Jul 2014 09:54:53 -0400 Received: from bband-dyn38.178-41-141.t-com.sk ([178.41.141.38]:17465 "EHLO ip4-83-240-18-248.cust.nbox.cz" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1030336AbaGRNJy (ORCPT ); Fri, 18 Jul 2014 09:09:54 -0400 Received: from ku by ip4-83-240-18-248.cust.nbox.cz with local (Exim 4.82) (envelope-from ) id 1X8721-0004wY-4E; Fri, 18 Jul 2014 14:12:57 +0200 From: Jiri Slaby To: stable@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Alex Elder , Jiri Slaby Subject: [PATCH 3.12 093/170] rbd: use reference counts for image requests Date: Fri, 18 Jul 2014 14:11:38 +0200 Message-Id: <1ef3eb2867700888172aec631f89275d44d583f8.1405685481.git.jslaby@suse.cz> X-Mailer: git-send-email 2.0.0 In-Reply-To: <48e8cad86bb1241c08bdaa80db022c25068ff8e0.1405685481.git.jslaby@suse.cz> References: <48e8cad86bb1241c08bdaa80db022c25068ff8e0.1405685481.git.jslaby@suse.cz> In-Reply-To: References: Sender: linux-kernel-owner@vger.kernel.org Precedence: list List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: linux-kernel-owner@vger.kernel.org X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.220.180 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , From: Alex Elder 3.12-stable review patch. If anyone has any objections, please let me know. =============== commit 0f2d5be792b0466b06797f637cfbb0f64dbb408c upstream. Each image request contains a reference count, but to date it has not actually been used. (I think this was just an oversight.) A recent report involving rbd failing an assertion shed light on why and where we need to use these reference counts. Every OSD request associated with an object request uses rbd_osd_req_callback() as its callback function. That function will call a helper function (dependent on the type of OSD request) that will set the object request's "done" flag if the object request if appropriate. If that "done" flag is set, the object request is passed to rbd_obj_request_complete(). In rbd_obj_request_complete(), requests are processed in sequential order. So if an object request completes before one of its predecessors in the image request, the completion is deferred. Otherwise, if it's a completing object's "turn" to be completed, it is passed to rbd_img_obj_end_request(), which records the result of the operation, accumulates transferred bytes, and so on. Next, the successor to this request is checked and if it is marked "done", (deferred) completion processing is performed on that request, and so on. If the last object request in an image request is completed, rbd_img_request_complete() is called, which (typically) destroys the image request. There is a race here, however. The instant an object request is marked "done" it can be provided (by a thread handling completion of one of its predecessor operations) to rbd_img_obj_end_request(), which (for the last request) can then lead to the image request getting torn down. And this can happen *before* that object has itself entered rbd_img_obj_end_request(). As a result, once it *does* enter that function, the image request (and even the object request itself) may have been freed and become invalid. All that's necessary to avoid this is to properly count references to the image requests. We tear down an image request's object requests all at once--only when the entire image request has completed. So there's no need for an image request to count references for its object requests. However, we don't want an image request to go away until the last of its object requests has passed through rbd_img_obj_callback(). In other words, we don't want rbd_img_request_complete() to necessarily result in the image request being destroyed, because it may get called before we've finished processing on all of its object requests. So the fix is to add a reference to an image request for each of its object requests. The reference can be viewed as representing an object request that has not yet finished its call to rbd_img_obj_callback(). That is emphasized by getting the reference right after assigning that as the image object's callback function. The corresponding release of that reference is done at the end of rbd_img_obj_callback(), which every image object request passes through exactly once. Signed-off-by: Alex Elder Reviewed-by: Ilya Dryomov Signed-off-by: Jiri Slaby --- drivers/block/rbd.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c index db60c91804c3..af7b44ffd190 100644 --- a/drivers/block/rbd.c +++ b/drivers/block/rbd.c @@ -1395,6 +1395,13 @@ static void rbd_obj_request_put(struct rbd_obj_request *obj_request) kref_put(&obj_request->kref, rbd_obj_request_destroy); } +static void rbd_img_request_get(struct rbd_img_request *img_request) +{ + dout("%s: img %p (was %d)\n", __func__, img_request, + atomic_read(&img_request->kref.refcount)); + kref_get(&img_request->kref); +} + static bool img_request_child_test(struct rbd_img_request *img_request); static void rbd_parent_request_destroy(struct kref *kref); static void rbd_img_request_destroy(struct kref *kref); @@ -2148,6 +2155,7 @@ static void rbd_img_obj_callback(struct rbd_obj_request *obj_request) img_request->next_completion = which; out: spin_unlock_irq(&img_request->completion_lock); + rbd_img_request_put(img_request); if (!more) rbd_img_request_complete(img_request); @@ -2244,6 +2252,7 @@ static int rbd_img_request_fill(struct rbd_img_request *img_request, goto out_partial; obj_request->osd_req = osd_req; obj_request->callback = rbd_img_obj_callback; + rbd_img_request_get(img_request); osd_req_op_extent_init(osd_req, 0, opcode, offset, length, 0, 0);