| Message ID | 20200831161854.70879-1-dwagner@suse.de |
|---|---|
| Headers | show |
| Series | qla2xxx: A couple crash fixes | expand |
Hi Daniel, On Mon, 31 Aug 2020, 9:18am, Daniel Wagner wrote: > changes since v1: > > - added dummy warn function to patch#1 > - added log entry to patch#4 > > as suggested by Martin > > > > Initial cover letter: > > The first crash we observed is due memory corruption in the srb memory > pool. Unforuntatly, I couldn't find the source of the problem but the > workaround by resetting the cleanup callbacks 'fixes' this problem > (patch #1). I think as intermeditate step this should be merged until > the real cause can be identified. > > The second crash is due a race condition(?) in the firmware. The sts > entries are not updated in time which leads to this crash pattern > which several customers have reported: > > #0 [c00000ffffd1bb80] scsi_dma_unmap at d00000001e4904d4 [scsi_mod] > #1 [c00000ffffd1bbe0] qla2x00_sp_compl at d0000000204803cc [qla2xxx] > #2 [c00000ffffd1bc20] qla24xx_process_response_queue at d0000000204c5810 [qla2xxx] > #3 [c00000ffffd1bd50] qla24xx_msix_rsp_q at d0000000204c8fd8 [qla2xxx] > #4 [c00000ffffd1bde0] __handle_irq_event_percpu at c000000000189510 > #5 [c00000ffffd1bea0] handle_irq_event_percpu at c00000000018978c > #6 [c00000ffffd1bee0] handle_irq_event at c00000000018984c > #7 [c00000ffffd1bf10] handle_fasteoi_irq at c00000000018efc0 > #8 [c00000ffffd1bf40] generic_handle_irq at c000000000187f10 > #9 [c00000ffffd1bf60] __do_irq at c000000000018784 > #10 [c00000ffffd1bf90] call_do_irq at c00000000002caa4 > #11 [c00000ecca417a00] do_IRQ at c000000000018970 > #12 [c00000ecca417a50] restore_check_irq_replay at c00000000000de98 > > From analyzing the crash dump it was clear that > qla24xx_mbx_iocb_entry() calls sp->done (qla2x00_sp_compl) which > crashes because the response is not a mailbox entry, it is a status > entry. Patch #4 changes the process logic for mailbox commands so that > the sp is parsed before calling the correct proccess function. > > Thanks, > Daniel > > Daniel Wagner (4): > qla2xxx: Warn if done() or free() are called on an already freed srb > qla2xxx: Simplify return value logic in qla2x00_get_sp_from_handle() > qla2xxx: Drop unused function argument from > qla2x00_get_sp_from_handle() > qla2xxx: Handle incorrect entry_type entries > > drivers/scsi/qla2xxx/qla_gbl.h | 3 +- > drivers/scsi/qla2xxx/qla_init.c | 10 ++++++ > drivers/scsi/qla2xxx/qla_inline.h | 5 +++ > drivers/scsi/qla2xxx/qla_isr.c | 74 +++++++++++++++++++++++---------------- > drivers/scsi/qla2xxx/qla_mr.c | 9 ++--- > 5 files changed, 62 insertions(+), 39 deletions(-) > > Thanks for the patches, my comments in the respective patches. Regards, -Arun
On Mon, 31 Aug 2020, 9:18am, Daniel Wagner wrote: > > Refactor qla2x00_get_sp_from_handle() to avoid the unecessary > goto if early returns are used. With this we can also avoid > preinitilzing the sp pointer. > > Signed-off-by: Daniel Wagner <dwagner@suse.de> > --- > drivers/scsi/qla2xxx/qla_isr.c | 8 +++----- > 1 file changed, 3 insertions(+), 5 deletions(-) > > diff --git a/drivers/scsi/qla2xxx/qla_isr.c b/drivers/scsi/qla2xxx/qla_isr.c > index 27bcd346af7c..5d278155e4e7 100644 > --- a/drivers/scsi/qla2xxx/qla_isr.c > +++ b/drivers/scsi/qla2xxx/qla_isr.c > @@ -1716,7 +1716,7 @@ qla2x00_get_sp_from_handle(scsi_qla_host_t *vha, const char *func, > { > struct qla_hw_data *ha = vha->hw; > sts_entry_t *pkt = iocb; > - srb_t *sp = NULL; > + srb_t *sp; > uint16_t index; > > index = LSW(pkt->handle); > @@ -1728,13 +1728,13 @@ qla2x00_get_sp_from_handle(scsi_qla_host_t *vha, const char *func, > set_bit(FCOE_CTX_RESET_NEEDED, &vha->dpc_flags); > else > set_bit(ISP_ABORT_NEEDED, &vha->dpc_flags); > - goto done; > + return NULL; > } > sp = req->outstanding_cmds[index]; > if (!sp) { > ql_log(ql_log_warn, vha, 0x5032, > "Invalid completion handle (%x) -- timed-out.\n", index); > - return sp; > + return NULL; > } > if (sp->handle != index) { > ql_log(ql_log_warn, vha, 0x5033, > @@ -1743,8 +1743,6 @@ qla2x00_get_sp_from_handle(scsi_qla_host_t *vha, const char *func, > } > > req->outstanding_cmds[index] = NULL; > - > -done: > return sp; > } > Looks good. Regards, -Arun
Hi Arun, On Mon, Sep 07, 2020 at 11:47:48PM -0700, Arun Easi wrote: > Could you drop the above comment about firmware, as it is speculation at > this point? Sure, no problem. > It'd be best to take a chip reset path, rather than assuming the > packet is good and having the appropriate handler called (hacky). > An approach similar to the one done at the beginning of > qla2x00_get_sp_from_handle() is what I had in mind. Ok, agreed a reset is probably the safest choice. > I'd have preferred a common approach across the different IOCB types > as an attempt to harden the code, but that will be a little more > involved work. This looks ok. Yes, I was pondering on it but I don't know enough to really come up with something reasonable. Currently our customers report only this hickup. So this is really only a partial workaround. Thanks, Daniel