• Steffen Maier's avatar
    scsi: zfcp: fix posting too many status read buffers leading to adapter shutdown · eed234bc
    Steffen Maier authored
    commit 60a161b7 upstream.
    Suppose adapter (open) recovery is between opened QDIO queues and before
    (the end of) initial posting of status read buffers (SRBs). This time
    window can be seconds long due to FSF_PROT_HOST_CONNECTION_INITIALIZING
    causing by design looping with exponential increase sleeps in the function
    performing exchange config data during recovery
    [zfcp_erp_adapter_strat_fsf_xconf()]. Recovery triggered by local link up.
    Suppose an event occurs for which the FCP channel would send an unsolicited
    notification to zfcp by means of a previously posted SRB.  We saw it with
    local cable pull (link down) in multi-initiator zoning with multiple
    NPIV-enabled subchannels of the same shared FCP channel.
    As soon as zfcp_erp_adapter_strategy_open_fsf() starts posting the initial
    status read buffers from within the adapter's ERP thread, the channel does
    send an unsolicited notification.
    Since v2.6.27 commit d26ab06e ("[SCSI] zfcp: receiving an unsolicted
    status can lead to I/O stall"), zfcp_fsf_status_read_handler() schedules
    adapter->stat_work to re-fill the just consumed SRB from a work item.
    Now the ERP thread and the work item post SRBs in parallel.  Both contexts
    call the helper function zfcp_status_read_refill().  The tracking of
    missing (to be posted / re-filled) SRBs is not thread-safe due to separate
    atomic_read() and atomic_dec(), in order to depend on posting
    success. Hence, both contexts can see
    atomic_read(&adapter->stat_miss) == 1. One of the two contexts posts
    one too many SRB. Zfcp gets QDIO_ERROR_SLSB_STATE on the output queue
    (trace tag "qdireq1") leading to zfcp_erp_adapter_shutdown() in
    An obvious and seemingly clean fix would be to schedule stat_work from the
    ERP thread and wait for it to finish. This would serialize all SRB
    re-fills. However, we already have another work item wait on the ERP
    thread: adapter->scan_work runs zfcp_fc_scan_ports() which calls
    zfcp_fc_eval_gpn_ft(). The latter calls zfcp_erp_wait() to wait for all the
    open port recoveries during zfcp auto port scan, but in fact it waits for
    any pending recovery including an adapter recovery. This approach leads to
    a deadlock.  [see also v3.19 commit 18f87a67 ("zfcp: auto port scan
    resiliency"); v2.6.37 commit d3e1088d
    ("[SCSI] zfcp: No ERP escalation on gpn_ft eval");
    v2.6.28 commit fca55b6f
    ("[SCSI] zfcp: fix deadlock between wq triggered port scan and ERP")
    fixing v2.6.27 commit c57a39a4
    ("[SCSI] zfcp: wait until adapter is finished with ERP during auto-port");
    v2.6.27 commit cc8c2829
    ("[SCSI] zfcp: Automatically attach remote ports")]
    Instead make the accounting of missing SRBs atomic for parallel execution
    in both the ERP thread and adapter->stat_work.
    Signed-off-by: default avatarSteffen Maier <maier@linux.ibm.com>
    Fixes: d26ab06e ("[SCSI] zfcp: receiving an unsolicted status can lead to I/O stall")
    Cc: <stable@vger.kernel.org> #2.6.27+
    Reviewed-by: default avatarJens Remus <jremus@linux.ibm.com>
    Signed-off-by: Martin K. Petersen's avatarMartin K. Petersen <martin.petersen@oracle.com>
    Signed-off-by: default avatarGreg Kroah-Hartman <gregkh@linuxfoundation.org>
Last commit
Last update
Makefile Loading commit data...
zfcp_aux.c Loading commit data...
zfcp_ccw.c Loading commit data...
zfcp_dbf.c Loading commit data...
zfcp_dbf.h Loading commit data...
zfcp_def.h Loading commit data...
zfcp_erp.c Loading commit data...
zfcp_ext.h Loading commit data...
zfcp_fc.c Loading commit data...
zfcp_fc.h Loading commit data...
zfcp_fsf.c Loading commit data...
zfcp_fsf.h Loading commit data...
zfcp_qdio.c Loading commit data...
zfcp_qdio.h Loading commit data...
zfcp_reqlist.h Loading commit data...
zfcp_scsi.c Loading commit data...
zfcp_sysfs.c Loading commit data...
zfcp_unit.c Loading commit data...