熱插拔 SAS 硬碟有問題。
初始資料：基於Nexenta 4.0.2伺服器的Supermicro（MB S5520HC，內部RAID控制器RMS2LL080/LSI 2008），12 HDD SAS 300G 10 HDD SATA 1T，2 SSD 160G。
磁碟分為三個池：
- 系統 SAS 2x300 鏡像
- 「快速」SAS 硬碟陣列 10 個（9 + 1 個備用）
- “慢”SATA 陣列 10 個驅動器
在某個時刻，兩個 SAS 硬碟發生故障：
NAME STATE READ WRITE CKSUM
sas DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
c1t5000C50007DCF821d0 ONLINE 0 0 0
c1t5000CCA0052FFDD5d0 ONLINE 0 0 0
c1t5000CCA005349D15d0 ONLINE 0 0 0
spare-3 FAULTED 0 0 0
c1t5000CCA00534D625d0 FAULTED 0 0 0 external device fault
c1t5000CCA0053658B5d0 ONLINE 0 0 0
c1t5000CCA00534F2D5d0 ONLINE 0 0 0
c1t5000CCA00534F33Dd0 ONLINE 0 0 0
c1t5000CCA00534F471d0 ONLINE 0 0 0
c1t5000CCA0053571D1d0 ONLINE 0 0 0
c1t5000CCA00535A3A5d0 ONLINE 0 0 0
logs
c0t500151795950C847d0 ONLINE 0 0 0
spares
c1t5000CCA0053658B5d0 INUSE currently in use
pool: syspool
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
repaired.
scan: scrub repaired 0 in 0h3m with 0 errors on Sun Oct 4 03:03:56 2015
config:
NAME STATE READ WRITE CKSUM
syspool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
c1t5000C500072BB235d0s0 FAULTED 1 20 0 external device fault
c1t5000C500072BE655d0s0 ONLINE 0 0 0
接下來，為每個磁碟執行 zpool Offline / detach 和 cfgadm -c unconfigure 替換程序。安裝完一個磁碟機（本例為syspool）後在日誌中可以看到以下內容：
Oct 9 18:28:43 nxstor genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3a40@1c/pci8086,350e@0/iport@ff/disk@w5000c50007dcf7ed,0 (sd26) online
Oct 9 18:28:43 nxstor pcplusmp: [ID 805372 kern.info] pcplusmp: pci-ide (pci-ide) instance 0 irq 0x12 vector 0x41 ioapic 0x8 intin 0x12 is bound to cpu 14
Oct 9 18:28:43 nxstor pcplusmp: [ID 805372 kern.info] pcplusmp: pci-ide (pci-ide) instance 1 irq 0x15 vector 0x42 ioapic 0x8 intin 0x15 is bound to cpu 15
Oct 9 18:28:43 nxstor pcplusmp: [ID 805372 kern.info] pcplusmp: pci-ide (pci-ide) instance 0 irq 0x12 vector 0x41 ioapic 0x8 intin 0x12 is bound to cpu 0
Oct 9 18:28:43 nxstor pcplusmp: [ID 805372 kern.info] pcplusmp: pci-ide (pci-ide) instance 1 irq 0x15 vector 0x41 ioapic 0x8 intin 0x15 is bound to cpu 1
Oct 9 18:39:53 nxstor genunix: [ID 888150 kern.warning] WARNING: Device not found in device tree. Skipping device unretire: /pci@0,0/pci8086,3a40@1c/pci8086,350e@0/iport@ff/disk@w5000c500072bb235,0
Oct 9 18:39:53 nxstor genunix: [ID 484473 kern.notice] NOTICE: Not retired: /pci@0,0/pci8086,3a40@1c/pci8086,350e@0/iport@ff/disk@w5000c500072bb235,0
Oct 9 18:39:53 nxstor genunix: [ID 888150 kern.warning] WARNING: Device not found in device tree. Skipping device unretire: /pci@0,0/pci8086,3a40@1c/pci8086,350e@0/iport@ff/disk@w5000cca00534d625,0
Disks 和 disk @ w5000cca00534d625,0 disk @ w5000c500072bb235,0 是兩個有問題的磁碟，已從系統中移除。
安裝的 HDD 定義的 cfgadm：
root@nxstor:/volumes# cfgadm -al
Ap_Id Type Receptacle Occupant Condition
Slot2 sas/hp connected configured ok
c1 scsi-sas connected configured unknown
c1::dsk/c1t5000C50007DCF7EDd0 disk connected configured unknown
<cut>
然而，這種光碟格式也看不到實用程式 fdisk：
root@nxstor:/volumes# fdisk /dev/rdsk/c1t5000C50007DCF7EDd0
fdisk: Cannot stat device /dev/rdsk/c1t5000C50007DCF7EDd0.
root@nxstor:/volumes#
root@nxstor:/volumes# ls -la /dev/rdsk/c1t5000C50007DCF7EDd0
/dev/rdsk/c1t5000C50007DCF7EDd0: No such file or directory
此外，NMS 還看到以下內容：
Trigger Name: nms-fmacheck
Fault ID: 5
Error Repeat Count: 5
Error Severity: CRITICAL
Error TimeStamp: Tue Oct 13 14:21:54 2015
Description:
FMA Module: ereport.io.scsi.disk.attach-failure
Details:
List of last errors :
Oct 13 13:48:02.6970 ereport.io.scsi.cmd.disk.tran
<cut>
Oct 13 14:21:05.7075 ereport.io.scsi.cmd.disk.tran
Oct 13 14:21:53.6196 ereport.io.scsi.cmd.disk.dev.rqs.derr
Oct 13 14:21:53.6197 ereport.io.scsi.disk.attach-failure
List of last errors :
=========: Event Details :========
SOURCE: ereport.io.scsi.disk.attach-failure
nvlist version: 0
class = ereport.io.scsi.disk.attach-failure
ena = 0x58420a3d24100401
detector = (embedded nvlist)
nvlist version: 0
version = 0x0
scheme = dev
device-path = /pci@0,0/pci8086,3a40@1c/pci8086,350e@0/iport@ff/disk@w5000c50007dcf7ed,0
devid = id1,sd@n5000c50007dcf7ef
(end detector)
devid = id1,sd@n5000c50007dcf7ef
__ttl = 0x1
__tod = 0x561cdb41 0x24f05942
=========
如果我們假設新驅動器也有缺陷，並讓其他驅動器得到更奇怪的圖片。日誌安裝盤根本不出現。但取出光碟時顯示訊息：
Oct 13 14:50:26 nxstor scsi: [ID 107833 kern.notice] /pci@0,0/pci8086,3a40@1c/pci8086,350e@0 (mpt_sas0):
Oct 13 14:50:26 nxstor PhysDiskNum 2 with DevHandle 0x23 in slot 0 for enclosure with handle 0x0 is now offline
根據經驗，我們發現該問題僅適用於 SAS 硬碟。這反過來又提示了有關 MPxIO 驅動程式和 mpt_sas 的 cfgadm 中的問題（事實證明這是已知的 Solaris 問題，並寫道她已解決，但不清楚它是否在 opensolaris 中傳遞）。
接下來做什麼？誰遇到類似的問題？
驅動程式和韌體 SAS 控制器： MPTSAS HBA 驅動程式 00.00.00.24 韌體版本 5.40.1.0 SAS 驅動器有兩種：Hitachi Ultrastar 15K300 HUS153030VLS300 和 Seagate Cheetah 15K300 HUS153030VLS300 和 Seagate Cheetah 15K.53300006530000006SS。