HPE 3PAR 更换硬盘详解（servicemag）
1. 了解servicemag start/resume 工作原理
2. 如何修复Degraded状态磁盘
3. 如何更换NORMAL磁盘的步骤
4. 更换磁盘过程可能遇到的情况总结
1. 了解servicemag start/resume 工作原理
我们首先要了解更换磁盘的操作就是通过执行servicemag命令来将有问题的磁盘更换。针对F/T/Vclass 我们需要通过手动方式执行servicemag start –log 将有问题的磁盘数据迁移到spare/freespace，更换完磁盘后再通过servicemag resume将数据回签到新替换的磁盘上。针对SS7000，系统会自动执行servicemag start将有问题的磁盘里的数据迁移到spare/freespace, 工程师更换完磁盘后系统会自动运行servicemag resume将数据回迁到新盘。
下面我们具体谈谈servicemag的工作原理：其实servicemag start 或servicemag resume 命令是由一系列的sub-command组成的脚本。
我们先来谈servicemag start 过程:
a. 首先存储要将这块盘的 ldalloc 标记为off ，目的是host io 不能访问这块有问题的磁盘，存储会执行 ‘setpd ldalloc off <pdid>’
b. 将这块盘的数据全部迁移到free/spare space，这时存储执行的是movepd<PDID>
c. 然后将磁盘停止转动，执行的是’controlpd spindown <disk WWPN>
d. 最后将这块盘从LOOP中剔除，执行的命令是 ‘controlmag offloop <cage id> <mag>’
接下来我们谈servicemag resume 过程：
a. 更换完磁盘后我们先要将磁盘加载到LOOP环境，执行的命令是
‘controlmag onloop <cage id> <magazine>’
b. 让磁盘转动起来，执行的命令是： ‘controlpd spindup <DISK WWPN>
c. 这时要将原来移出去的数据回迁到这块新盘上，执行的命令是：
‘ moverelocpd <NEW PDID>’ showpdch –from oldpdid
d. 最后将这块新换的磁盘能让HOST IO 继续访问，执行的命令是：
’ setpd ldalloc on <NEW PDID>’
以上就是更换磁盘的servicemag工作原理。
2. 如何修复Degraded状态磁盘
了解更换磁盘后的原理，我们要知道什么时候去更换磁盘，什么条件才需要更换磁盘。
我们知道当一块物理盘中超过6chunklets标示为坏的时候，物理磁盘会显示Failed 状态，这时我们才需要更换磁盘，如果只是’degraded’状态我们是不需要更换的。
那我们如果遇到degraded状态的时候，改如何处理：多数情况是由于磁盘只有单loop情况下磁盘状态才会变为degraded状态，而不是物理盘体坏。
1. 首先我们执行 ‘showpd –degraded –failed’ 输出，检查哪些磁盘是degraded状态或failed状态。
2. 然后我们执行 ‘showpd–i ‘找出有问题磁盘的WWPN
3. 这时执行： ‘controlpd clearerr <drive wwn>’
4. 等待60秒后，执行 ‘controlmag offloop –disk <pos> cage<num><mag num>’
5. 再等待60 秒后，执行 ‘controlmag onloop –disk <pos> cage<num><magnum>’
6. 接着等待60秒后，再清除有问题磁盘的错误信息 ‘controlpd clearerr <drive wwn>’
7. 如果问题依旧，就需要升级做进一步分析。
建议：如果遇到的是failed状态的磁盘，也可以尝试按照degraded磁盘的操作步骤，如果还是failed状态就更换磁盘。
案例如下：
cli% controlmag offloop -disk 3 cage13 0
cli% controlmag on loop -disk 3 cage13 0
cli% controlpd clearerr 2210000A33006FB6
3. 如何更换NORMAL磁盘的步骤
有的客户在操作系统层面发现有许多SCSI IO错误信息，强烈要求更换磁盘，但此时我们的磁盘状态都是NORMAL状态的，原则上我们是不予更换的。但如果客户非要更换避免潜在风险，我们就不能按照servicemag start/resume的命令来操作，这点切记，如果执行了，也会报错，执行不下去。
下面的步骤是在一个客户V-Class下更换的NORMAL磁盘步骤：
a. 将要更换的磁盘 ldalloc 置成 off
V400 cli% setpd ldalloc off 28
V400 cli% showpd -c 28
------- Normal Chunklets -------- ---- Spare Chunklets ----
- Used - -------- Unused -------- - Used - ---- Unused ----
Id CagePos Type State Total OK FailFree Uninit Unavail Fail OK Fail Free Uninit Fail
28 3:9:0 FC degraded 272 56 0 0 0 216 0 0 0 0 0 0
------------------------------------------------------------------------------------------
1total 272 56 0 0 0 216 0 0 0 0 0 0
V400 cli% showpd -c
-------- Normal Chunklets -------- ---- Spare Chunklets ----
- Used -- -------- Unused -------- - Used - ---- Unused ----
Id CagePos Type State Total OKFail Free Uninit Unavail Fail OK Fail Free Uninit Fail
0 0:0:0 FC normal 272 58 0 180 0 0 0 0 0 34 0 0
... ...
28 3:9:0 FC degraded 272 56 0 0 0 216 0 0 0 0 0 0
29 3:9:1 FC normal 272 58 0 180 0 0 0 0 0 34 0 0
30 3:9:2 FC normal 272 57 0 181 0 0 0 0 0 34 0 0
31 3:9:3 FC normal 272 58 0 180 0 0 0 0 0 34 0 0
-------------------------------------------------------------------------------------------
32total 8704 1869 0 5565 0 216 0 0 0 1054 0 0
b. 将要更换的磁盘的chunklet 迁移到Spare Space
V400 cli% movepdtospare -nowait 28
Are you sure you want to move the pd to spare?
select q=quit y=yes n=no: y
Move -State-- ----------------------Detailed_State-----------------------
28:0-18:218 normal ch_move_pending,normal
28:1-29:231 degradedch_move_pending,remote_disks_used
28:2-24:220 normal ch_move_pending,normal
......
28:267-6:235 normal ch_move_pending,normal
28:268-6:236 normal ch_move_pending,normal
28:269-6:237 normal ch_move_pending,normal
28:270-6:238 normal ch_move_pending,normal
28:271-6:239 normal ch_move_pending,normal
c. 我们通过 showpdch <pdid>查看要更换的磁盘是否已经将chunklet 移出去了
V400 cli% showpdch 28
Pdid Chnk LdName LdCh State Usage Media Sp Cl From To
28 0 tp-15-sa-0.0 0 normal relsrc valid N N ---18:218
28 1 tp-15-sd-0.0 72 normal relsrc valid N N ---29:231
... ...
28 269 log0.0 31normal relsrc valid N N --- 6:237
-------------------------------------------------------------------
Total chunklets: 54
d.查看磁盘状态：
V400 cli% showpd -c
-------- Normal Chunklets -------- ---- Spare Chunklets ----
- Used -- -------- Unused -------- - Used - ---- Unused ----
Id CagePos Type State Total OKFail Free Uninit Unavail Fail OK Fail Free Uninit Fail
0 0:0:0 FC normal 272 58 0 180 0 0 0 2 0 32 0 0
... ...
27 3:0:3 FC normal 272 59 0 179 0 0 0 6 0 28 0 0
28 3:9:0 FC degraded 272 52 0 0 0 220 0 0 0 0 0 0
29 3:9:1 FC normal 272 58 0 180 0 0 0 6 0 28 0 0
30 3:9:2 FC normal 272 57 0 181 0 0 0 3 0 31 0 0
31 3:9:3 FC normal 272 58 0 180 0 0 0 0 0 34 0 0
-------------------------------------------------------------------------------------------
32total 8704 1865 0 5565 0 220 0 56 0 998 0 0
e.我们这时才执行 servicemag start命令
V400 cli% servicemag start -log -pdid 28
Are you sure you want to run servicemag?
select q=quit y=yes n=no: y
servicemag start -log -pdid 28
... servicing disks in mag: 3 9
... normal disks: WWN[2000B452537B9D40] Id [28] diskpos [0]
.................... WWN[2000B452537B9DC4] Id [29] diskpos [1]
.................... WWN[2000B452537B99C1] Id [30] diskpos [2]
.................... WWN[2000B452537B9988] Id [31] diskpos [3]
... not normal disks:
The servicemag start operation will continue in thebackground.
V400 cli% servicemag status
Cage 3, magazine 9:
The magazine is being brought offline due to a servicemagstart.
The last status update was at Sun Feb 15 11:25:34 2015.
Unable to provide a relocation estimate
servicemag start -log -pdid 28 -- is in Progress
V400 cli% showpd -c
-------- Normal Chunklets -------- ---- Spare Chunklets ----
- Used -- -------- Unused -------- - Used - ---- Unused ----
Id CagePos Type State Total OK Fail FreeUninit Unavail Fail OK Fail Free Uninit Fail
0 0:0:0 FC normal 272 58 0 180 0 0 0 3 0 31 0 0
... ...
27 3:0:3 FC normal 272 59 0 179 0 0 0 6 0 28 0 0
28 3:9:0 FC normal 272 0 0 0 0 272 0 0 0 0 0 0
29 3:9:1 FC normal 272 58 0 0 0 208 0 6 0 0 0 0
30 3:9:2 FC normal 272 57 0 0 0 212 0 3 0 0 0 0
31 3:9:3 FC normal 272 58 0 0 0 214 0 0 0 0 0 0
-----------------------------------------------------------------------------------------
32total 8704 1813 0 5024 0 906 0 100 0 861 0 0
f. 检查servicemag status :
V400 cli% servicemag status
Cage 3, magazine 9:
The magazine is being brought offline due to a servicemagstart.
The last status update was at Sun Feb 15 11:25:34 2015.
Unable to provide a relocation estimate
servicemag start -log -pdid 28 -- is in Progress
V400 cli% servicemag status -d
Cage 3, magazine 9:
The magazine is being brought offline due to a servicemagstart.
The last status update was at Sun Feb 15 11:25:34 2015.
Unable to provide a relocation estimate
The cumulative output so far is:
servicemag start -log -pdid 28
... servicing disks in mag: 3 9
... normal disks: WWN[2000B452537B9D40] Id [28] diskpos [0]
.................... WWN[2000B452537B9DC4] Id [29] diskpos [1]
.................... WWN[2000B452537B99C1] Id [30] diskpos [2]
.................... WWN[2000B452537B9988] Id [31] diskpos [3]
... not normal disks:
... normal disk WWN [2000B452537B9D40] Id [28] and notlogging it
... vacant disks: WWN[2000B452537B9D40] Id [28] diskpos [0]
... relocating chunklets from degraded raid sets to sparespace
g. 通过showpdch <pdid>检查这个要更换的磁盘是否已经全部迁移出去了：
V400 cli% showpdch 28
No chunklet information available.
h. servicemag start正常运行结束：
V400 cli% servicemag status -d
Cage 3, magazine 9:
The magazine was successfully brought offline by a servicemagstart command.
The command completed Sun Feb 15 11:32:34 2015.
The output of the servicemag start was:
servicemag start -log -pdid 28
... servicing disks in mag: 3 9
... normal disks: WWN[2000B452537B9D40] Id [28] diskpos [0]
.................... WWN[2000B452537B9DC4] Id [29] diskpos [1]
.................... WWN[2000B452537B99C1] Id [30] diskpos [2]
.................... WWN[2000B452537B9988] Id [31] diskpos [3]
... not normal disks:
... normal disk WWN [2000B452537B9D40] Id [28] and notlogging it
... vacant disks: WWN[2000B452537B9D40] Id [28] diskpos [0]
... relocating chunklets from degraded raid sets to sparespace
... logging chunklets from pd WWN [2000B452537B9DC4] Id [29]
... logging chunklets from pd WWN [2000B452537B99C1] Id [30]
... logging chunklets from pd WWN [2000B452537B9988] Id [31]
... retrying logging chunklets...
... spinning down disk WWN [2000B452537B9D40] Id [28]
... spinning down disk WWN [2000B452537B9DC4] Id [29]
... spinning down disk WWN [2000B452537B99C1] Id [30]
... spinning down disk WWN [2000B452537B9988] Id [31]
... bypassing mag 3 9
... bypassed mag 3 9
servicemag start -log -pdid 28 --Succeeded
V400 cli% showpd -c
-------- Normal Chunklets -------- ---- Spare Chunklets ----
- Used -- -------- Unused -------- - Used - ---- Unused ----
Id CagePos Type State Total OKFail Free Uninit Unavail Fail OK Fail Free Uninit Fail
0 0:0:0 FC normal 272 58 0 180 0 0 0 3 0 31 0 0
... ...
28 3:9:0? FC degraded 272 0 0 0 0 272 0 0 0 0 0 0
29 3:9:1? FC degraded 272 0 45 0 0 221 0 0 6 0 0 0
30 3:9:2? FC degraded 272 0 41 0 0 228 0 0 3 0 0 0
31 3:9:3? FC degraded 272 0 43 0 0 229 0 0 0 0 0 0
-------------------------------------------------------------------------------------------
32total 8704 1640 129 5024 0 950 0 91 9 861 0 0
i. 更换PDID28 磁盘：确保要更换的磁盘所在的magazine 四个硬盘状态灯为绿灯。
j. 更换完磁盘后会出现一个新盘，但没有盘的PDID：
V400 cli% showpd -c
-------- Normal Chunklets -------- ---- Spare Chunklets ----
- Used -- -------- Unused -------- - Used - ---- Unused ----
Id CagePos Type State Total OKFail Free Uninit Unavail Fail OK Fail Free Uninit Fail
--- 3:9:0 FC degraded 272 0 0 0 0 0 0 0 0 0 0 0
28 3:9:0 FC degraded 272 0 0 0 0 272 0 0 0 0 0 0
29 3:9:1 FC degraded 272 0 41 0 0 225 0 0 6 0 0 0
30 3:9:2 FC degraded 272 0 38 0 0 231 0 0 3 0 0 0
31 3:9:3 FC degraded 272 0 40 0 0 232 0 0 0 0 0 0
--------------------------------------------------------------------------------------------
33total 8976 1640 119 5024 0 960 0 101 9 851 0 0
k. 需要手动将这个盘添加进来：
V400 cli% showpd -i
Id CagePos State ----Node_WWN---- --MFR-------Model------ -Serial- -FW_Rev- Protocol MediaType
--- 3:9:0 degraded 5000CCA017A49568HITACHI HVIPC0300GBFC15K JXXXHZ3N 3P02 FC Magnetic
... ...
28 3:9:0 degraded 2000B452537B9D40 SEAGATESEGLE0300GBFC15K 6SJ415EY 3P00 FC Magnetic
29 3:9:1 degraded 2000B452537B9DC4 SEAGATESEGLE0300GBFC15K 6SJ41A0N 3P00 FC Magnetic
30 3:9:2 degraded 2000B452537B99C1 SEAGATESEGLE0300GBFC15K 6SJ419D1 3P00 FC Magnetic
31 3:9:3 degraded 2000B452537B9988 SEAGATESEGLE0300GBFC15K 6SJ4198V 3P00 FC Magnetic
---------------------------------------------------------------------------------------------------
33total
V400 cli% admitpd 5000CCA017A49568
Warning: The following disks are marked as being part ofanother system.
Admitting these disks to this system will cause any existingdata on them
to be erased.
-Disk_WWN- System_IDSystem_Name
5000CCA017A49568 5322 0
Are you sure?
select y=yes n=no: y
l. 检查新添加这个盘有一个新的PDID：（我们这里出现新的PDID 32）
V400 cli% showpd -c
-------- Normal Chunklets -------- ---- Spare Chunklets ----
-Used -- -------- Unused -------- - Used - ---- Unused ----
Id CagePos Type State Total OKFail Free Uninit Unavail Fail OK Fail Free Uninit Fail
28 3:9:0 FC degraded 272 0 0 0 0 272 0 0 0 0 0 0
29 3:9:1 FC degraded 272 0 41 0 0 225 0 0 6 0 0 0
30 3:9:2 FC degraded 272 0 38 0 0 231 0 0 3 0 0 0
31 3:9:3 FC degraded 272 0 40 0 0 232 0 0 0 0 0 0
32 3:9:0 FC degraded 272 0 0 0 272 0 0 0 0 0 0 0
-------------------------------------------------------------------------------------------
33total 8976 1640 119 5024 272 960 0 101 9 851 0 0
m.确保能识别到所有磁盘：
V400 cli% showpd
---Size(MB)--------Ports----
Id CagePos Type RPM State Total Free A B Capacity(GB)
... ...
28 3:9:0 FC 15 degraded 278528 0 ----- ----- 300
29 3:9:1 FC 15normal 278528 00:6:4 1:6:4* 300
30 3:9:2 FC 15normal 278528 0 0:6:4*1:6:4 300
31 3:9:3 FC 15normal 278528 00:6:4 1:6:4* 300
32 3:9:0 FC 15 degraded 278528 278528 0:6:4*1:6:4 300
-----------------------------------------------------------------------
33total 91914245423104
n. 这时需要执行servicemag resume将数据回迁到新盘：
V400 cli% servicemag resume 3 9
Are you sure you want to run servicemag?
select q=quit y=yes n=no: y
servicemag resume 3 9
... mag 3 9 already onlooped
... firmware is current on pd WWN [2000B452537B9DC4] Id [29]
... firmware is current on pd WWN [2000B452537B99C1] Id [30]
... firmware is current on pd WWN [2000B452537B9988] Id [31]
... firmware is current on pd WWN [5000CCA017A49568] Id [32]
... firmware is current on pd WWN [2000B452537B9D40] Id [28]
... checking for valid disks...
... disks in mag : 3 9
... normal disks: WWN[2000B452537B9DC4] Id [29] diskpos [1]
.................... WWN[2000B452537B99C1] Id [30] diskpos [2]
.................... WWN[2000B452537B9988] Id [31] diskpos [3]
.................... WWN[5000CCA017A49568] Id [32] diskpos [0]
... not normal disks: WWN [2000B452537B9D40] Id[28]
... verifying spare space for disks 29 and 29
... verifying spare space for disks 30 and 30
... verifying spare space for disks 31 and 31
... verifying spare space for disks 28 and 32
... playback chunklets from pd WWN [2000B452537B9DC4] Id[29]
... playback chunklets from pd WWN [2000B452537B99C1] Id[30]
... playback chunklets from pd WWN [2000B452537B9988] Id[31]
... playback chunklets from pd WWN [5000CCA017A49568] Id[32]
The servicemag resume operation will continue in thebackground.
o. 确认没有servicemag 进程，说明数据回迁成功：
V400 cli% servicemag status -d
No servicemag operations logged.
V400 cli% showpd -c
-------- Normal Chunklets -------- ---- Spare Chunklets ----
- Used -- -------- Unused -------- - Used - ---- Unused ----
Id CagePos Type State Total OK Fail FreeUninit Unavail Fail OK Fail Free Uninit Fail
29 3:9:1 FC normal 272 58 0 180 0 0 0 0 0 32 2 0
30 3:9:2 FC normal 272 57 0 181 0 0 0 0 0 34 0 0
31 3:9:3 FC normal 272 58 0 180 0 0 0 0 0 34 0 0
32 3:9:0 FC normal 272 56 0 138 44 0 0 0 0 0 34 0
-----------------------------------------------------------------------------------------
32total 8704 1869 0 5703 44 0 0 0 0 1043 45 0
4. 更换磁盘过程可能遇到的情况总结
以上总结了更换磁盘的工作原理，以及修复degraded状态磁盘和normal状态磁盘的过程。下面我们会列举更换磁盘过程失败的处理方法：其实我们在执行servicemagstart/resume 失败后，会单独执行里面的子命令来解决问题。
1. 当做servicemag resume失败后我们如何处理：
cli% servicemag status -d
Cage 1, magazine 12:
A servicemag resume command failed on thismagazine.
The command completed at WedApr 9 05:37:33 2014.
The output of the servicemag resume was:servicemag resume 1 12
... mag 1 12 already onlooped
... firmware is current on pd WWN[xxxxxxxxxxxxxxxx] Id [yy]
... firmware is current on pd WWN[xxxxxxxxxxxxxxxx] Id [zz]
... checking for valid disks...
... disks in mag :1 12
... normal disks: WWN [xxxxxxxxxxxxxxxx] Id [yy] diskpos [0]
... not normal disks: WWN [xxxxxxxxxxxxxxxx] Id [zz]
... verifying spare space for disksyy and yy
... playback chunklets from pd WWN[xxxxxxxxxxxxxxxx] Id [yy]
... All chunklets played back /relocated.
... cleared logging mode for cage 1mag 12
... relocating chunklets from sparespace
Failed
--
Could not find a destination forchunklet aa:bbb
All chunklets from the target driveyy are in use
Please also check for unclean ormedia failed chunklets
Use the -partial flag to move asmany of the rest as possible
servicemag resume 1 12 -- Failed
这时servicemag resume失败了。
我们需要通过下面命令去处理：
a. setpd ldalloc on<PD_ID> // Will start the re-initialization of thedrive.
b. showpd -c<PD_ID> // 这时工程师需要等待知道uninit 那列值为0以后再执行下一步
c. servicemag unmark<CAGE_ID><BAY_ID> // 通过servicemag unmark 命令取消这次serviemag resume操作，这时回答 ‘y’
d. servicemag clearstatus <CAGE_ID><BAY_ID>
e. servicemag resume <CAGE_ID><BAY_ID>这时再执行servicemag resume 操作，将数据恢复到新换磁盘。
还有一种情况是在serviemagresume过程中有些logicaldisk stuck，无法回迁回来。
root@1629459-0 Mon Jun 3011:46:50:~# showpdch 82
PdidChnk LdNameLdCh State Usage Media Sp Cl From To
82 234media-fail.82.234 0 normal ld failed N N -------------------------------------------------------------------------
Total chunklets: 1
root@1629459-0 Mon Jun 3011:47:53:~# removeld media-fail.82.234
Removing ld media-fail.82.234
select q=quit y=yes n=no: y
root@1629459-0 Mon Jun 3011:49:27:~#
这时需要将failedmedia LD 删除后继续执行servicemagresume 操作。
以上总结的更换磁盘遇到的情况，请大家在升级前务必把你们的操作步骤都记录到OPT中，这样L2 工程师知道你都做了哪些操作。