正文文本
San Volume Controller （SVC）
实施专家课程
第三部分 最佳实践
云存储-存储虚拟化实施专家级课程
© Copyright IBM Corporation 2009
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
课程内容说明
内容概述
SVC 基本要求和限制
SVC 连线及上架最佳实践
SVC zone最佳实践（主机zone&存储zone）
SVC连接后端存储最佳实践
SVC连接XIV最佳实践
SVC迁入步骤
SVC EasyTier 最佳实践
SVC 数据复制服务copy services最佳实践
SVC Stretch Cluster最佳实践
SVC与VMware 最佳实践
Page 2
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
SVC 基本要求和限制
Page 3
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
IBM SAN Volume Controller (SVC)
Implementation General Guidelines vDisk
1. Each SVC node contains four ports Win2k AIX Linux SUN
2. SVC nodes are deployed in pairs Common Multi-Path Driver
3. SANs must use supported switches
4. Different vendor switches may not be used for
connecting to the SVC nodes in the same fabric
5. SVC‟s should be deployed with redundant Host Zone
SVC Node Pair
fabrics. A single fabric is supported but will have
reduced reliability CACHE SAN
6. SVC nodes must connect to SAN switches.
MDISK
Direct connections to host or controllers are not
supported Storage Zone Managed Disk Group
7. Front-end hosts require special software to
access both SVC and other controllers
8. All SVC nodes in the cluster must be able to see
IBM all back-end storage to be managed.
9. SVC nodes now support mixed fabric speeds.
Page 4
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
IBM SAN Volume Controller (SVC)
Implementation General Guidelines
Storage Engine 336 SVC10. SVC supports up to 3 ISL hops in the fabric and this means TotalStorage
Node that the server and the SVC cluster may be separated by up TotalStorage Storage Engine 336
to 5 actual fibre channel links. Note SVC and Storage must Pair 0
TotalStorage Storage Engine 336 be on same director. SVC
11. Maximum hop count with Metro Mirror between SVC‟s in TotalStorage Storage Engine 336 Node
different fabrics is 7, with a maximum of 3 ISL hops and a Pair 1
TotalStorage Storage Engine 336 maximum of 3 local hops. SVC
Cluster SVC
TotalStorage Storage Engine 33612. The supplied UPS signal cable is 1.8 metres long. UPS Node
must be in the same rack as the node. TotalStorage Storage Engine 336 Pair 2
13. Switch configurations must follow switch vendor rules.
TotalStorage Storage Engine 336 SVC
14. Zoning should NOT include front-end hosts and back-end Node
System x3250 storage in the same zone. Pair 3
Management Node
15. SVC nodes should use managed disks created from RAID
arrays. JBODs are supported but not recommended.
16. The number of paths to a volume from a host to the nodes
in the I/O Group that owns the volume must not exceed
eight, even if eight is not the maximum number of paths
supported by the multipath driver (SDD supports up to 32).
To restrict the number of paths to a host volume, the fabrics
must be zoned so that each host FC port is zoned to no
more than two ports from each SVC node in the I/O Group
that owns the volume. UPSUPS
PairPair
19”19” RackRack MountedMounted
Page 5
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
SVC 连线及上架最佳实践
Page 6
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
连线及上架 Best Practices
Install one node/UPS per I/O group in rack 1 and other node/UPS per
I/O group in rack 2
Main reason is to avoid accidental loss of access
E.g. Shutdown node 1 for maintenance and accidentally pull node 2 power cord
- I/O Group goes offline, VDisks go offline, applications go offline
Consider not setting racks side by side to avoid localized disaster
Nodes must be roughly within 100 meters of each other
Except for split I/O groups
Utilize the Redundant AC Power Switch – FC8300
Provides dual input power to each SVC UPS thus power redundancy to
node
Helps improve resilience and thus SVC cluster availability
SVC cluster not interrupted if one circuit breaker or power grid is
unavailable
Page 7
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
Page 8
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
SVC zone最佳实践
Page 9
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
IBM_2145:admin> svcinfo lsnode 1
Back panel of SVC node: id 1
Ports Physically name node1
Fabric A Fabric B
Numbered 1-4 Left UPS_serial_number 10L3BMQ
to Right WWNN 5005076801000135 3 4
status online
2
1 IO_group_id 0
IO_group_name io_grp0
8G4 4 3 1 2 partner_node_id 2
partner_node_name node2
4 3 1 2 config_node yes
8F4 UPS_unique_id 202378101C0D2761
port_id 5005076801100135 (port 3)
4 3 1 2
port_status active8F2
port_id 5005076801200135 (port 4)
4F2 4 3 1 2 port_status active
port_id 5005076801300135 (port 2)
port_status active
Numbers in yellow used port_id 5005076801400135 (port 1)
to make WWPN unique port_status active
Page 10
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
Zoning 最佳实践
Connect port 1 and 3 to Fabric A – WWPNs are 4 and 1
Fabric Alias might be SVC_Node1_Even and SVC_Node1_Odd
Connect port 2 and 4 to Fabric B – WWPNs are 3 and 2
Fabric Alias might be SVC_Node1_Odd and SVC_Node1_Even
Now you have an odd and even WWPN in each fabric
Zone odd hosts with odd WWPN alias and zone even hosts with even alias on each
fabric
Balances workload and simplifies administration operations
Meets SVC requirements and guidelines
Four paths per VDisk
Zoning of one server HBA port to one port per SVC node associated with host object
Page 11
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
Zoning 最佳实践
Up to 4 fabrics supported using
Zone one port from each node
Configuration SVC_Cluster1
or Zone set Fabric B
SVC_Cluster1
Fabric A
Host
zones
Storage Zones Host Zones
One per disk One per
Cluster Zone subsystem server
Just do it!
SVC Remote
Non-SVC Zones Copy Zone
Use all SVC node ports for both host and storage I/O
Page 12
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
SVC-Cluster Zone1 SVC-Cluster Zone2
(11,1;11,2;11,3;11,4) (12,1;12,2;12,3;12,4)
Cluster zones with
all SVC ports
ID ID
21 22
Fabric 1 Fabric 2
0 1 2 3 4 0 1 2 3 4
ID ID
11 21 13 23 12 22 14 24 11 12
11 12 13 14 21 22 23 24
SVC1 SVC2
Page 13
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
SVC-DS Zone1 SVC-DS Zone2
(11,1;11,2;11,3;11,4;11,5) (12,1;12,2;12,3;12,4;12,5)
SVC-EMC Zone1 SVC-EMC Zone2
All storage ports (11,1;11,2;11,3;11,4;11,6) (12,1;12,2;12,3;12,4;12,6)
and all SVC ports
ID ID
21 22
Fabric 1 Fabric 2
0 1 2 3 4 5 6 0 1 2 3 4 5 6
ID ID
11 21 13 23 D1 E1 12 22 14 24 D2 E2 11 12
11 12 13 14 D1 D2 E1 E2 DS Family
SVC1
SVC EMC
21 22 23 24
L x SVC2
Page 14
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
SVC_AIX1-P1 Zone AIX1 SVC_AIX1-P2 Zone
(21,1;11,3;11,4) P1 P2 (22,1;12,1;12,2)
SVC_Node1_Odd;SVC SVC_Node1_Odd;SVC
Node2_Odd;AIX1_P1 Node2_Odd;AIX1_P2
1 1
ID ID
P1 21 P2 22
Fabric 1 Fabric 2 3 4 1 2
ID ID
11 21 13 23 D1 E1 12 22 14 24 D2 E2 11 12
AIX1 Host Port 11 12 13 14 21 22 23 24 and
SVC1 SVC2
One SVC Port
Note: No storage Per SVC Node
subsystem ports!
Page 15
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
SVC_AIX2-P1 Zone AIX2 SVC_AIX2-P2 Zone
(21,1;11,1;11,2) P1 P2 (22,1;12,3;12,4)
SVC_Node1_Even;SVC SVC_Node1_Even;SVC
Node2_Even;AIX2_P1 Node2_Even;AIX2_P2
1 1
ID ID
P1 21 P2 22
Fabric 1 Fabric 2 1 2 3 4
ID ID
11 21 13 23 D1 E1 12 22 14 24 D2 E2 11 12
AIX2 Host Port 11 12 13 14 21 22 23 24 and
SVC1 SVC2
One SVC Port
Note: No storage Per SVC Node
subsystem ports!
Page 16
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
SVC 连接后端存储
最佳实践
Page 17
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
Disk Controller Best Practices
Multipathing from SVC to disk controller built into SVC software
No need for Powerpath or SDD to talk to disk controllers
Default Round-Robin Multi-pathing to External Storage System Ports after
V6.3
All SVC nodes must see same set of LUNs from disk controller
Otherwise degraded mode on controller and/or MDisks
All SVC ports zoned to controller ports must see same set of LUNs
Otherwise degraded mode on controller and/or Mdisks
In general configure disk systems as you would with SVC
Disk drives
Be careful with large disk drives you don‟t end up with too few spindles to handle load
RAID-5 suggested, but RAID-10 is viable and useful, SVC does not negate this
When adding more disks to a subsystem consider adding the new MDisks
to existing MDGs versus creating additional small MDGs
Page 18
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
Disk Controller Best Practices
Maximum of 256 WWNNs
EMC DMX/SYMM, All HDS and SUN/HP HDS clones use one WWNN per
port; each appears as a separate controller to SVC
Upgrade to 4.2.1 or later so you can map LUNs thru up to 16 FA ports
- Results in 16 WWNNs/WWPNs used out of the max of 256
IBM, EMC Clariion, HP, etc. use one WWNN per subsystem; each
appears as a single controller with multiple ports/WWPNs
Maximum of 16 ports/WWPNs per WWNN using 1 out of the max of 256
Page 19
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
Disk Controller Best Practices
If there is no plan/strategy for MDG configuration use only one
MDG per backend disk system
MDG RAS Considerations:
May make sense to create multiple MDGs if you ensure a host only gets its‟ VDisks built
from one of the MDGs
- If MDG goes offline then it impacts only a subset of all the hosts using SVC
If not going to isolate hosts to MDGs then create one large MDG per disk controller
- Assumes physical disks are all same size and speed and same RAID level
MDG Performance Considerations:
May make sense to create multiple MDGs if attempting to isolate workloads to different
disk spindles
With SATA disk drives avoid more then one MDG if possible
We tend to see to many MDGs with too few MDisks and thus over driven MDisks
We tend to see too few large physical disk spindles and I/O suffers
- Avoid selling capacity and focus on spindle counts to meet workload requirements
Page 21
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
Disk Controller Best Practices
When adding disk drives to an existing disk system consider adding
the new MDisks to existing MDGs versus creating additional MDGs
Use Perl script to restripe VDisk extents evenly across all MDisks in MDG
Go to: www.ibm.com/alphaworks and search on „svctools‟
Managed Disk Group Extent Size
Maximum cluster capacity is related to extent size
16MB extent = 64TB and doubles for each increment in extent size e.g. 32MB = 128TB
Strongly recommend minimum 128/256MB
SPC benchmarks used 256MB extent
Does not really affect performance supposedly
Pick extent size and use for all MDGs
Can‟t migrate VDisks between MDGs with different extent sizes
- V4.3.x helps here with introduction of VDisk Mirroring
Page 22
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
Disk Controller Best Practices
All MDisks in an MDG should have similar performance
Create minimum of one MDG per disk controller
Add MDisks with same drive size, speed, RAID type
Preferred path for VDisks in an I/O Group can be tailored
Default algorithm will usually provide good results
SVC alternates VDisks across nodes to attempt to balance workload
Use TPC to monitor node workload balance
Override defaults to assign new VDisks to least used nodes
Use striped mode VDisk layout in most cases
Consider using sequential VDisks with applications/DBs that do their own
striping
Balance need for copy services with need for a few large VDisks
Remember limit of 64MB/s per VDisk for FC
Remember limit of 25MB/s per VDisk for MM/GM sync/resync
Default balances load across MDisks and randomizes starting MDisk per
VDisk
Page 23
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
Server 1 Server 2 SVC Cabling
A B A B Best Practice
and Zoning
SAN Fabric A SAN Fabric B
HOST ZONING SVC CLUSTER
Create a SVC/Host zone ZONING
for each server that
receives storage from Create one zone in the
the SVC cluster. RED fabric with all the
SVC node ports cabled
Example: I/O G-0 I/O G-1 to Fabric A and create one zone in the BLUE
Zone Server 1 port A (RED) 1 2 1 2 fabric with all the SVCwith all SVC node port 1's. node ports cabled to
Node 1 Node 3 Fabric B.
Zone Server 1 port B (BLUE)
with all SVC node port 2's. 3 4 3 4 Example:
Zone Server 2 port A (RED) All odd (RED) SVC node
with all SVC node port 3's. 1 2 1 2 ports in one zone and all
even (BLUE) SVC node ports
Zone Server 2 port B (BLUE) Node 2 Node 4 in one zone.
with all SVC node port 4's. 3 4 3 4 Note: For a cluster to be SVC *** NOTE *** created and to operate
SVC supports a maximum of Cluster correctly all node ports must
256 host objects per I/O be zoned together.
group thus a maximum of
1024 per cluster. The above STORAGE
host zoning results in each
server being seen by every I/ MDisk Group 1 / DS4000_1 ZONING
O group and the default host
object creation behavior VDisk 1 Create a SVC/Storage
results in each host object zone for each storage
counting as one towards subsystem virtualized
this 256 maximum. A2 VDisk 3 by the SVC cluster.
To create more then 256 Example:
host objects in the clusteryou must zone a host to a A1 MDisk1 MDisk2 MDisk3 MDisk4 MDisk5 MDisk6 MDisk7 MDisk8 MDisk9 MDisk10 MDisk11 MDisk12 MDisk13 Zone DS4000_1 controller
subset of the I/O groups, / / / / / / / / / / / / / B1 ports A1 and B1 with all SVC
you must assign the host node ports 1 and 3 in the
object at host creation time RED fabric.
to that same subset of IO
B2 Zone DS4000_1 controllergroups and then you must VDisk 2
ports A2 and B2 with all SVCassign that host‟s VDisks to Array1 Array2 Array3 Array4 Array5 Array6 Array7 Array8 Array9 Array10 Array11 Array12 Array13 node ports 2 and 4 in theone of those I/O groups in
that same subset. VDisk 4 BLUE fabric.
Page 25
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
SVC 连接XIV最佳实践
Page 26
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
SVC and XIV Best Practices
Full 15 Module XIV Recommendations – 79TB Useable
Use 2 interface host ports from each of the 6 interface modules
Use ports 1 and 3 from each interface module
Zone these 12 ports with all SVC node ports
Create forty-eight LUNs of equal size each a multiple of 17GB
1632GB approximately if using entire full frame XIV with SVC
Create one Managed Disk Group per XIV using 1GB or larger extent size
Large extent size ensures extent striped across all SATA disks even if some day XIV
supports more then one frame with 180 spindles (e.g.. 5 frames and 900 spindles)
Extent size really not critical and smaller size can be chosen if desired
Recommend minimum of 256MB extent size to ensure an extent is striped across all 180
spindles in single frame, but again not required since XIV stripes over all spindles anyway
Map LUNs to SVC as 48 MDisks and add all of them to the one XIV MDG
SVC will drive I/O to 4 MDisks/LUNs per each of the 12 XIV fibre ports
This provides good queue depth on SVC to drive XIV adequately
Create striped VDisks using all 48 MDisks in MDG
Page 27
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
SVC and XIV Best Practices
Six Module XIV Recommendations – 27TB Useable
Use 2 interface host ports from each of the 2 active interface modules
Use ports 1 and 3 from interface modules 4 and 5 (Interface module 6 is inactive)
Zone these 4 ports with all SVC node ports
Create 16 LUNs of equal size each a multiple of 17GB
1632GB approximately if using entire XIV with SVC
Create one Managed Disk Group per XIV using 1GB or larger extent size
Large extent size ensures extent striped across all SATA disks even if some day XIV
supports more then one frame with 180 spindles (e.g.. 5 frames and 900 spindles)
Extent size really not critical and smaller size can be chosen if desired
Recommend minimum of 256MB extent size to ensure an extent is striped across all 180
spindles in single frame, but again not required since XIV stripes over all spindles anyway
Map LUNs to SVC as 16 MDisks and add all of them to the one XIV MDG
SVC will drive I/O to 4 MDisks/LUNs per each of the 4 XIV fibre ports
This provides good queue depth on SVC to drive XIV adequately
Create striped VDisks using all 16 MDisks in MDG
Page 28
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
SVC and XIV Best Practices
Nine Module XIV Recommendations – 43TB Useable
Use 2 interface host ports from each of the 4 active interface modules
Use ports 1 and 3 from interface modules 4,5,7,8 (Interface modules 6 and 9 are inactive)
Zone these 8 ports with all SVC node ports
Create 26 LUNs of equal size each a multiple of 17GB
1632GB approximately if using entire XIV with SVC
Create one Managed Disk Group per XIV using 1GB or larger extent size
Large extent size ensures extent striped across all SATA disks even if some day XIV
supports more then one frame with 180 spindles (e.g.. 5 frames and 900 spindles)
Extent size really not critical and smaller size can be chosen if desired
Recommend minimum of 256MB extent size to ensure an extent is striped across all 180
spindles in single frame, but again not required since XIV stripes over all spindles anyway
Map LUNs to SVC as 26 MDisks and add all of them to the one XIV MDG
SVC will drive I/O to 3 MDisks/LUNs on each of 6 ports and 4 on the other 2 XIV fibre ports
This provides good queue depth on SVC to drive XIV adequately
Create striped VDisks using all 26 MDisks in MDG
Page 29
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
SVC and XIV Performance Results
Random Reads and Writes
No statistically significant performance difference with/without SVC
Sequential Writes
4K block size - SVC increases performance by 192%
8K block size - SVC increases performance by 88%
16K block size - SVC increases performance by 15%
32K block size - SVC decreases performance by 17%
64K block size - SVC decreases performance by 28%
Sequential Reads
4K block size - SVC increases performance by 61%
8K block size - SVC increases performance by 24%
16K block size - No significant difference with/without SVC
32K block size - No significant difference with/without SVC
64K block size - SVC decreases performance 5%
Page 31
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
SVC迁入步骤
Page 33
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
SVC迁入实施步骤参考
迁入前准备工作
前期SAN规划/存储阵列规划/SVC配置规划，相关人员到位
兼容性检查，必要的设备微码升级（交换机/磁盘阵列）
SVC上架上电，健康检查，初始化SVC集群
SVC zone配置（Host zone/Storage zone/Internal zone/inter-cluster zone)
在SVC上创建Host
EMC CX上为SVC建立新的Storage Group，加入SVC节点的WWPN
准备AIX主机版本所对应SDDPCM
迁入实施
停止应用，卸载VG，删除原有PV逻辑设备
Disable主机和CX直连的Zone
在CX上将原有的LUN从原主机Storage Group删除，加入到SVC的Storage Group当中
在SVC发现新的LUN，使用Image模式导入
将导入的Image模式Volume映射给建好的主机
主机上卸载原有多路径软件，安装Sddpcm，使用Cfgmgr识别新的PV，Importvg
启动应用，检查数据
迁入后续工作
条带化性能优化
卷镜像本地保护机制建立
快照备份机制建立
容灾机制建立
Page 34
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
SVC EasyTier
最佳实践
Page 35
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
Easy Tier Best Practice
Easy Tier between SSD and SAS or FC drives will be supported
Nothing in the product will disallow other combinations
Example: SAS to NL SAS allowed but will not be tested
Using Easy Tier with other disk combinations could lead to unpredictable performance and
should be tested before production implementation
Easy Tier algorithms are “tuned specifically” to optimize with SSDs involved
Not recommended to run Easy Tier in multiple disk controller levels
Example: At SVC and Storwize V7000 with DS8000 backend on the same
Volumes
Easy Tier is designed to work with either SVC / Storwize V7000 internal
SSDs or SAN based external SSDs
Striped Volume can have Easy Tier set to on/off (default is on)
Sequential and Image volumes will not be managed by Easy Tier
Pool can have Easy Tier set to on/auto/off (default is auto)
If set to “ON” rather than the default Auto, a single pool will be measured
36 Page 36
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
Easy Tier Best Practice
Image and Sequential Volumes can be measured but NOT managed by Easy
Tier
Easy Tier is supported for FlashCopy source and target and Metro or Global
Mirror primary and secondary Volumes
MM or GM secondary Volumes do not inherit characteristics of primary as part of
DR activities
Example: If secondary becomes primary and usage changes the Easy Tier will have to
adjust the extents over time to accommodate the new access patterns
Easy Tier is supported on mirrored Volumes
Both copies are managed by Easy Tier independently
Easy Tier is supported for Thin Provisioned Volumes
Only the real storage will be subject to management
Volume migrate
Management of the Volume will be disabled during a migration
Measurement of the Volume will reflect the status of the source storage pool
until the migration completes
Page 37
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
SVC 数据复制服务
最佳实践
Page 38
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
Copy Services – FlashCopy Best Practices
FlashCopy
Background copy rate is configurable and default is 50 meaning 2MBs per
VDisk
Customers sometimes complain it is taking a long time to do a full copy
Can dynamically change the copy rate up to a maximum of 64MBs per VDisk
Balance speed of copy versus impact on production applications
Page 40
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
Copy Services – Multiple-cycling Global Mirror
1) Initial GM Cycle Started at
Production host 6pm Oct. 10
2) Initial synchronization
completes at 6am Oct. 11
Host I/O
3) Freeze time will show 6pm
Oct. 10 so RPO is 12 hours
4) Change volumes reset
5) Next cycle begins
immediately since more then
five minutes has passed Global Mirror Relationship
Ay B C D E F G Ay B C D E F G Cycle Period 5 Minutes 6) May be lots of changes to
256K H I J K L M N H I J K L M N send from last 12 hours butGrains Inconsistent_copyingConsistent_copying statestate RPO will get smaller as GM
O P Q R S T Uz O P Q R S T Uz progresses through cycles
Change
Primary Volume Secondary Volume volume
FlashCopy A U FlashCopy A U
„nocopy‟ „nocopy‟
Mapping Change Mapping Change
Volume Volume
A10 0 0 0 0 0 0 AAAA1000 1000 1000 1000 1000 1000 1000
GM Bit 0 0 0 0 0 0 0 1000 1000 1000 1000 1000 1000 1000
Maps
0 0 0 0 0 0 U01 1100 1100 1100 1100 1100 1100 UUUU1110
Page 41
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
Copy Services – Change Volumes
Change volumes are used to record changes to the Primary and Secondary volumes of
a remote copy relationship
FlashCopy mapping exists between a Primary and its Change volume and a Secondary
and its Change volume
RC Relationship Primary Secondary
volume volume
Change Change
volume volume
FlashCopy FlashCopy
mapping mapping
Change volume requirements
Must be same virtual size and in the same I/O group as the Primary or Secondary volume of a
remote copy relationship
The Change volume will be Thin Provisioned (TP)
Can not be used for host I/O and cannot be mapped to a host
Can not be used by remote copy or FlashCopy
FlashCopy mapping is for internal use
User cannot manipulate it like a normal FlashCopy mapping
Most CLI commands will fail
Uses „nocopy‟ option so minimal real capacity used on TP change volumes
Page 42
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
Copy Services – Multiple-cycling Global Mirror
最大 RPO
如果GM复制每次都能在“cycling period”时间内完成，那么最大RPO = 2 * cycling period
否则， 最大RPO = 以前所有未完成复制的cycling period 的总和+当前的cycle times
当前RPO: 用lsrcrelationship查看 freeze time，再计算
Cycle period 是缺省5分钟，并且在 6:00am发起第一个Cycle，此时并没有时Freeze Time
6:03am第一个Cycle的拷贝完成，那么此刻的Freeze time是6:00am，并且RPO是3分钟
当第二个Cycle在6:05am开始，此刻RPO变为5分钟（ Freeze time 仍然是6:00am ）
如果第二个Cycle的复制在6:12am还未完成，那么此刻RPO将变成12分钟（这时6:00am仍然是上次拷贝完成的完整数据）
在 6:13am第二个Cycle的复制完成了，那么RPO将立刻减少到8分钟，因为6:05am发起的第二个Cycle已经复制完成,Freeze
Time已经变为6:05am
由于从6:05~6:13am已经超过第三个Cycle发起的时间，第三个复制将立刻开始
6：00 6：03 6：05 6：12 6：13
Freeze时间点
复制完成时间点
Cycling 0 Cycling 1 Cycling 3
Page 43
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
Copy Services – Replication Best Practices
Distance: MM <303KM, GM <8000KM
Use Fibre Channel extenders,SAN routers,DWDM
round-trip latency between sites cannot exceed 80 ms
Default link tolerance in noncycling Global Mirror is 5
minutes
Metro Mirror and Global Mirror require a specific
amount of bandwidth for intersystem heartbeat traffic
Metro Mirror or noncycling Global Mirror:
bandwidth between two sites must meet the peak
workload requirements and maintain the maximum
round-trip latency between the sites
Table 1. Intersystem heartbeat traffic in Mbps multiple-cycling Global Mirror:
System 1 System 2 bandwidth between two sites according to RPO need
2 nodes 4 nodes 6 nodes 8 nodes RPO minutes to hours
2 nodes 2.6 4.0 5.4 6.7 For example: consider the average write workload
4 nodes 4.0 5.5 7.1 8.6 when application systems are busy and the required
6 nodes 5.4 7.1 8.8 10.5 initial synchronization copy bandwidth
8 nodes 6.7 8.6 10.5 12.4
Redundancy link can tolerate single failures
Page 44
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
Copy Services – Replication Best Practices
Bandwidth Considerations
If the SAN network bandwidth is too small to handle the traffic then
application write I/O response time may be elongated
For SVC noncycling GM must support short term „Peak Write‟ bandwidth
requirements or expect write performance delays
SVC noncycling GM much more sensitive to lack of bandwidth than DS8000
- Bursts should exceed bandwidth for a „few‟ minutes only
- Default link tolerance in GM is 5 minutes
Pipe must be able to handle normal and peak write I/O
- Even after loss of redundant link where no interruption to operation is required
Need to consider initial sync and re-sync workload as well
Requires SCORE/RPQ if data change rate will exceed communication
bandwidth or round trip latency exceeds 80ms
Bandwidth configurable in SVC based on pipe between clusters
Only affects initial sync and any resyncs
Set bandwidth setting to 30-50% of pipe capacity initially
Page 45
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
Copy Services – Replication Best Practices
Metro/Global Mirror background copy rate is pre-defined
Per VDisk limit is 25MB/s
Maximum per I/O Group is roughly 250MB/s
Be careful using slower disk subsystem for the secondary VDisks for
high performance primary VDisks
SVC cache may not be able to buffer all the writes
Flushing cache writes to SATA may slow I/O at production site
Use TPC to determine change rate for bandwidth sizing
Page 46
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
SVC 三站点HA＋DR（HACMP/XD自动切换）
HACMP/XD实现DR全自动切换
Host Node Host Node Host Node
Oracle RAC
SVC SVC Cluster
Cluster
0101101010010101010101010010001001FC or SAN SAN SAN
extension
SVC
Cluster
(split IO 010110101010 01011010101
group) 0101101010010101
SVC Metro Mirror
SVC VDM SVC Global Mirror
Vendor_Y Vendor_Z
Vendor_X
Site 1 Site 2 Site 3
Management LAN
SVC Console IP
Page 47
FC
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
SVC容灾解决方案的优势
IBM SVC解决方案优势：
架构清晰，管理简单。
节约投资，有效保护投资成本
存储HA＋DR （1:1:1）
license投资不会随硬件的更换而丢失。（VDM免费，MM/GM永久有效）
存储HA 任意存储损坏不会影响DR
当DR初始化同步完成后，不需要再次Initial Copy。
各种灾难场景
两边存储设备的到期替换
具有独特领先的存储服务级别（SLA）保障以应对各个数据保护阶段及灾
难场景。
Page 48
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
SVC的Quorum disks
最佳实践
Page 49
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
Concept of Quorum disks
The cluster uses the quorum disk for two purposes:
• Quorum disks are used when there is a problem in the SAN fabric or when nodes are shut
down, leaving half of the nodes remaining in the cluster. A quorum disk determines which
group of nodes stops operating and processing I/O requests.
• Hold a copy of important cluster configuration data.
Just over 256 MB is reserved for this purpose on each quorum disk candidate.
Need for T3 recovery.
The cluster automatically chooses three MDisks as quorum disk candidates.
• There is only one active quorum disk that is used to determine which nodes operate as the
cluster.
• The purpose of the other quorum disk candidates is to provide redundancy.
Running an SVC cluster without a quorum disk can seriously affect operation.
• A lack of available quorum disks for storing metadata will prevent any migration operation:
Migrating multiple extents (within a storage pool)
Migrating extents off an MDisk that is being deleted(including a forced MDisk delete)
Migrating a volume between storage pools
Migrating the volume to image mode
Migrating a volume between I/O Groups
Monitoring the migration progress
• Mirrored virtual disks (VDisks) also are taken offline when all the quorum disks are on a
single controller that experiences a failure, because synchronization status for mirrored
Pagevolumes50 is recorded on the quorum disk.
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
Quorum disks best practice
Must set quorum disks on multiple controllers to avoid the possibility of losing all of
the quorum disks with a single failure.
follow guidelines for changing the MDisk that are assigned as quorum disk
candidates:
• It must be in managed mode (no image mode disks).
• Ensure that the status of the MDisk that is being set as a quorum disk is online before
setting it as the quorum disk.
• It must have sufficient free extents to hold the cluster state information, plus the stored
configuration metadata.
• It must be visible to all of the nodes in the cluster.
• When possible, distribute the quorum candidate disks so that each MDisk is provided by a
different storage system. For a list of storage systems that support quorum disks, refer to
the Supported Hardware List.
• Wait at least two minutes between setting quorum disks. This delay allows time for the
update to the quorum disk to complete before setting another quorum disk.
An SVC cluster automatically assigns quorum disk candidates, but it is good
practice to review the quorum disk assignments when adding new storage to a
cluster or removing existing storage.
Page 51
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
Additional guidance for split-cluster configurations
In the general case, where the nodes in a cluster have been split between sites,
the SVC cluster should be configured such that:
• Site 1: Half of SVC cluster nodes + one quorum disk candidate
• Site 2: Half of SVC cluster nodes + one quorum disk candidate
• Site 3: Active quorum disk
• This configuration ensures that there is always a quorum disk available, even after a
single site failure.
Page 52
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
SVC – How Quorum Disks Affect Availability (1)
Failure Failure
Domain 1 Domain 2 1) Loss of active quorum:
SVC selects another quorum
Continuation of operations
2) Loss of storage system:
Loss of active quorum
ISL 1
SVC selects another quorum
Continuation of operations
Mirrored Volumes continue
TotalStorage Storage Engine 336 TotalStorage Storage Engine 336
operation but may take
Node 1 Node 2 60sec or more since active
quorum disk failed
Note: The loss of all quorum
ISL 2 disks will not cause the
cluster to stop as long as
there are a majority of the
nodes operational in the
cluster. However, mirrored
Volumes will likely go
Volume Mirroring offline. This is why you
would manually configure
the cluster so the quorum
disk candidates are located
on disk systems in bothSVC Quorum 3 SVC Quorum 2 SVC Quorum 1
failure domains. Active Quorum
Page 53
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
SVC – How Quorum Disks Affect Availability (2)
Failure Failure
Domain 1 Domain 2 Lose of Failure Domain 1 :
Active quorum not affected
Continuation of operations
Lose of Failure Domain 2:
ISL 1 Active quorum lost
Half of nodes lost
Loss of cluster majority
TotalStorage Storage Engine 336 TotalStorage Storage Engine 336 Node 1 can not utilize
quorum candidate to Node 1 Node 2 recover and survive
Node 1 shuts down and
cluster stopped
ISL 2
May not be recoverable
and may require cluster
rebuild and data restore
from backups
No Active
Quorum Volume Mirroring
No Access to
Data on Disk
SVC Quorum 3 SVC Quorum 2 SVC Quorum 1
Active Quorum Active Quorum
Page 54
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
Current Supported Configuration for Split I/O Group
Failure Failure Failure
Domain 1 Domain 2 Domain 3
Automated failover
with SVC handling
The loss of:
- SVC node
- Quorum disk ISL 1
- Storage subsystem
TotalStorage Storage Engine 336 TotalStorage Storage Engine 336 Can incorporate
MM/GM to provide
Node 1 Node 2
disaster recovery
- 3 site like capability
ISL 2
Disk system
that supports
“Extended Quorum”
Volume Mirroring
SVC Quorum 1
SVC Quorum 3 SVC Quorum 2 SVC Quorum 1 Active Quorum
Active Quorum Active Quorum
Page 55
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
SVC Split I/O Group Summary
Site 1 Site 2 Site 3 Cluster Status
SVC Node 1 SVC Node 2 Quorum disk
Operational Operational Operational Operational, optimal
Failed Operational Operational Operational, Write cache disabled
Operational Failed Operational Operational, Write cache disabled
Operational Operational Failed Operational, Write Cache enabled, but
different active Quorum disk
Operational, link to Site 2 Operational, link to Site 1 failed: Operational Whichever node accesses the active
failed: Split Brain Split Brain quorum disk first survives and the partner
node goes offline
Operational Failed same time with Site 3 Failed same time with Site 2 Stopped
Failed same time with Site 3 Operational Failed same time with Site 1 Stopped Page 56
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
SVC Stretch Cluster
最佳实践
Page 57
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
Extension of Currently Supported Configuration
User chooses number of ISLs on SAN
Server Cluster 1 Server Cluster 2
SAN SAN
Active DWDM
over shared
single mode
fibre(s)
SVC + UPS 0-10 KM Fibre Channel distance SVC + UPS
supported up to 8Gbps
11-20KM Fibre Channel distance
SAN supported up to 4Gbps SAN
21-40KM Fibre Channel distance
supported up to 2Gbps
User chooses number of ISLs on SAN
Two ports per SVC node attached to local SANs
Two ports per SVC node attached to remote SANs via DWDM
Hosts and storage attached to SANs via ISLs sufficient for workload
Page 58 3rd site quorum (not shown) attached to SANs
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
Stretch Cluster – Distance
The new stretch cluster configuration will support distances of up to 300KM
Same recommendation as for Metro Mirror
However typical deployment of stretch cluster will only be 1/2 or 1/3 of this distance
because of added latency and buffer credit limitations/considerations
Page 59
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
Configuration With 4 Switches at Each Site
User chooses number of ISLs on public SAN
Server Cluster 1 Server Cluster 2
Public Private 1 ISL per I/O group Private Public
Configured as trunk SAN SAN SAN SAN
SVC + UPS SVC + UPS
Public Private 1 ISL per I/O group Private Public
SAN SAN Configured as trunk SAN SAN
User chooses number of ISLs on public SAN
Two ports per SVC node attached to public SANs
Two ports per SVC node attached to private SANs
Hosts and storage attached to public SANs
Page 60 3rd site quorum (not shown) attached to public SANs
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
Configuration Using CISCO VSANs
Switches are partitioned using VSANs
Server Cluster 1 Server Cluster 2
Public Private Private Public
VSAN VSAN VSAN VSAN
SVC + UPS SVC + UPS
Public Private Private Public
VSAN VSAN VSAN VSAN
Note ISLs/Trunks for private VSANs are dedicated
rather than being shared to guarantee dedicated
bandwidth is available for node to node traffic
Page 61 3rd site quorum (not shown) attached to public VSANs
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
Configuration Using Brocade Virtual Fabrics
Physical switches are partitioned into
two logical switches
Server Cluster 1 Server Cluster 2
Public Private Private Public
SAN SAN SAN SAN
SVC + UPS SVC + UPS
Public Private Private Public
SAN SAN SAN SAN
Note ISLs/Trunks for private SANs are dedicated
rather than being shared to guarantee dedicated
bandwidth is available for node to node traffic
Page 62 3rd site quorum (not shown) attached to public SANs
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
SVC与VMware 最佳实践
Page 63
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
VMware Best Practise
SVC supports vSphere V4/5 MRU, Fixed (default now) and Round-Robin
SRM supported with Metro/Global Mirror
SVC support VAAI(vStorage API for Array Integration)
HW accelerated ZERO HW accelerated Copy HW accelerated Locking
2-10x lower 2-10x better
number of IO Copy Area A to Area B Fill Area with 0 data movement
operations 10-100x
better
01011010101 A metadata
00000000000 B scaling
(SCSI Write Same) (SCSI XCopy) (Atomic Operation instead of Reserve/Release)
客户行业 测试环境 使用VAAI之前 使用VAAI之后 时间减少(%)
金融行业 2 VMs 433 Sec 180 Sec 59%
电力行业 2 VMs 944 sec 517 sec 45%
石油行业 40 VMs 1 hour 20 min 67% Page 65
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
VAAI--full copy
Without hardware-accelerated With hardware-accelerated
full copy (clone) full copy (clone)
Page 66
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
VAAI--Block zeroing
Block zeroing
Page 67
实施专家级课程 Storage Cloud – IBM SAN VOLUME CONTROLLER
VAAI--Hardware-acceleratedLocking
Without Hardware-assisted With Hardware-assisted
Locking Locking
Page 68
Thank
You!
© Copyright IBM Corporation 2009
