GPFS 文件系统后端存储迁移

由于 GPFS 后端文件系统存储性能比较低下，所以决定迁移 GPFS 文件系统后端存储到新的存储上。

GPFS 现状：

mmlsconfig all

adminMode central

autoload yes

automountDir /gpfs/automountdir

cipherList (undefined)

clusterId 720875322369538284

clusterName gpfscluster.dsprdsrv01

cnfsMountdPort (undefined)

cnfsNFSDprocs 32

cnfsSharedRoot (undefined)

cnfsVIP (undefined)

dataStructureDump yes

defaultHelperNodes (undefined)

defaultMountDir /gpfs

dmapiDataEventRetry 2

dmapiEventTimeout 86400000

dmapiMountEvent all

dmapiMountTimeout 60

dmapiSessionFailureTimeout 0

failureDetectionTime 35

maxblocksize 1M

maxFcntlRangesPerFile 200

maxFilesToCache 10000

maxMBpS 1600

maxStatCache 60000

minReleaseLevel 3.4.0.0

mmapRangeLock yes

nsdServerWaitTimeForMount 300

nsdServerWaitTimeWindowOnMount 600

pagepool 8G

pagepoolMaxPhysMemPct 75

prefetchThreads 72

subnets (undefined)

tscCmdPortRange (undefined)

tiebreakerDisks gpfs1nsd

uidDomain lining.com.cn

unmountOnDiskFail no

usePersistentReserve no

verbsPorts (undefined)

verbsRdma disable

worker1Threads 48

mmlsnsd

File system Disk name NSD servers

sharelv gpfs1nsd dsprdsrv01,dsprdsrv02

sharelv gpfs2nsd dsprdsrv02,dsprdsrv01

sharelv gpfs3nsd dsprdsrv01,dsprdsrv02

sharelv gpfs4nsd dsprdsrv02,dsprdsrv01

sharelv gpfs5nsd dsprdsrv01,dsprdsrv02

sharelv gpfs6nsd dsprdsrv02,dsprdsrv01

sharelv gpfs7nsd dsprdsrv01,dsprdsrv02

sharelv gpfs8nsd dsprdsrv02,dsprdsrv01

sharelv gpfs9nsd dsprdsrv01,dsprdsrv02

sharelv gpfs10nsd dsprdsrv02,dsprdsrv01

sharelv gpfs11nsd dsprdsrv01,dsprdsrv02

sharelv gpfs12nsd dsprdsrv02,dsprdsrv01

sharelv gpfs13nsd dsprdsrv01,dsprdsrv02

sharelv gpfs14nsd dsprdsrv02,dsprdsrv01

sharelv gpfs15nsd dsprdsrv01,dsprdsrv02

sharelv gpfs16nsd dsprdsrv01,dsprdsrv02

sharelv gpfs17nsd dsprdsrv02,dsprdsrv01

mmlsdisk sharelv

disk driver sector failure holds holds storage

name type size group metadata data status availability pool

gpfs1nsd nsd 512 4001 yes yes ready up system

gpfs2nsd nsd 512 4002 yes yes ready up system

gpfs3nsd nsd 512 4001 yes yes ready up system

gpfs4nsd nsd 512 4002 yes yes ready up system

gpfs5nsd nsd 512 4001 yes yes ready up system

gpfs6nsd nsd 512 4002 yes yes ready up system

gpfs7nsd nsd 512 4001 yes yes ready up system

gpfs8nsd nsd 512 4002 yes yes ready up system

gpfs9nsd nsd 512 4001 yes yes ready up system

gpfs10nsd nsd 512 4002 yes yes ready up system

gpfs11nsd nsd 512 4001 yes yes ready up system

gpfs12nsd nsd 512 4002 yes yes ready up system

gpfs13nsd nsd 512 4001 yes yes ready up system

gpfs14nsd nsd 512 4002 yes yes ready up system

gpfs15nsd nsd 512 4001 yes yes ready up system

gpfs16nsd nsd 512 4001 yes yes ready up system

gpfs17nsd nsd 512 4002 yes yes ready up system

df -g

Filesystem GB blocks Free %Used Iused %Iused Mounted on

/dev/hd4 4.00 3.76 6% 13713 2% /

/dev/hd2 5.00 1.94 62% 49732 10% /usr

/dev/hd9var 4.00 2.64 35% 11634 2% /var

/dev/hd3 10.00 9.98 1% 398 1% /tmp

/dev/hd1 5.00 4.75 5% 183 1% /home

/dev/hd11admin 0.50 0.50 1% 5 1% /admin

/proc - - - - - /proc

/dev/hd10opt 4.00 3.71 8% 9279 2% /opt

/dev/livedump 0.50 0.50 1% 4 1% /var/adm/ras/livedump

/dev/ora11gsoftlv 15.00 11.51 24% 12667 1% /oradb

/dev/lv00 0.50 0.48 4% 18 1% /var/adm/csd

/dev/odslv 10.00 10.00 1% 7 1% /ods

/dev/gpclv 2.00 1.33 34% 6161 2% /gpc

/dev/islv 80.00 72.54 10% 30869 1% /is

/dev/moialv 20.00 12.56 38% 1447603 32% /moia

/dev/waslv 40.00 37.22 7% 48014 1% /was

/dev/sharelv 1360.00 1318.59 4% 398340 33% /ods/gpfs

划分 EMC 存储到主机识别添加到 GPFS 现有集群

touch gpfs_emc_nsd

hdiskpower0:dsprdsrv01:dsprdsrv02:dataAndMetadata::

hdiskpower1:dsprdsrv02:dsprdsrv01:dataAndMetadata::

mmcrnsd -F /etc/gpfs_emc_nsd

mmcrnsd: Processing disk hdiskpower0

mmcrnsd: Processing disk hdiskpower1

mmcrnsd: Propagating the cluster configuration data to all

affected nodes. This is an asynchronous process.

lspv

hdisk0 00ccde156e60ba27 rootvg active

hdisk1 00cd3e11f748fbda gpfs1nsd

hdisk2 000c7a4ce9887166 gpfs2nsd

hdisk3 000c7a4ce98885c6 gpfs3nsd

hdisk4 00ccde15132f6ae9 gpfs4nsd

hdisk5 00ccde15132f6c55 gpfs5nsd

hdisk6 00ccde15132f7035 gpfs6nsd

hdisk7 00ccde15132f6e77 gpfs7nsd

hdisk8 00ccde15132f7063 gpfs8nsd

hdisk9 000c7a4ce9889cf0 gpfs9nsd

hdisk10 000c7a4ce988b86b gpfs10nsd

hdisk11 000c7a4ce988ce6c gpfs11nsd

hdisk12 00ccde15132f715f gpfs12nsd

hdisk13 00ccde15132f72af gpfs13nsd

hdisk14 00ccde15132f73fa gpfs14nsd

hdisk15 00ccde15132f75e3 gpfs15nsd

hdisk16 00f6f2b30f3b2579 None

hdisk17 none gpfs16nsd

hdisk18 none gpfs17nsd

hdisk19 none gpfs18nsd

hdisk20 none gpfs19nsd

hdisk21 none gpfs18nsd

hdisk22 none gpfs19nsd

hdisk23 none gpfs18nsd

hdisk24 none gpfs19nsd

hdisk25 none gpfs18nsd

hdisk26 none gpfs19nsd

hdiskpower0 00cce1158df5f037 gpfs18nsd

hdiskpower1 00cce1158dfa5355 gpfs19nsd

mmadddisk sharelv -F /etc/gpfs_emc_nsd

The following disks of sharelv will be formatted on node dsprdsrv01:

gpfs18nsd: size 327776512 KB

gpfs19nsd: size 327776512 KB

Extending Allocation Map

Checking Allocation Map for storage pool 'system'

Completed adding disks to file system sharelv.

mmadddisk: Propagating the cluster configuration data to all

affected nodes. This is an asynchronous process.

删除 GPFS 现有后端磁盘， GPFS 自动重新平衡条带化，迁移数据。

删除 gpfs17nsd 操作

mmdeldisk sharelv gpfs17nsd

Deleting disks ...

Scanning file system metadata, phase 1 ...

Scan completed successfully.

Scanning file system metadata, phase 2 ...

Scan completed successfully.

Scanning file system metadata, phase 3 ...

Scan completed successfully.

Scanning file system metadata, phase 4 ...

Scan completed successfully.

Scanning user file metadata ...

6.11 % complete on Tue Apr 7 15:18:24 2015 ( 65536 inodes 2423 MB)

14.30 % complete on Tue Apr 7 15:18:45 2015 ( 218308 inodes 5674 MB)

20.43 % complete on Tue Apr 7 15:19:13 2015 ( 360448 inodes 8103 MB)

30.84 % complete on Tue Apr 7 15:19:33 2015 ( 480894 inodes 12234 MB)

39.10 % complete on Tue Apr 7 15:19:53 2015 ( 557056 inodes 15510 MB)

100.00 % complete on Tue Apr 7 15:20:00 2015

Scan completed successfully.

Checking Allocation Map for storage pool 'system'

81 % complete on Tue Apr 7 15:20:05 2015

100 % complete on Tue Apr 7 15:20:07 2015

tsdeldisk64 completed.

mmdeldisk: Propagating the cluster configuration data to all

affected nodes. This is an asynchronous process.

mmlsdisk sharelv

disk driver sector failure holds holds storage

name type size group metadata data status availability pool

gpfs1nsd nsd 512 4001 yes yes ready up system

gpfs2nsd nsd 512 4002 yes yes ready up system

gpfs3nsd nsd 512 4001 yes yes ready up system

gpfs4nsd nsd 512 4002 yes yes ready up system

gpfs5nsd nsd 512 4001 yes yes ready up system

gpfs6nsd nsd 512 4002 yes yes ready up system

gpfs7nsd nsd 512 4001 yes yes ready up system

gpfs8nsd nsd 512 4002 yes yes ready up system

gpfs9nsd nsd 512 4001 yes yes ready up system

gpfs10nsd nsd 512 4002 yes yes ready up system

gpfs11nsd nsd 512 4001 yes yes ready up system

gpfs12nsd nsd 512 4002 yes yes ready up system

gpfs13nsd nsd 512 4001 yes yes ready up system

gpfs14nsd nsd 512 4002 yes yes ready up system

gpfs15nsd nsd 512 4001 yes yes ready up system

gpfs16nsd nsd 512 4001 yes yes ready up system

mmlsnsd

File system Disk name NSD servers

sharelv gpfs1nsd dsprdsrv01,dsprdsrv02

sharelv gpfs2nsd dsprdsrv02,dsprdsrv01

sharelv gpfs3nsd dsprdsrv01,dsprdsrv02

sharelv gpfs4nsd dsprdsrv02,dsprdsrv01

sharelv gpfs5nsd dsprdsrv01,dsprdsrv02

sharelv gpfs6nsd dsprdsrv02,dsprdsrv01

sharelv gpfs7nsd dsprdsrv01,dsprdsrv02

sharelv gpfs8nsd dsprdsrv02,dsprdsrv01

sharelv gpfs9nsd dsprdsrv01,dsprdsrv02

sharelv gpfs10nsd dsprdsrv02,dsprdsrv01

sharelv gpfs11nsd dsprdsrv01,dsprdsrv02

sharelv gpfs12nsd dsprdsrv02,dsprdsrv01

sharelv gpfs13nsd dsprdsrv01,dsprdsrv02

sharelv gpfs14nsd dsprdsrv02,dsprdsrv01

sharelv gpfs15nsd dsprdsrv01,dsprdsrv02

sharelv gpfs16nsd dsprdsrv01,dsprdsrv02

(free disk) gpfs17nsd dsprdsrv02,dsprdsrv01

按照以上操作依次删除其他磁盘，指定最后一个磁盘 gpfs1nsd （集群仲裁盘）

mmlsnsd

File system Disk name NSD servers

sharelv gpfs1nsd dsprdsrv01,dsprdsrv02

sharelv gpfs18nsd dsprdsrv01,dsprdsrv02

sharelv gpfs19nsd dsprdsrv02,dsprdsrv01

(free disk) gpfs10nsd dsprdsrv02,dsprdsrv01

(free disk) gpfs11nsd dsprdsrv01,dsprdsrv02

(free disk) gpfs12nsd dsprdsrv02,dsprdsrv01

(free disk) gpfs13nsd dsprdsrv01,dsprdsrv02

(free disk) gpfs14nsd dsprdsrv02,dsprdsrv01

(free disk) gpfs15nsd dsprdsrv01,dsprdsrv02

(free disk) gpfs16nsd dsprdsrv01,dsprdsrv02

(free disk) gpfs17nsd dsprdsrv02,dsprdsrv01

(free disk) gpfs2nsd dsprdsrv02,dsprdsrv01

(free disk) gpfs3nsd dsprdsrv01,dsprdsrv02

(free disk) gpfs4nsd dsprdsrv02,dsprdsrv01

(free disk) gpfs5nsd dsprdsrv01,dsprdsrv02

(free disk) gpfs6nsd dsprdsrv02,dsprdsrv01

(free disk) gpfs7nsd dsprdsrv01,dsprdsrv02

(free disk) gpfs8nsd dsprdsrv02,dsprdsrv01

(free disk) gpfs9nsd dsprdsrv01,dsprdsrv02

mmlsdisk sharelv

disk driver sector failure holds holds storage

name type size group metadata data status availability pool

gpfs1nsd nsd 512 4001 yes yes ready up system

gpfs18nsd nsd 512 4001 yes yes ready up system

gpfs19nsd nsd 512 4002 yes yes ready up system

然后使用 mmdelnsd 删除 gpfs17nsd 到 gpfs2nsd ：

mmdelnsd gpfs16nsd

mmdelnsd: Processing disk gpfs16nsd

mmdelnsd: Propagating the cluster configuration data to all

affected nodes. This is an asynchronous process.

安排时间窗口停止 GPFS 集群，修改仲裁盘为 gpfs18nsd ，然后重新启动，最后删除 gpfs1nsd 磁盘，到此数据迁移完毕。

删除原有系统上的定性能磁盘：

mpio_get_config -Av

Frame id 0:

Storage Subsystem worldwide name: 60ab80050e8f200004b31f12

Controller count: 2

Partition count: 1

Partition 0:

Storage Subsystem Name = 'DS4700-01'

hdisk LUN # Ownership User Label

hdisk1 0 A (preferred) datastging_lun1

hdisk2 1 A (preferred) datastging_lun2

hdisk3 2 A (preferred) datastging_lun3

hdisk4 3 A (preferred) datastging_lun4

hdisk5 4 A (preferred) datastging_lun5

hdisk6 5 A (preferred) datastging_lun6

hdisk7 6 A (preferred) datastging_lun7

hdisk8 7 A (preferred) datastging_lun8

hdisk9 8 A (non-preferred) datastging_lun9

hdisk10 9 A (non-preferred) datastging_lun10

hdisk11 10 A (non-preferred) datastging_lun11

hdisk12 11 A (non-preferred) datastging_lun12

hdisk13 12 A (non-preferred) datastging_lun13

hdisk14 13 A (non-preferred) datastging_lun14

hdisk15 14 A (non-preferred) datastging_lun15

hdisk16 15 A (preferred) datastage_gpfs_zongcai_disk

hdisk17 16 A (preferred) datastaging_lun16

hdisk18 17 A (preferred) datastaging_lun17

以此类推

rmdev -Rdl hdisk1

hdisk1 deleted

最后，删除存储映射回收空间，删除 zone 配置。

完毕！