GPFS 完全手册
GPFS 介绍
GPFS 是 IBM 公司第一个共享文件系统,起源于 IBM SP 系统上使用的虚拟共享磁盘技术( VSD )。作为这项技术的核心, GPFS 是一个并行的磁盘文件系统,它保证在资源组内的 所有节点可以并行访问整个文件系统;而且针对此文件系统的服务操作,可以同时安全地在 使用此文件系统的多个节点上实现。 GPFS 允许客户共享文件,而这些文件可能分布在不同节点的不同硬盘上;它提供了许多标 转的 UNIX 文件系统接口,允许应用不需修改或者重新编辑就可以在其上运行。 GPFS 也支持 UNIX 文件系统的工具,即:用户可以像使用普通文件系统一样使用基于 GPFS 的文件系统, 唯一不同的在于管理 GPFS 文件系统的命令。 GPFS 提供的文件系统操作服务可以支持并行应用和串行应用,它允许任何节点(这些节点 属于同一个节点组)上的并行应用同时访问同一个文件或者不同的文件。
一、规划:
12 10.66.3.X 98 hbzqc13 风控系统测试服务器
13 10.66.3.X 99 hbzqd13 风控系统运行服务器
17 10.66.5.X 51 hbzqa13 交易系统应用服务器1
18 10.66.5.X 52 hbzqb13 交易系统应用服务器2
19 10.66.5.X 53 hbzqa14 交易系统应用服务器3
20 10.66.5.X 54 hbzqb14 交易系统应用服务器4
pNSD server:hbzqc13 ,sNSD server:hbzqd13
二、准备工作:
SAN配置:
配置磁盘阵列盘,所有刀片划分zone,
可看到磁盘
磁盘都能认识
hbzqa13[/home/scripts]>lspv
hdisk0 00003e843c877ae9 rootvg active
hdisk1 00003e843c877b40 rootvg active
hdisk2 none None
hdisk3 00003e846ffa7a6e None
hbzqa13[/home/scripts]>pcmpath query essmap
Disk Path P Location adapter LUN SN Type Size LSS Vol Rank C/A S Connection port RaidMode
hdisk2 path0 01-10-01[FC] fscsi0 75L33510025 IBM 2107-900 65.5GB 0 37 fffc 0e Y R1-B1-H3-ZC 32 RAID5
hdisk2 path1 01-10-01[FC] fscsi0 75L33510025 IBM 2107-900 65.5GB 0 37 fffc 0e Y R1-B2-H1-ZC 102 RAID5
hdisk2 path2 01-11-01[FC] fscsi1 75L33510025 IBM 2107-900 65.5GB 0 37 fffc 0e Y R1-B3-H3-ZC 232 RAID5
hdisk2 path3 01-11-01[FC] fscsi1 75L33510025 IBM 2107-900 65.5GB 0 37 fffc 0e Y R1-B4-H1-ZC 302 RAID5
hdisk3 path0 01-10-01[FC] fscsi0 75L33510125 IBM 2107-900 65.5GB 1 37 fffb 17 Y R1-B1-H3-ZC 32 RAID5
hdisk3 path1 01-10-01[FC] fscsi0 75L33510125 IBM 2107-900 65.5GB 1 37 fffb 17 Y R1-B2-H1-ZC 102 RAID5
hdisk3 path2 01-11-01[FC] fscsi1 75L33510125 IBM 2107-900 65.5GB 1 37 fffb 17 Y R1-B3-H3-ZC 232 RAID5
hdisk3 path3 01-11-01[FC] fscsi1 75L33510125 IBM 2107-900 65.5GB 1 37 fffb 17 Y R1-B4-H1-ZC 302 RAID5
1.Make rsh and rcp workable
2.hbzqc13:
/etc/hosts
10.66.5.11 hbzqa01
10.66.5.12 hbzqb01
10.66.5.13 hbzqa02
10.66.5.14 hbzqb02
10.66.5.21 hbzqa05
10.66.5.22 hbzqb05
10.66.5.23 hbzqa06
10.66.5.24 hbzqb06
10.66.5.31 hbzqa09
10.66.5.32 hbzqb09
10.66.5.33 hbzqa10
10.66.5.34 hbzqb10
10.66.5.41 hbzqc01
10.66.5.42 hbzqd01
10.66.5.43 hbzqc02
10.66.5.44 hbzqd02
10.66.5.51 hbzqa13
10.66.5.52 hbzqb13
10.66.5.53 hbzqa14
10.66.5.54 hbzqb14
10.66.5.55 hbzqc03
10.66.5.56 hbzqd03
10.66.5.57 hbzqc04
10.66.5.58 hbzqd04
3.hbzqc13:
/etc/hosts
10.66.3.98 hbzqc13
10.66.3.99 hbzqd13
/.rhosts
hbzqc14 root
hbzqd14 root
hbzqc13
hbzqd13
rsh ok。
hbzqc13]root>run_cmd.sh hostname
rsh 10.66.5.51 "hostname;$1 $2 $3 $4 $5"
rsh 10.66.5.52 "hostname;$1 $2 $3 $4 $5"
rsh 10.66.5.53 "hostname;$1 $2 $3 $4 $5"
rsh 10.66.5.54 "hostname;$1 $2 $3 $4 $5"
rsh 10.66.5.55 "hostname;$1 $2 $3 $4 $5"
rsh 10.66.5.56 "hostname;$1 $2 $3 $4 $5"
rsh 10.66.5.57 "hostname;$1 $2 $3 $4 $5"
rsh 10.66.5.58 "hostname;$1 $2 $3 $4 $5"
hbzqa13
hbzqa13
hbzqb13
hbzqb13
nettest1[/mnt/gpfs]>rsh nettest2 date
Thu May 24 13:29:32 BEIST 2007
nettest2[/mnt/gpfs]>rsh nettest1 date
Thu May 24 13:29:32 BEIST 2007
4.export PATH=$PATH:/usr/lpp/mmfs/bin:.
"/.profile"
export PATH=$PATH:/usr/lpp/mmfs/bin:.
5.Install GPFS code, and Fix
a)# installp –agXYd . all
b)# smitty installp, to install fixes
6.Directories prepare
a)/tmp/mmfs -> link to a large file system, /home/gpfs/mmfs, for example
cd /tmp
mkdir /share
ln -s /share mmfs
7.Create the GPFS Cluster
Prepare the node file, and create the GPFS cluster.
node1:quorum
node2:quorum
hbzqc13[/]>mkdir /tmp/gpfs
hbzqc13[/]>vi /tmp/gpfs/nodefile
"/tmp/gpfs/nodefile" 2 lines, 32 characters
hbzqc13:quorum
hbzqd13:quorum
hbzqa13:client
hbzqb13:client
hbzqb14:client
hbzqc03:client
hbzqd03:client
hbzqc04:client
hbzqd04:client
hbzqa01:quorum
hbzqb01:quorum
hbzqa02:client
hbzqb02:client
hbzqa05:client
hbzqb05:client
hbzqa06:client
hbzqb06:client
hbzqa09:client
hbzqb09:client
hbzqa10:client
hbzqb10:client
hbzqc01:client
hbzqd01:client
hbzqc02:client
hbzqd02:client
启动cluster
hbzqc13[/tmp/gpfs]>mmdelnode -f
hbzqc13[/tmp/gpfs]>mmcrcluster -C hbzqrun -U hbzq -N /tmp/gpfs/nodefile -p hbzqc13 -s hbzqd13
Thu Jun 28 15:42:57 BEIST 2007: 6027-1664 mmcrcluster: Processing node hbzqc13
Thu Jun 28 15:42:57 BEIST 2007: 6027-1664 mmcrcluster: Processing node hbzqd13
Thu Jun 28 15:42:58 BEIST 2007: 6027-1664 mmcrcluster: Processing node hbzqa13
Thu Jun 28 15:42:59 BEIST 2007: 6027-1664 mmcrcluster: Processing node hbzqb13
Thu Jun 28 15:42:59 BEIST 2007: 6027-1664 mmcrcluster: Processing node hbzqb14
Thu Jun 28 15:43:00 BEIST 2007: 6027-1664 mmcrcluster: Processing node hbzqc03
Thu Jun 28 15:43:01 BEIST 2007: 6027-1664 mmcrcluster: Processing node hbzqd03
Thu Jun 28 15:43:01 BEIST 2007: 6027-1664 mmcrcluster: Processing node hbzqc04
Thu Jun 28 15:43:02 BEIST 2007: 6027-1664 mmcrcluster: Processing node hbzqd04
mmcrcluster: Command successfully completed
mmcrcluster: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
察看
hbzqc13[/tmp/gpfs]>mmlscluster
GPFS cluster name: hbzqrun.hbzqd13
GPFS cluster id: 739157013761844865
GPFS UID domain: hbzq
Remote shell command: /usr/bin/rsh
Remote file copy command: /usr/bin/rcp
Primary server: hbzqd13
Secondary server: hbzqc13
1 hbzqc13 10.66.3.98 hbzqc13 quorum
2 hbzqd13 10.66.3.99 hbzqd13 quorum
3 hbzqa13 10.66.5.51 hbzqa13
4 hbzqb13 10.66.5.52 hbzqb13
5 hbzqb14 10.66.5.54 hbzqb14
6 hbzqc03 10.66.5.55 hbzqc03
7 hbzqd03 10.66.5.56 hbzqd03
8 hbzqc04 10.66.5.57 hbzqc04
9 hbzqd04 10.66.5.58 hbzqd04
8.Create a Network Shared Disk(NSD)
Create the NSD file, and Create NSDs.
DiskName:PrimaryServer:BackupServer:DiskUsage:FailureGroup
hdiskX::::
vi /tmp/gpfs/nsdfile
"/tmp/gpfs/nsdfile" 1 line, 26 characters
hdisk3:hbzqd13:hbzqc13: dataAndMetadata:4
hbzqc13[/tmp/gpfs]>mmcrnsd -F /tmp/gpfs/nsdfile
mmcrnsd: Processing disk hdisk3
mmcrnsd: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
转换
hbzqc13[/tmp/gpfs]>mmcrnsd -F /tmp/gpfs/nsdfile
hbzqc13[/tmp/gpfs]>mmcrnsd -?
Usage: mmcrnsd -F DescFile [-v {yes | no}]
hbzqc13[/tmp/gpfs]>cat nsdfile
hbzqd13[/tmp/gpfs]>vi nsdfile
"nsdfile" 2 lines, 60 characters
gpfs1nsd:::dataAndMetadata:4:
Check the disk configuration
hbzqc13[/tmp/gpfs]>lspv
hdisk0 00003e9eac795247 rootvg active
hdisk1 00003e9eac795297 rootvg active
hdisk2 00003e8c52782293 hbzqd13_vg
hdisk3 00003e846ffa7a6e gpfs1nsd
9.Startup GPFS subsystem
hbzqd13[/tmp/gpfs]>mmstartup -a
Thu Jun 28 15:52:12 BEIST 2007: 6027-1642 mmstartup: Starting GPFS ...
hbzqd13: 6027-2114 The GPFS subsystem is already active.
hbzqb14: 6027-2114 The GPFS subsystem is already active.
hbzqb13: 6027-2114 The GPFS subsystem is already active.
hbzqa13: 6027-2114 The GPFS subsystem is already active.
hbzqc04: 6027-2114 The GPFS subsystem is already active.
hbzqc03: 6027-2114 The GPFS subsystem is already active.
hbzqd03: 6027-2114 The GPFS subsystem is already active.
hbzqd04: 6027-2114 The GPFS subsystem is already active.
10.Create the GPFS filesystem
hbzqd13[/]>mmcrfs /share sharelv -F /tmp/gpfs/nsdfile -A yes -B 64K -n 30 -v no
GPFS: 6027-531 The following disks of sharelv will be formatted on node hbzqc13:
gpfs1nsd: size 67108864 KB
GPFS: 6027-540 Formatting file system ...
GPFS: 6027-535 Disks up to size 140 GB can be added to storage pool 'system'.
Creating Inode File
Creating Allocation Maps
Clearing Inode Allocation Map
Clearing Block Allocation Map
GPFS: 6027-572 Completed creation of file system /dev/sharelv.
mmcrfs: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
hbzqd13[/]>mmchfs /dev/sharelv -A automount
hbzqd13[/tmp/gpfs]>mmlsfs sharetlv
flag value description
-s roundRobin Stripe method
-f 2048 Minimum fragment size in bytes
-i 512 Inode size in bytes
-I 8192 Indirect block size in bytes
-m 1 Default number of metadata replicas
-M 1 Maximum number of metadata replicas
-r 1 Default number of data replicas
-R 1 Maximum number of data replicas
-j cluster Block allocation type
-D posix File locking semantics in effect
-k posix ACL semantics in effect
-a 1048576 Estimated average file size
-n 30 Estimated number of nodes that will mount file system
-B 65536 Block size
-Q none Quotas enforced
none Default quotas enabled
-F 49280 Maximum number of inodes
-V 9.03 File system version. Highest supported version: 9.03
-u yes Support for large LUNs?
-z no Is DMAPI enabled?
-E yes Exact mtime mount option
-S no Suppress atime mount option
-K whenpossible Strict replica allocation option
-P system Disk storage pools in file system
-d gpfs1nsd Disks in file system
-A automount Automatic mount option
-o none Additional mount options
-T /share Default mount point
mount /share
hbzqd13[/home/scripts/gpfs]>run_cmd.sh mount /share
11.Change the GPFS configuration attributes
nettest1[/mnt/gpfs]>mmchconfig autoload=yes
mmchconfig: Command successfully completed
mmchconfig: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
nettest1[/mnt/gpfs]>mmchconfig singleNodeQuorum=no
mmchconfig: 6027-1119 Obsolete option: singleNodeQuorum
nettest1[/mnt/gpfs]>mmlsconfig
clusterName nettest1
clusterId 739159728178146586
clusterType lc
autoload yes
useDiskLease yes
maxFeatureLevelAllowed 901
[nettest1]
takeOverSdrServ yes
/dev/gpfs01
hbzqc13[/]>mmgetstate -a
1 hbzqc13 active
2 hbzqd13 active
3 hbzqa13 active
4 hbzqb13 active
5 hbzqb14 active
10 hbzqa14 active
12.If you shutdown GPFS on all nodes
hbzqd13[/home/scripts/gpfs]>mmshutdown -a
Mon Jul 30 09:56:02 BEIST 2007: 6027-1341 mmshutdown: Starting force unmount of GPFS file systems
hbzqc13: forced unmount of /share
hbzqa13: forced unmount of /share
hbzqb13: forced unmount of /share
hbzqa14: forced unmount of /share
hbzqb14: forced unmount of /share
Mon Jul 30 09:56:07 BEIST 2007: 6027-1344 mmshutdown: Shutting down GPFS daemons
hbzqc13: Shutting down!
hbzqa13: Shutting down!
hbzqb13: Shutting down!
hbzqa13: Shutting down!
hbzqb14: Shutting down!
hbzqa14: Shutting down!
hbzqc13: 'shutdown' command about to kill process 368890
hbzqa13: 'shutdown' command about to kill process 123086
hbzqb13: 'shutdown' command about to kill process 376940
hbzqa13: 'shutdown' command about to kill process 172040
hbzqa14: 'shutdown' command about to kill process 272940
hbzqb14: 'shutdown' command about to kill process 474040
Mon Jul 30 09:56:13 BEIST 2007: 6027-1345 mmshutdown: Finished
hbzqc09[/home/scripts/gpfs]>mmgetstate -a
1 hbzqc13 down
2 hbzqd13 down
3 hbzqa13 down
4 hbzqb13 down
5 hbzqb14 down
10 hbzqa14 down
Delete GPFS Cluster
1.# fuser –kcu /gpfs1
2.# unmount /gpfs1 (on all GPFS nodes)
3.# mmdelfs gpfs01, #mmlsfs gpfs01, to check the results
4.# mmdelnsd –F /tmp/gpfs/nsdfile, or # mmdelnsd hdiskX;hdiskY
5.# mmshutdown –a
6.# mmdelnode –n /tmp/gpfs/nodefile, or #mmdelnode node1, node2… To delete the GPFS cluster
Other Commands
nettest1[/usr/lpp/mmfs/bin]>mmlspv
hdisk2 gpfs1nsd
nettest1[/usr/lpp/mmfs/bin]>mmgetstate
1 nettest1 down
nettest1[/usr/lpp/mmfs/bin]>mmlscluster
GPFS cluster name: nettest1
GPFS cluster id: 739159728178146586
GPFS UID domain: nettest1
Remote shell command: /usr/bin/rsh
Remote file copy command: /usr/bin/rcp
Primary server: nettest1
Secondary server: nettest2
1 nettest1 10.66.5.219 nettest1 quorum
2 nettest2 10.66.3.220 nettest2 quorum
nettest1[/usr/lpp/mmfs/bin]>mmlsmgr
mmcommon: 6027-1562 mmlsmgr command cannot be executed. Either none of the
nodes in the cluster are reachable, or GPFS is down on all of the nodes.
nettest1[/usr/lpp/mmfs/bin]>mmlsdisk
mmlsdisk: 6027-1268 Missing arguments
Usage:
mmlsdisk Device [-d "DiskName[;DiskName...]"] [-e] [-L]
or
mmlsdisk Device [-d "DiskName[;DiskName...]"] {-m | -M}
nettest1[/usr/lpp/mmfs/bin]>mmlsnode
GPFS nodeset Node list
nettest1 nettest1 nettest2
nettest1[/usr/lpp/mmfs/bin]>mmlsnsd
(free disk) gpfs1nsd nettest1 nettest2
新增一个节点:
run_cmd.sh mount 10.66.05:/soft_ins /mnt
run_cmd.sh installp -acYd /mnt/sddpcm all
run_cmd.sh installp -acYd /mnt/gpfs all
run_cmd.sh "chdev -l hdisk3 -a pv=yes"
run_cmd.sh errclear 0
run_cmd.sh pcmpath query essmap
rcp_file.sh /.profile
run_cmd.sh mkdir /share
run_cmd.sh ln -s /share mmfs
run_cmd.sh mkdir /tmp/gpfs
hbzqc13[/home/scripts/gpfs]>mmaddnode -N hbzqa14:client
Thu Jun 28 16:28:21 BEIST 2007: 6027-1664 mmaddnode: Processing node hbzqa14
mmaddnode: Command successfully completed
mmaddnode: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
hbzqc13[/home/scripts/gpfs]>mmlscluster
GPFS cluster name: hbzqrun.hbzqd13
GPFS cluster id: 739157013761844865
GPFS UID domain: hbzq
Remote shell command: /usr/bin/rsh
Remote file copy command: /usr/bin/rcp
Primary server: hbzqd13
Secondary server: hbzqc13
1 hbzqc13 10.66.3.98 hbzqc13 quorum
2 hbzqd13 10.66.3.99 hbzqd13 quorum
3 hbzqa13 10.66.5.51 hbzqa13
4 hbzqb13 10.66.5.52 hbzqb13
5 hbzqb14 10.66.5.54 hbzqb14
6 hbzqc03 10.66.5.55 hbzqc03
7 hbzqd03 10.66.5.56 hbzqd03
8 hbzqc04 10.66.5.57 hbzqc04
9 hbzqd04 10.66.5.58 hbzqd04
10 hbzqa14 10.66.5.53 hbzqa14
如果觉得我的文章对您有用,请点赞。您的支持将鼓励我继续创作!
赞4
添加新评论0 条评论