mxin辛旻
作者mxin辛旻2017-11-03 08:50
资深工程师, 上海宝信软件股份有限公司

GPFS 完全手册

字数 16629阅读 7880评论 0赞 4

GPFS 完全手册
GPFS 介绍
GPFS 是 IBM 公司第一个共享文件系统,起源于 IBM SP 系统上使用的虚拟共享磁盘技术( VSD )。作为这项技术的核心, GPFS 是一个并行的磁盘文件系统,它保证在资源组内的 所有节点可以并行访问整个文件系统;而且针对此文件系统的服务操作,可以同时安全地在 使用此文件系统的多个节点上实现。 GPFS 允许客户共享文件,而这些文件可能分布在不同节点的不同硬盘上;它提供了许多标 转的 UNIX 文件系统接口,允许应用不需修改或者重新编辑就可以在其上运行。 GPFS 也支持 UNIX 文件系统的工具,即:用户可以像使用普通文件系统一样使用基于 GPFS 的文件系统, 唯一不同的在于管理 GPFS 文件系统的命令。 GPFS 提供的文件系统操作服务可以支持并行应用和串行应用,它允许任何节点(这些节点 属于同一个节点组)上的并行应用同时访问同一个文件或者不同的文件。

一、规划:

12 10.66.3.X 98 hbzqc13 风控系统测试服务器
13 10.66.3.X 99 hbzqd13 风控系统运行服务器
17 10.66.5.X 51 hbzqa13 交易系统应用服务器1
18 10.66.5.X 52 hbzqb13 交易系统应用服务器2
19 10.66.5.X 53 hbzqa14 交易系统应用服务器3
20 10.66.5.X 54 hbzqb14 交易系统应用服务器4

pNSD server:hbzqc13 ,sNSD server:hbzqd13
二、准备工作:
SAN配置:
配置磁盘阵列盘,所有刀片划分zone,
可看到磁盘
磁盘都能认识
hbzqa13[/home/scripts]>lspv
hdisk0 00003e843c877ae9 rootvg active
hdisk1 00003e843c877b40 rootvg active
hdisk2 none None
hdisk3 00003e846ffa7a6e None
hbzqa13[/home/scripts]>pcmpath query essmap
Disk Path P Location adapter LUN SN Type Size LSS Vol Rank C/A S Connection port RaidMode


hdisk2 path0 01-10-01[FC] fscsi0 75L33510025 IBM 2107-900 65.5GB 0 37 fffc 0e Y R1-B1-H3-ZC 32 RAID5
hdisk2 path1 01-10-01[FC] fscsi0 75L33510025 IBM 2107-900 65.5GB 0 37 fffc 0e Y R1-B2-H1-ZC 102 RAID5
hdisk2 path2 01-11-01[FC] fscsi1 75L33510025 IBM 2107-900 65.5GB 0 37 fffc 0e Y R1-B3-H3-ZC 232 RAID5
hdisk2 path3 01-11-01[FC] fscsi1 75L33510025 IBM 2107-900 65.5GB 0 37 fffc 0e Y R1-B4-H1-ZC 302 RAID5
hdisk3 path0 01-10-01[FC] fscsi0 75L33510125 IBM 2107-900 65.5GB 1 37 fffb 17 Y R1-B1-H3-ZC 32 RAID5
hdisk3 path1 01-10-01[FC] fscsi0 75L33510125 IBM 2107-900 65.5GB 1 37 fffb 17 Y R1-B2-H1-ZC 102 RAID5
hdisk3 path2 01-11-01[FC] fscsi1 75L33510125 IBM 2107-900 65.5GB 1 37 fffb 17 Y R1-B3-H3-ZC 232 RAID5
hdisk3 path3 01-11-01[FC] fscsi1 75L33510125 IBM 2107-900 65.5GB 1 37 fffb 17 Y R1-B4-H1-ZC 302 RAID5

1.Make rsh and rcp workable
2.hbzqc13:
/etc/hosts
10.66.5.11 hbzqa01
10.66.5.12 hbzqb01
10.66.5.13 hbzqa02
10.66.5.14 hbzqb02
10.66.5.21 hbzqa05
10.66.5.22 hbzqb05
10.66.5.23 hbzqa06
10.66.5.24 hbzqb06
10.66.5.31 hbzqa09
10.66.5.32 hbzqb09
10.66.5.33 hbzqa10
10.66.5.34 hbzqb10
10.66.5.41 hbzqc01
10.66.5.42 hbzqd01
10.66.5.43 hbzqc02
10.66.5.44 hbzqd02
10.66.5.51 hbzqa13
10.66.5.52 hbzqb13
10.66.5.53 hbzqa14
10.66.5.54 hbzqb14
10.66.5.55 hbzqc03
10.66.5.56 hbzqd03
10.66.5.57 hbzqc04
10.66.5.58 hbzqd04
3.hbzqc13:
/etc/hosts

gpfs start

10.66.3.98 hbzqc13
10.66.3.99 hbzqd13

gpfs end

/.rhosts
hbzqc14 root
hbzqd14 root
hbzqc13
hbzqd13
rsh ok。
hbzqc13]root>run_cmd.sh hostname
rsh 10.66.5.51 "hostname;$1 $2 $3 $4 $5"
rsh 10.66.5.52 "hostname;$1 $2 $3 $4 $5"
rsh 10.66.5.53 "hostname;$1 $2 $3 $4 $5"
rsh 10.66.5.54 "hostname;$1 $2 $3 $4 $5"
rsh 10.66.5.55 "hostname;$1 $2 $3 $4 $5"
rsh 10.66.5.56 "hostname;$1 $2 $3 $4 $5"
rsh 10.66.5.57 "hostname;$1 $2 $3 $4 $5"
rsh 10.66.5.58 "hostname;$1 $2 $3 $4 $5"
hbzqa13
hbzqa13
hbzqb13
hbzqb13
nettest1[/mnt/gpfs]>rsh nettest2 date
Thu May 24 13:29:32 BEIST 2007
nettest2[/mnt/gpfs]>rsh nettest1 date
Thu May 24 13:29:32 BEIST 2007

4.export PATH=$PATH:/usr/lpp/mmfs/bin:.
"/.profile"
export PATH=$PATH:/usr/lpp/mmfs/bin:.
5.Install GPFS code, and Fix
a)# installp –agXYd . all
b)# smitty installp, to install fixes
6.Directories prepare
a)/tmp/mmfs -> link to a large file system, /home/gpfs/mmfs, for example

cd /tmp

mkdir /share
ln -s /share mmfs
7.Create the GPFS Cluster
Prepare the node file, and create the GPFS cluster.

vi /tmp/gpfs/nodefile

node1:quorum
node2:quorum

hbzqc13[/]>mkdir /tmp/gpfs
hbzqc13[/]>vi /tmp/gpfs/nodefile
"/tmp/gpfs/nodefile" 2 lines, 32 characters
hbzqc13:quorum
hbzqd13:quorum
hbzqa13:client
hbzqb13:client
hbzqb14:client
hbzqc03:client
hbzqd03:client
hbzqc04:client
hbzqd04:client
hbzqa01:quorum
hbzqb01:quorum
hbzqa02:client
hbzqb02:client
hbzqa05:client
hbzqb05:client
hbzqa06:client
hbzqb06:client
hbzqa09:client
hbzqb09:client
hbzqa10:client
hbzqb10:client
hbzqc01:client
hbzqd01:client
hbzqc02:client
hbzqd02:client

mmcrcluster –t lc –n /tmp/gpfs/nodefile –p node1 –s node2

mmlscluster

启动cluster
hbzqc13[/tmp/gpfs]>mmdelnode -f
hbzqc13[/tmp/gpfs]>mmcrcluster -C hbzqrun -U hbzq -N /tmp/gpfs/nodefile -p hbzqc13 -s hbzqd13
Thu Jun 28 15:42:57 BEIST 2007: 6027-1664 mmcrcluster: Processing node hbzqc13
Thu Jun 28 15:42:57 BEIST 2007: 6027-1664 mmcrcluster: Processing node hbzqd13
Thu Jun 28 15:42:58 BEIST 2007: 6027-1664 mmcrcluster: Processing node hbzqa13
Thu Jun 28 15:42:59 BEIST 2007: 6027-1664 mmcrcluster: Processing node hbzqb13
Thu Jun 28 15:42:59 BEIST 2007: 6027-1664 mmcrcluster: Processing node hbzqb14
Thu Jun 28 15:43:00 BEIST 2007: 6027-1664 mmcrcluster: Processing node hbzqc03
Thu Jun 28 15:43:01 BEIST 2007: 6027-1664 mmcrcluster: Processing node hbzqd03
Thu Jun 28 15:43:01 BEIST 2007: 6027-1664 mmcrcluster: Processing node hbzqc04
Thu Jun 28 15:43:02 BEIST 2007: 6027-1664 mmcrcluster: Processing node hbzqd04
mmcrcluster: Command successfully completed
mmcrcluster: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
察看
hbzqc13[/tmp/gpfs]>mmlscluster

GPFS cluster information

GPFS cluster name: hbzqrun.hbzqd13
GPFS cluster id: 739157013761844865
GPFS UID domain: hbzq
Remote shell command: /usr/bin/rsh
Remote file copy command: /usr/bin/rcp

GPFS cluster configuration servers:

Primary server: hbzqd13
Secondary server: hbzqc13

Node Daemon node name IP address Admin node name Designation

1 hbzqc13 10.66.3.98 hbzqc13 quorum
2 hbzqd13 10.66.3.99 hbzqd13 quorum
3 hbzqa13 10.66.5.51 hbzqa13
4 hbzqb13 10.66.5.52 hbzqb13
5 hbzqb14 10.66.5.54 hbzqb14
6 hbzqc03 10.66.5.55 hbzqc03
7 hbzqd03 10.66.5.56 hbzqd03
8 hbzqc04 10.66.5.57 hbzqc04
9 hbzqd04 10.66.5.58 hbzqd04
8.Create a Network Shared Disk(NSD)
Create the NSD file, and Create NSDs.

vi /tmp/gpfs/nsdfile

DiskName:PrimaryServer:BackupServer:DiskUsage:FailureGroup
hdiskX::::

mmcrnsd –F /tmp/gpfs/nsdfile

vi /tmp/gpfs/nsdfile
"/tmp/gpfs/nsdfile" 1 line, 26 characters
hdisk3:hbzqd13:hbzqc13: dataAndMetadata:4

hbzqc13[/tmp/gpfs]>mmcrnsd -F /tmp/gpfs/nsdfile
mmcrnsd: Processing disk hdisk3
mmcrnsd: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
转换
hbzqc13[/tmp/gpfs]>mmcrnsd -F /tmp/gpfs/nsdfile
hbzqc13[/tmp/gpfs]>mmcrnsd -?
Usage: mmcrnsd -F DescFile [-v {yes | no}]
hbzqc13[/tmp/gpfs]>cat nsdfile
hbzqd13[/tmp/gpfs]>vi nsdfile
"nsdfile" 2 lines, 60 characters

hdisk3:hbzqd13:hbzqc13: dataAndMetadata:4

gpfs1nsd:::dataAndMetadata:4:

Check the disk configuration

lspv

hbzqc13[/tmp/gpfs]>lspv
hdisk0 00003e9eac795247 rootvg active
hdisk1 00003e9eac795297 rootvg active
hdisk2 00003e8c52782293 hbzqd13_vg
hdisk3 00003e846ffa7a6e gpfs1nsd

9.Startup GPFS subsystem

mmstartup –a

ps –ef | grep mm

hbzqd13[/tmp/gpfs]>mmstartup -a
Thu Jun 28 15:52:12 BEIST 2007: 6027-1642 mmstartup: Starting GPFS ...
hbzqd13: 6027-2114 The GPFS subsystem is already active.
hbzqb14: 6027-2114 The GPFS subsystem is already active.
hbzqb13: 6027-2114 The GPFS subsystem is already active.
hbzqa13: 6027-2114 The GPFS subsystem is already active.
hbzqc04: 6027-2114 The GPFS subsystem is already active.
hbzqc03: 6027-2114 The GPFS subsystem is already active.
hbzqd03: 6027-2114 The GPFS subsystem is already active.
hbzqd04: 6027-2114 The GPFS subsystem is already active.
10.Create the GPFS filesystem

mmcrfs /gpfs1 gpfs01 –F /tmp/gpfs/nsdfile –A yes –B 64K –n 2

mmlsfs gpfs01

hbzqd13[/]>mmcrfs /share sharelv -F /tmp/gpfs/nsdfile -A yes -B 64K -n 30 -v no

GPFS: 6027-531 The following disks of sharelv will be formatted on node hbzqc13:

gpfs1nsd: size 67108864 KB

GPFS: 6027-540 Formatting file system ...
GPFS: 6027-535 Disks up to size 140 GB can be added to storage pool 'system'.
Creating Inode File
Creating Allocation Maps
Clearing Inode Allocation Map
Clearing Block Allocation Map
GPFS: 6027-572 Completed creation of file system /dev/sharelv.
mmcrfs: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.

hbzqd13[/]>mmchfs /dev/sharelv -A automount

mmlsfs sharelv

hbzqd13[/tmp/gpfs]>mmlsfs sharetlv
flag value description


-s roundRobin Stripe method
-f 2048 Minimum fragment size in bytes
-i 512 Inode size in bytes
-I 8192 Indirect block size in bytes
-m 1 Default number of metadata replicas
-M 1 Maximum number of metadata replicas
-r 1 Default number of data replicas
-R 1 Maximum number of data replicas
-j cluster Block allocation type
-D posix File locking semantics in effect
-k posix ACL semantics in effect
-a 1048576 Estimated average file size
-n 30 Estimated number of nodes that will mount file system
-B 65536 Block size
-Q none Quotas enforced

 none           Default quotas enabled

-F 49280 Maximum number of inodes
-V 9.03 File system version. Highest supported version: 9.03
-u yes Support for large LUNs?
-z no Is DMAPI enabled?
-E yes Exact mtime mount option
-S no Suppress atime mount option
-K whenpossible Strict replica allocation option
-P system Disk storage pools in file system
-d gpfs1nsd Disks in file system
-A automount Automatic mount option
-o none Additional mount options
-T /share Default mount point

mount /gpfs1 (on all GPFS cluster nodes)

mount /share
hbzqd13[/home/scripts/gpfs]>run_cmd.sh mount /share
11.Change the GPFS configuration attributes

mmchconfig autoload=yes start gpfs deamon when node up

mmchconfig singleNodeQuorum=yes obsolete parameter

mmlsconfig

nettest1[/mnt/gpfs]>mmchconfig autoload=yes
mmchconfig: Command successfully completed
mmchconfig: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
nettest1[/mnt/gpfs]>mmchconfig singleNodeQuorum=no
mmchconfig: 6027-1119 Obsolete option: singleNodeQuorum
nettest1[/mnt/gpfs]>mmlsconfig

Configuration data for cluster nettest1:

clusterName nettest1
clusterId 739159728178146586
clusterType lc
autoload yes
useDiskLease yes
maxFeatureLevelAllowed 901
[nettest1]
takeOverSdrServ yes

File systems in cluster nettest1:

/dev/gpfs01

hbzqc13[/]>mmgetstate -a

Node number Node name GPFS state

   1      hbzqc13          active
   2      hbzqd13          active
   3      hbzqa13          active
   4      hbzqb13          active
   5      hbzqb14          active
  10      hbzqa14          active

12.If you shutdown GPFS on all nodes

unmount /gpfs1 (on all GPFS nodes)

mmshutdown –a

hbzqd13[/home/scripts/gpfs]>mmshutdown -a
Mon Jul 30 09:56:02 BEIST 2007: 6027-1341 mmshutdown: Starting force unmount of GPFS file systems
hbzqc13: forced unmount of /share
hbzqa13: forced unmount of /share
hbzqb13: forced unmount of /share
hbzqa14: forced unmount of /share
hbzqb14: forced unmount of /share

Mon Jul 30 09:56:07 BEIST 2007: 6027-1344 mmshutdown: Shutting down GPFS daemons
hbzqc13: Shutting down!
hbzqa13: Shutting down!
hbzqb13: Shutting down!
hbzqa13: Shutting down!
hbzqb14: Shutting down!
hbzqa14: Shutting down!
hbzqc13: 'shutdown' command about to kill process 368890
hbzqa13: 'shutdown' command about to kill process 123086
hbzqb13: 'shutdown' command about to kill process 376940
hbzqa13: 'shutdown' command about to kill process 172040
hbzqa14: 'shutdown' command about to kill process 272940
hbzqb14: 'shutdown' command about to kill process 474040
Mon Jul 30 09:56:13 BEIST 2007: 6027-1345 mmshutdown: Finished
hbzqc09[/home/scripts/gpfs]>mmgetstate -a

Node number Node name GPFS state

   1      hbzqc13          down
   2      hbzqd13          down
   3      hbzqa13          down
   4      hbzqb13          down 
   5      hbzqb14          down 
  10      hbzqa14          down 

Delete GPFS Cluster

1.# fuser –kcu /gpfs1
2.# unmount /gpfs1 (on all GPFS nodes)
3.# mmdelfs gpfs01, #mmlsfs gpfs01, to check the results
4.# mmdelnsd –F /tmp/gpfs/nsdfile, or # mmdelnsd hdiskX;hdiskY
5.# mmshutdown –a
6.# mmdelnode –n /tmp/gpfs/nodefile, or #mmdelnode node1, node2… To delete the GPFS cluster

mmdelnode –f force delete current node from GPFS Cluster

Other Commands

mmchcluster –C cluster.name

mmchconfig

nettest1[/usr/lpp/mmfs/bin]>mmlspv
hdisk2 gpfs1nsd
nettest1[/usr/lpp/mmfs/bin]>mmgetstate

Node number Node name GPFS state

   1      nettest1         down

nettest1[/usr/lpp/mmfs/bin]>mmlscluster

GPFS cluster information

GPFS cluster name: nettest1
GPFS cluster id: 739159728178146586
GPFS UID domain: nettest1
Remote shell command: /usr/bin/rsh
Remote file copy command: /usr/bin/rcp

GPFS cluster configuration servers:

Primary server: nettest1
Secondary server: nettest2

Node Daemon node name IP address Admin node name Designation

1 nettest1 10.66.5.219 nettest1 quorum
2 nettest2 10.66.3.220 nettest2 quorum

nettest1[/usr/lpp/mmfs/bin]>mmlsmgr
mmcommon: 6027-1562 mmlsmgr command cannot be executed. Either none of the
nodes in the cluster are reachable, or GPFS is down on all of the nodes.
nettest1[/usr/lpp/mmfs/bin]>mmlsdisk
mmlsdisk: 6027-1268 Missing arguments
Usage:
mmlsdisk Device [-d "DiskName[;DiskName...]"] [-e] [-L]

  or

mmlsdisk Device [-d "DiskName[;DiskName...]"] {-m | -M}
nettest1[/usr/lpp/mmfs/bin]>mmlsnode
GPFS nodeset Node list


nettest1 nettest1 nettest2
nettest1[/usr/lpp/mmfs/bin]>mmlsnsd

File system Disk name Primary node Backup node

(free disk) gpfs1nsd nettest1 nettest2

新增一个节点:
run_cmd.sh mount 10.66.05:/soft_ins /mnt
run_cmd.sh installp -acYd /mnt/sddpcm all
run_cmd.sh installp -acYd /mnt/gpfs all
run_cmd.sh "chdev -l hdisk3 -a pv=yes"
run_cmd.sh errclear 0
run_cmd.sh pcmpath query essmap
rcp_file.sh /.profile
run_cmd.sh mkdir /share
run_cmd.sh ln -s /share mmfs
run_cmd.sh mkdir /tmp/gpfs
hbzqc13[/home/scripts/gpfs]>mmaddnode -N hbzqa14:client
Thu Jun 28 16:28:21 BEIST 2007: 6027-1664 mmaddnode: Processing node hbzqa14
mmaddnode: Command successfully completed
mmaddnode: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
hbzqc13[/home/scripts/gpfs]>mmlscluster

GPFS cluster information

GPFS cluster name: hbzqrun.hbzqd13
GPFS cluster id: 739157013761844865
GPFS UID domain: hbzq
Remote shell command: /usr/bin/rsh
Remote file copy command: /usr/bin/rcp

GPFS cluster configuration servers:

Primary server: hbzqd13
Secondary server: hbzqc13

Node Daemon node name IP address Admin node name Designation

1 hbzqc13 10.66.3.98 hbzqc13 quorum
2 hbzqd13 10.66.3.99 hbzqd13 quorum
3 hbzqa13 10.66.5.51 hbzqa13
4 hbzqb13 10.66.5.52 hbzqb13
5 hbzqb14 10.66.5.54 hbzqb14
6 hbzqc03 10.66.5.55 hbzqc03
7 hbzqd03 10.66.5.56 hbzqd03
8 hbzqc04 10.66.5.57 hbzqc04
9 hbzqd04 10.66.5.58 hbzqd04
10 hbzqa14 10.66.5.53 hbzqa14

如果觉得我的文章对您有用,请点赞。您的支持将鼓励我继续创作!

4

添加新评论0 条评论

Ctrl+Enter 发表

本文隶属于专栏

AIX运维专栏
专注于AIX系统运维,系统管理。

作者其他文章

相关文章

相关问题

相关资料

X社区推广