Ceph是一种为优秀的性能、可靠性和可扩展性而设计的统一的、分布式文件系统。
特点如下:
高性能
高可用性
高可扩展性
特性丰富
使用场景:
块存储 (适合单客户端使用)
使用场景:
文件存储 (适合多客户端有目录结构)
使用场景:
对象存储 (适合更新变动较少的数据,没有目录结构,不能直接打开/修改文件)
使用场景:
系统架构:
Ceph 生态系统架构可以划分为四部分:
1. NAS(Network Attached Storage)
2. NFS(Network File System)
名称 | 功能 | 适合场景 | 优缺点 |
---|---|---|---|
MFS | 1. 单点MDS<br/> 2. 支持FUSE<br/>3. 数据分片分布<br/>4. 多副本<br/>5. 故障手动恢复 | 大量小文件读写 | 1. 运维实施简单<br/> 2. 但存在单点故障 |
Ceph | 1. 多个MDS,可扩展<br/> 2. 支持FUSE<br/>3. 数据分片(crush)分布<br/>4. 多副本/纠删码<br/>5. 故障自动恢复 | 统一小文件存储 | 1. 运维实施简单 <br/>2. 故障自愈,自我恢复<br/>3. MDS锁的问题<br/>4. J版本很多坑, L版本可以上生产环境 |
ClusterFS | 1. 不存在元数据节点<br/> 2. 支持FUSE<br/>3. 数据分片分布<br/>4. 镜像<br/> 5. 故障自动恢复 | 适合大文件 | 1. 运维实施简单<br/> 2. 不存储元数据管理 <br/> 3. 增加了客户端计算负载 |
Lustre | 1. 双MDS互备,不可用扩展<br/> 2. 支持FUSE<br/>3. 数据分片分布<br/>4. 冗余(无)<br/>5. 故障自动恢复 | 大文件读写 | 1. 运维实施复杂 <br/> 2. 太庞大<br/> 3. 比较成熟 |
说明:
核心组件:MDS、Clients、RADOS。
挂载方式:
可扩展性
共享文件系统
多个clients可以同时读写 。
文件/目录Layouts
POSIX ACLs
NFS-Ganesha
Client Quotas
负载均衡
说明:
说明:
standby mds
消息: Client name failing to respond to cache pressure
说明: 客户端有各自的元数据缓存,客户端缓存中的条目(比如索引节点)也会存在于 MDS 缓存中,
所以当 MDS 需要削减其缓存时(保持在 mds_cache_size 以下),它也会发消息给客户端让它们削减自己的缓存。如果某个客户端的响应时间超过了 mds_recall_state_timeout (默认为 60s ),这条消息就会出现。
如果有客户端没响应或者有缺陷,就会妨碍 MDS 将缓存保持在 mds_cache_size 以下, MDS 就有可能耗尽内存而后崩溃。
客户端由于网络问题或者其他问题,导致客户端不可用。
v12 luminous版本已修复:
https://github.com/ceph/ceph/commit/51c926a74e5ef478c11ccbcf11c351aa520dde2a
mds: fix false "failing to respond to cache pressure" warning
//read.c
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdio.h>
#include <pthread.h>
int main()
{
int i = 0;
for(i = 0; ;i++)
{
char *filename = "test.log";
int fd = open(filename, O_RDONLY);
printf("fd=[%d]", fd);
fflush(stdout);
sleep(5);
}
}
//write.c
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdio.h>
#include <pthread.h>
int main()
{
int i = 0;
for(i = 0; ;i++)
{
char *filename = "test.log";
int fd = open(filename, O_CREAT | O_WRONLY | O_APPEND, S_IRUSR | S_IWUSR);
write(fd, "aaaa\\n", 6);
printf("fd=[%d] buffer=[%s]", fd, "aaaa");
close(fd);
fflush(stdout);
sleep(5);
}
}
018-12-13 19:56:11.222816 7fffee6d0700 0 log_channel(cluster) log [WRN] : 1 slow requests, 1 included below; oldest blocked for > 30.670943 secs
2018-12-13 19:56:11.222826 7fffee6d0700 0 log_channel(cluster) log [WRN] : slow request 30.670943 seconds old, received at 2018-12-13 19:55:40.551820: client_request(client.22614489:538 lookup #0x1/test.log 2018-12-13 19:55:40.551681 caller_uid=0, caller_gid=0{0,}) currently failed to rdlock, waiting
2018-12-13 19:56:13.782378 7ffff0ed5700 1 mds.ceph-xxx-osd02.ys Updating MDS map to version 229049 from mon.0
2018-12-13 19:56:33.782572 7ffff0ed5700 1 mds.ceph-xxx-osd02.ys Updating MDS map to version 229050 from mon.0
2018-12-13 20:00:26.226405 7fffee6d0700 0 log_channel(cluster) log [WRN] : evicting unresponsive client ceph-xxx-osd01.ys (22532339), after 303.489228 seconds
总结:
有两种情况可以自动剔除客户:
2018-04-27 19:24:15.984156 7f53015d7700 1 mds.0.2738 rejoin_start
2018-04-27 19:25:15.987531 7f53015d7700 1 mds.0.2738 rejoin_joint_start
2018-04-27 19:27:40.105134 7f52fd4ce700 1 mds.0.2738 rejoin_done
2018-04-27 19:27:42.206654 7f53015d7700 1 mds.0.2738 handle_mds_map i am now mds.0.2738
2018-04-27 19:27:42.206658 7f53015d7700 1 mds.0.2738 handle_mds_map state change up:rejoin --> up:active
MDS守护进程至少在mds_beacon_grace中未能向监视器发送消息,而它们应该在每个mds_beacon_interval发送消息。此时Ceph监视器将自动将MDS切换为备用MDS。 如果MDS的Session Inode过多导致MDS繁忙,只从切换未能及时发送消息,就可能会出现循环切换的概率。一般建设增大mds_beacon_grace。
mds beacon grace
描述: 多久没收到标识消息就认为 MDS 落后了(并可能替换它)。
类型: Float
默认值: 15
client: fix fuse client hang because its pipe to mds is not ok
There is a risk client will hang if fuse client session had been killed by mds and
the mds daemon restart or hot-standby switch happens right away but the client
did not receive any message from monitor due to network or other whatever reason
untill the mds become active again.Thus cause client didn't do closed_mds_session
lead the seession still is STATE_OPEN but client can't send any message to
mds because its pipe is not ok.
So we should create pipe to mds guarantee any meta request can be sent to
server.When mds recevie the message will send a CLOSE_SESSION to client
becasue its session for this client is STATE_CLOSED.After the previous
steps the old session of client can be closed and new session and pipe
can be established and the mountpoint will be ok.
调节mds session autoclose(默认300s),尽快剔除有问题的客户端。
有两种情况可以自动驱逐客户:
也叫: multi-mds 、 active-active MDS
每个 CephFS 文件系统默认情况下都只配置一个活跃 MDS 守护进程。在大型系统中,为了扩展元数据性能你可以配置多个活跃的 MDS 守护进程,它们会共同承担元数据负载。
CephFS 在Luminous版本中多元数据服务器(Multi-MDS)的功能和目录分片(dirfragment)的功能宣称已经可以在生产环境中使用。
说明:
为了实现文件系统数据和元数据的负载均衡,业界一般有几种分区方法:
静态子树分区
Hash计算分区
动态子树分区
说明:
多个活动的MDSs可以迁移目录以平衡元数据负载。何时、何地以及迁移多少的策略都被硬编码到元数据平衡模块中。
Mantle是一个内置在MDS中的可编程元数据均衡器。其思想是保护平衡负载(迁移、复制、碎片化)的机制,但使用Lua定制化平衡策略。
大多数实现都在MDBalancer中。度量通过Lua栈传递给均衡器策略,负载列表返回给MDBalancer。这些负载是“发送到每个MDS的数量”,并直接插入MDBalancer“my_targets”向量。
暴露给Lua策略的指标与已经存储在mds_load_t中的指标相同:auth.meta_load()、all.meta_load()、req_rate、queue_length、cpu_load_avg。
它位于当前的均衡器实现旁边,并且它是通过“ceph.conf”中的字符串启用的。如果Lua策略失败(无论出于何种原因),我们将回到原来的元数据负载均衡器。
均衡器存储在RADOS元数据池中,MDSMap中的字符串告诉MDSs使用哪个均衡器。
This PR does not not have the following features from the Supercomputing paper:
targets
table, where each index is the amount of load to send to each MDSmeta_load()
参考:
$ ceph fs set test1_fs max_mds 2
$ ceph fs status
test1_fs - 3 clients
========
+------+--------+------------------------+---------------+-------+-------+
| Rank | State | MDS | Activity | dns | inos |
+------+--------+------------------------+---------------+-------+-------+
| 0 | active | ceph-xxx-osd02.ys | Reqs: 0 /s | 3760 | 14 |
| 1 | active | ceph-xxx-osd01.ys | Reqs: 0 /s | 11 | 13 |
+------+--------+------------------------+---------------+-------+-------+
+-----------------+----------+-------+-------+
| Pool | type | used | avail |
+-----------------+----------+-------+-------+
| cephfs_metadata | metadata | 194M | 88.7T |
| cephfs_data | data | 0 | 88.7T |
+-----------------+----------+-------+-------+
+------------------------+
| Standby MDS |
+------------------------+
| ceph-xxx-osd03.ys |
+------------------------+
MDS version: didi_dss version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)
通过设置max_mds增加active mds。
$ ceph-fuse /mnt/
$ df
ceph-fuse 95330861056 40960 95330820096 1% /mnt
$ ceph fs status
test1_fs - 3 clients
========
+------+--------+------------------------+---------------+-------+-------+
| Rank | State | MDS | Activity | dns | inos |
+------+--------+------------------------+---------------+-------+-------+
| 0 | active | ceph-xxx-osd03.ys | Reqs: 5624 /s | 139k | 133k |
+------+--------+------------------------+---------------+-------+-------+
+-----------------+----------+-------+-------+
| Pool | type | used | avail |
+-----------------+----------+-------+-------+
| cephfs_metadata | metadata | 238M | 88.7T |
| cephfs_data | data | 2240M | 88.7T |
+-----------------+----------+-------+-------+
+------------------------+
| Standby MDS |
+------------------------+
| ceph-xxx-osd01.ys |
| ceph-xxx-osd02.ys |
+------------------------+
MDS version: didi_dss version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)
rados put --pool=cephfs_metadata_a greedyspill.lua ../src/mds/balancers/greedyspill.lua
ceph fs set test1_fs max_mds 2
ceph fs set test1_fs balancer greedyspill.lua
$ ceph fs status
test1_fs - 3 clients
========
+------+--------+------------------------+---------------+-------+-------+
| Rank | State | MDS | Activity | dns | inos |
+------+--------+------------------------+---------------+-------+-------+
| 0 | active | ceph-xxx-osd03.ys | Reqs: 2132 /s | 4522 | 1783 |
| 1 | active | ceph-xxx-osd02.ys | Reqs: 173 /s | 306 | 251 |
+------+--------+------------------------+---------------+-------+-------+
+-----------------+----------+-------+-------+
| Pool | type | used | avail |
+-----------------+----------+-------+-------+
| cephfs_metadata | metadata | 223M | 88.7T |
| cephfs_data | data | 27.1M | 88.7T |
+-----------------+----------+-------+-------+
+------------------------+
| Standby MDS |
+------------------------+
| ceph-xxx-osd01.ys |
+------------------------+
MDS version: didi_dss version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)
#mds00绑定到/mnt/test0
#mds01绑定到/mnt/test1
#setfattr -n ceph.dir.pin -v <rank> <path>
setfattr -n ceph.dir.pin -v 0 /mnt/test0
setfattr -n ceph.dir.pin -v 1 /mnt/test1
#检查mds请求负责情况
$ ceph fs status
test1_fs - 3 clients
========
+------+--------+------------------------+---------------+-------+-------+
| Rank | State | MDS | Activity | dns | inos |
+------+--------+------------------------+---------------+-------+-------+
| 0 | active | ceph-xxx-osd03.ys | Reqs: 3035 /s | 202k | 196k |
| 1 | active | ceph-xxx-osd02.ys | Reqs: 3039 /s | 70.8k | 66.0k |
+------+--------+------------------------+---------------+-------+-------+
+-----------------+----------+-------+-------+
| Pool | type | used | avail |
+-----------------+----------+-------+-------+
| cephfs_metadata | metadata | 374M | 88.7T |
| cephfs_data | data | 4401M | 88.7T |
+-----------------+----------+-------+-------+
+------------------------+
| Standby MDS |
+------------------------+
| ceph-xxx-osd01.ys |
+------------------------+
MDS version: didi_dss version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)
$ ceph fs status
test1_fs - 4 clients
========
+------+--------+------------------------+---------------+-------+-------+
| Rank | State | MDS | Activity | dns | inos |
+------+--------+------------------------+---------------+-------+-------+
| 0 | active | ceph-xxx-osd02.ys | Reqs: 0 /s | 75.7k | 72.6k |
| 1 | active | ceph-xxx-osd01.ys | Reqs: 0 /s | 67.8k | 64.0k |
+------+--------+------------------------+---------------+-------+-------+
+-----------------+----------+-------+-------+
| Pool | type | used | avail |
+-----------------+----------+-------+-------+
| cephfs_metadata | metadata | 311M | 88.7T |
| cephfs_data | data | 3322M | 88.7T |
+-----------------+----------+-------+-------+
+------------------------+
| Standby MDS |
+------------------------+
| ceph-xxx-osd03.ys |
+------------------------+
MDS version: didi_dss version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)
$ systemctl stop ceph-mds.target
$ ceph fs status
test1_fs - 2 clients
========
+------+--------+------------------------+---------------+-------+-------+
| Rank | State | MDS | Activity | dns | inos |
+------+--------+------------------------+---------------+-------+-------+
| 0 | replay | ceph-xxx-osd03.ys | | 0 | 0 |
| 1 | active | ceph-xxx-osd01.ys | Reqs: 0 /s | 67.8k | 64.0k |
+------+--------+------------------------+---------------+-------+-------+
+-----------------+----------+-------+-------+
| Pool | type | used | avail |
+-----------------+----------+-------+-------+
| cephfs_metadata | metadata | 311M | 88.7T |
| cephfs_data | data | 3322M | 88.7T |
+-----------------+----------+-------+-------+
+-------------+
| Standby MDS |
+-------------+
+-------------+
MDS version: didi_dss version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)
#进行压测rank0, 发现请求能正常落在mds3上
$ ceph fs status
test1_fs - 4 clients
========
+------+--------+------------------------+---------------+-------+-------+
| Rank | State | MDS | Activity | dns | inos |
+------+--------+------------------------+---------------+-------+-------+
| 0 | active | ceph-xxx-osd03.ys | Reqs: 2372 /s | 72.7k | 15.0k |
| 1 | active | ceph-xxx-osd01.ys | Reqs: 0 /s | 67.8k | 64.0k |
+------+--------+------------------------+---------------+-------+-------+
+-----------------+----------+-------+-------+
| Pool | type | used | avail |
+-----------------+----------+-------+-------+
| cephfs_metadata | metadata | 367M | 88.7T |
| cephfs_data | data | 2364M | 88.7T |
+-----------------+----------+-------+-------+
+------------------------+
| Standby MDS |
+------------------------+
| ceph-xxx-osd02.ys |
+------------------------+
MDS version: didi_dss version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)
工具 | 集群模式 | 客户端数量(压测端) | 性能 |
---|---|---|---|
filebench | 1MDS | 2个客户端 | 5624 ops/s |
filebench | 2MDS | 2个客户端 | 客户端1:3035 ops/s<br/>客户端2:3039 ops/s |
单活mds
多活mds 默认
多活mds 动态负载均衡 (目前12.2版本不推荐使用)
多活mds 静态分区 (推荐使用,外界社区也有用到生产环境)
多活mds 主备模式
说明:
状态 | 说明 |
---|---|
up:active | This is the normal operating state of the MDS. It indicates that the MDS and its rank in the file system is available.<br><br>这个状态是正常运行的状态。 这个表明该mds在rank中是可用的状态。 |
up:standby | The MDS is available to takeover for a failed rank (see also :ref:mds-standby ). The monitor will automatically assign an MDS in this state to a failed rank once available.<br><br>这个状态是灾备状态,用来接替主挂掉的情况。 |
up:standby_replay | The MDS is following the journal of another up:active MDS. Should the active MDS fail, having a standby MDS in replay mode is desirable as the MDS is replaying the live journal and will more quickly takeover. A downside to having standby replay MDSs is that they are not available to takeover for any other MDS that fails, only the MDS they follow.<br><br>灾备守护进程就会持续读取某个处于 up 状态的 rank 的元数据日志。这样它就有元数据的热缓存,在负责这个 rank 的守护进程失效时,可加速故障切换。<br><br> 一个正常运行的 rank 只能有一个灾备重放守护进程( standby replay daemon ),如果两个守护进程都设置成了灾备重放状态,那么其中任意一个会取胜,另一个会变为普通的、非重放灾备状态。<br><br>一旦某个守护进程进入灾备重放状态,它就只能为它那个 rank 提供灾备。如果有另外一个 rank 失效了,即使没有灾备可用,这个灾备重放守护进程也不会去顶替那个失效的。 |
up:boot | This state is broadcast to the Ceph monitors during startup. This state is never visible as the Monitor immediately assign the MDS to an available rank or commands the MDS to operate as a standby. The state is documented here for completeness. <br><br> 此状态在启动期间被广播到CEPH监视器。这种状态是不可见的,因为监视器立即将MDS分配给可用的秩或命令MDS作为备用操作。这里记录了完整性的状态。 |
up:creating | The MDS is creating a new rank (perhaps rank 0) by constructing some per-rank metadata (like the journal) and entering the MDS cluster. |
up:starting | The MDS is restarting a stopped rank. It opens associated per-rank metadata and enters the MDS cluster. |
up:stopping | When a rank is stopped, the monitors command an active MDS to enter the up:stopping state. In this state, the MDS accepts no new client connections, migrates all subtrees to other ranks in the file system, flush its metadata journal, and, if the last rank (0), evict all clients and shutdown (see also :ref:cephfs-administration ). |
up:replay | The MDS taking over a failed rank. This state represents that the MDS is recovering its journal and other metadata.<br><br>日志恢复阶段,他将日志内容读入内存后,在内存中进行回放操作。 |
up:resolve | The MDS enters this state from up:replay if the Ceph file system has multiple ranks (including this one), i.e. it's not a single active MDS cluster. The MDS is resolving any uncommitted inter-MDS operations. All ranks in the file system must be in this state or later for progress to be made, i.e. no rank can be failed/damaged or up:replay. <br><br>用于解决跨多个mds出现权威元数据分歧的场景,对于服务端包括子树分布、Anchor表更新等功能,客户端包括rename、unlink等操作。 |
up:reconnect | An MDS enters this state from up:replay or up:resolve. This state is to solicit reconnections from clients. Any client which had a session with this rank must reconnect during this time, configurable via mds_reconnect_timeout.<br><br>恢复的mds需要与之前的客户端重新建立连接,并且需要查询之前客户端发布的文件句柄,重新在mds的缓存中创建一致性功能和锁的状态。mds不会同步记录文件打开的信息,原因是需要避免在访问mds时产生多余的延迟,并且大多数文件是以只读方式打开。 |
up:rejoin | The MDS enters this state from up:reconnect. In this state, the MDS is rejoining the MDS cluster cache. In particular, all inter-MDS locks on metadata are reestablished.<br>If there are no known client requests to be replayed, the MDS directly becomes up:active from this state.<br><br>把客户端的inode加载到mds cache |
up:clientreplay | The MDS may enter this state from up:rejoin. The MDS is replaying any client requests which were replied to but not yet durable (not journaled). Clients resend these requests during up:reconnect and the requests are replayed once again. The MDS enters up:active after completing replay. |
down:failed | No MDS actually holds this state. Instead, it is applied to the rank in the file system |
down:damaged | No MDS actually holds this state. Instead, it is applied to the rank in the file system |
down:stopped | No MDS actually holds this state. Instead, it is applied to the rank in the file system |
This state diagram shows the possible state transitions for the MDS/rank. The legend is as follows:
//src/ceph_mds.cc
int main(int argc, const char **argv)
{
ceph_pthread_setname(pthread_self(), "ceph-mds");
vector<const char*> args;
argv_to_vec(argc, argv, args);
env_to_vec(args);
//初始化全局信息
auto cct = global_init(NULL, args,
CEPH_ENTITY_TYPE_MDS, CODE_ENVIRONMENT_DAEMON,
0, "mds_data");
//初始化堆栈分析器
ceph_heap_profiler_init();
std::string val, action;
for (std::vector<const char*>::iterator i = args.begin(); i != args.end(); ) {
if (ceph_argparse_double_dash(args, i)) {
break;
}
else if (ceph_argparse_flag(args, i, "--help", "-h", (char*)NULL)) {
// exit(1) will be called in the usage()
usage();
}
else if (ceph_argparse_witharg(args, i, &val, "--hot-standby", (char*)NULL)) {
int r = parse_rank("hot-standby", val);
dout(0) << "requesting standby_replay for mds." << r << dendl;
char rb[32];
snprintf(rb, sizeof(rb), "%d", r);
g_conf->set_val("mds_standby_for_rank", rb);
g_conf->set_val("mds_standby_replay", "true");
g_conf->apply_changes(NULL);
}
else {
derr << "Error: can't understand argument: " << *i << "\\n" << dendl;
usage();
}
}
pick_addresses(g_ceph_context, CEPH_PICK_ADDRESS_PUBLIC);
// Normal startup
if (g_conf->name.has_default_id()) {
derr << "must specify '-i name' with the ceph-mds instance name" << dendl;
usage();
}
if (g_conf->name.get_id().empty() ||
(g_conf->name.get_id()[0] >= '0' && g_conf->name.get_id()[0] <= '9')) {
derr << "deprecation warning: MDS id '" << g_conf->name
<< "' is invalid and will be forbidden in a future version. "
"MDS names may not start with a numeric digit." << dendl;
}
uint64_t nonce = 0;
get_random_bytes((char*)&nonce, sizeof(nonce));
std::string public_msgr_type = g_conf->ms_public_type.empty() ? g_conf->get_val<std::string>("ms_type") : g_conf->ms_public_type;
//创建通信的messenger
Messenger *msgr = Messenger::create(g_ceph_context, public_msgr_type,
entity_name_t::MDS(-1), "mds",
nonce, Messenger::HAS_MANY_CONNECTIONS);
if (!msgr)
exit(1);
msgr->set_cluster_protocol(CEPH_MDS_PROTOCOL);
cout << "starting " << g_conf->name << " at " << msgr->get_myaddr()
<< std::endl;
uint64_t required =
CEPH_FEATURE_OSDREPLYMUX;
msgr->set_default_policy(Messenger::Policy::lossy_client(required));
msgr->set_policy(entity_name_t::TYPE_MON,
Messenger::Policy::lossy_client(CEPH_FEATURE_UID |
CEPH_FEATURE_PGID64));
msgr->set_policy(entity_name_t::TYPE_MDS,
Messenger::Policy::lossless_peer(CEPH_FEATURE_UID));
msgr->set_policy(entity_name_t::TYPE_CLIENT,
Messenger::Policy::stateful_server(0));
int r = msgr->bind(g_conf->public_addr);
if (r < 0)
exit(1);
global_init_daemonize(g_ceph_context);
common_init_finish(g_ceph_context);
// get monmap
MonClient mc(g_ceph_context);
if (mc.build_initial_monmap() < 0)
return -1;
global_init_chdir(g_ceph_context);
//开始接收消息
msgr->start();
//创建MDSDaemon,启动MDS
mds = new MDSDaemon(g_conf->name.get_id().c_str(), msgr, &mc);
// in case we have to respawn...
mds->orig_argc = argc;
mds->orig_argv = argv;
r = mds->init();
if (r < 0) {
msgr->wait();
goto shutdown;
}
...
msgr->wait();
...
return 0;
}
//MDSDaemon.cc
/***************************admin socket相关,统计mds埋点信息及状态信息***************************/
bool MDSDaemon::asok_command(string command, cmdmap_t& cmdmap, string format,
ostream& ss);
void MDSDaemon::dump_status(Formatter *f);
void MDSDaemon::set_up_admin_socket();
void MDSDaemon::clean_up_admin_socket()
/*********************************************************************************************/
/***************************初始化***************************/
int MDSDaemon::init()
{
...
//初始化MonClient
int r = 0;
r = monc->init();
if (r < 0) {
derr << "ERROR: failed to get monmap: " << cpp_strerror(-r) << dendl;
mds_lock.Lock();
suicide();
mds_lock.Unlock();
return r;
}
...
//初始化mgrclient
mgrc.init();
messenger->add_dispatcher_head(&mgrc);
mds_lock.Lock();
if (beacon.get_want_state() == CEPH_MDS_STATE_DNE) {
dout(4) << __func__ << ": terminated already, dropping out" << dendl;
mds_lock.Unlock();
return 0;
}
monc->sub_want("mdsmap", 0, 0);
monc->sub_want("mgrmap", 0, 0);
monc->renew_subs();
mds_lock.Unlock();
//初始化SaftTimer
timer.init();
//初始化Beacon
beacon.init(mdsmap);
messenger->set_myname(entity_name_t::MDS(MDS_RANK_NONE));
// 重置tick
reset_tick();
mds_lock.Unlock();
return 0;
}
/*********************************************************/
/***************************重置tick相关***************************/
void MDSDaemon::reset_tick();
void MDSDaemon::tick();
/****************************************************************/
/***************************处理命令,返回信息***************************/
void MDSDaemon::handle_command(MCommand *m);
void MDSDaemon::send_command_reply(MCommand *m, MDSRank *mds_rank,
int r, bufferlist outbl,
boost::string_view outs);
//mds map信息
void MDSDaemon::handle_mds_map(MMDSMap *m);
/*********************************************************************/
/***************************处理信号,自杀,重生*************************/
void MDSDaemon::handle_signal(int signum);
void MDSDaemon::suicide();
void MDSDaemon::respawn();
/********************************************************************/
/***************************消息调度处理*************************/
bool MDSDaemon::ms_dispatch(Message *m)
{
Mutex::Locker l(mds_lock);
if (stopping) {
return false;
}
//mds处于shutdown状态,不处理消息
if (beacon.get_want_state() == CEPH_MDS_STATE_DNE) {
dout(10) << " stopping, discarding " << *m << dendl;
m->put();
return true;
}
// 优先处理daemon message
const bool handled_core = handle_core_message(m);
if (handled_core) {
return true;
}
// 不是核心的,尝试给rank发送消息
if (mds_rank) {
return mds_rank->ms_dispatch(m);
} else {
return false;
}
}
//高优先级处理的消息MON,MDS,OSD
bool MDSDaemon::handle_core_message(Message *m)
{
switch (m->get_type()) {
// MON
case CEPH_MSG_MON_MAP:
ALLOW_MESSAGES_FROM(CEPH_ENTITY_TYPE_MON);
m->put();
break;
// MDS
case CEPH_MSG_MDS_MAP:
ALLOW_MESSAGES_FROM(CEPH_ENTITY_TYPE_MON | CEPH_ENTITY_TYPE_MDS);
handle_mds_map(static_cast<MMDSMap*>(m));
break;
// OSD
case MSG_COMMAND:
handle_command(static_cast<MCommand*>(m));
break;
case CEPH_MSG_OSD_MAP:
ALLOW_MESSAGES_FROM(CEPH_ENTITY_TYPE_MON | CEPH_ENTITY_TYPE_OSD);
if (mds_rank) {
mds_rank->handle_osd_map();
}
m->put();
break;
case MSG_MON_COMMAND:
ALLOW_MESSAGES_FROM(CEPH_ENTITY_TYPE_MON);
clog->warn() << "dropping `mds tell` command from legacy monitor";
m->put();
break;
default:
return false;
}
return true;
}
//重置消息,不进行处理
bool MDSDaemon::ms_handle_reset(Connection *con);
void MDSDaemon::ms_handle_remote_reset(Connection *con);
bool MDSDaemon::ms_handle_refused(Connection *con)
/***************************************************************/
/***************************auth模块*************************/
//mon生成auth
bool MDSDaemon::ms_get_authorizer(int dest_type, AuthAuthorizer **authorizer, bool force_new);
//验证授权
bool MDSDaemon::ms_verify_authorizer(Connection *con, int peer_type,
int protocol, bufferlist& authorizer_data, bufferlist& authorizer_reply,
bool& is_valid, CryptoKey& session_key,
std::unique_ptr<AuthAuthorizerChallenge> *challenge);
/**************************************************************/
/***************************session 连接accept*************************/
void MDSDaemon::ms_handle_accept(Connection *con)
{
Mutex::Locker l(mds_lock);
if (stopping) {
return;
}
Session *s = static_cast<Session *>(con->get_priv());
dout(10) << "ms_handle_accept " << con->get_peer_addr() << " con " << con << " session " << s << dendl;
if (s) {
if (s->connection != con) {
dout(10) << " session connection " << s->connection << " -> " << con << dendl;
s->connection = con;
// send out any queued messages
while (!s->preopen_out_queue.empty()) {
con->send_message(s->preopen_out_queue.front());
s->preopen_out_queue.pop_front();
}
}
s->put();
}
}
/*************************************************************/
/***************************clean shutdown*************************/
bool MDSDaemon::is_clean_shutdown()
{
if (mds_rank) {
return mds_rank->is_stopped();
} else {
return true;
}
}
/************************************************************/
//MDSRank.cc
/***************************init初始化***************************/
void MDSRankDispatcher::init()
{
//Objecter初始化,并且添加到消息头部,然后启动
objecter->init();
messenger->add_dispatcher_head(objecter);
objecter->start();
//更新配置文件中log配置信息
update_log_config();
create_logger();
handle_osd_map();
progress_thread.create("mds_rank_progr");
purge_queue.init();
finisher->start();
}
/***************************************************************/
/***************************tick***************************/
void MDSRankDispatcher::tick()
{
//重置heartbeat超时时间,避免被monitor kill
heartbeat_reset();
if (beacon.is_laggy()) {
dout(5) << "tick bailing out since we seem laggy" << dendl;
return;
}
//从op_tracker中读取到所有in_flight的操作名称
check_ops_in_flight();
//唤醒progress_thread线程
progress_thread.signal();
// make sure mds log flushes, trims periodically
mdlog->flush();
//如果是active,stopping去除cache,client_leases,log
if (is_active() || is_stopping()) {
mdcache->trim();
mdcache->trim_client_leases();
mdcache->check_memory_usage();
mdlog->trim(); // NOT during recovery!
}
// 更新log
if (logger) {
logger->set(l_mds_subtrees, mdcache->num_subtrees());
mdcache->log_stat();
}
// ...
if (is_clientreplay() || is_active() || is_stopping()) {
server->find_idle_sessions();
locker->tick();
}
//如果处于reconnect 标记
if (is_reconnect())
server->reconnect_tick();
if (is_active()) {
balancer->tick();
mdcache->find_stale_fragment_freeze();
mdcache->migrator->find_stale_export_freeze();
if (snapserver)
snapserver->check_osd_map(false);
}
if (is_active() || is_stopping()) {
update_targets(ceph_clock_now());
}
// shut down?
if (is_stopping()) {
mdlog->trim();
if (mdcache->shutdown_pass()) {
uint64_t pq_progress = 0 ;
uint64_t pq_total = 0;
size_t pq_in_flight = 0;
if (!purge_queue.drain(&pq_progress, &pq_total, &pq_in_flight)) {
dout(7) << "shutdown_pass=true, but still waiting for purge queue"
<< dendl;
// This takes unbounded time, so we must indicate progress
// to the administrator: we do it in a slightly imperfect way
// by sending periodic (tick frequency) clog messages while
// in this state.
clog->info() << "MDS rank " << whoami << " waiting for purge queue ("
<< std::dec << pq_progress << "/" << pq_total << " " << pq_in_flight
<< " files purging" << ")";
} else {
dout(7) << "shutdown_pass=true, finished w/ shutdown, moving to "
"down:stopped" << dendl;
stopping_done();
}
}
else {
dout(7) << "shutdown_pass=false" << dendl;
}
}
// Expose ourselves to Beacon to update health indicators
beacon.notify_health(this);
}
/***********************************************************/
/***************************shutdown***************************/
void MDSRankDispatcher::shutdown()
{
// It should never be possible for shutdown to get called twice, because
// anyone picking up mds_lock checks if stopping is true and drops
// out if it is.
assert(stopping == false);
stopping = true;
dout(1) << __func__ << ": shutting down rank " << whoami << dendl;
//关闭定时器
timer.shutdown();
//关闭mdlog
mdlog->shutdown();
//关闭mdcache
mdcache->shutdown();
purge_queue.shutdown();
mds_lock.Unlock();
finisher->stop(); // no flushing
mds_lock.Lock();
//关闭objecter
if (objecter->initialized)
objecter->shutdown();
//关闭monclient
monc->shutdown();
//关闭op_tracker
op_tracker.on_shutdown();
//关闭progress_thread
progress_thread.shutdown();
// release mds_lock for finisher/messenger threads (e.g.
// MDSDaemon::ms_handle_reset called from Messenger).
mds_lock.Unlock();
//关闭messenger
messenger->shutdown();
mds_lock.Lock();
//删除handle
if (hb) {
g_ceph_context->get_heartbeat_map()->remove_worker(hb);
hb = NULL;
}
}
/***********************************************************/
/*****************************admin socket asok******************************/
bool MDSRankDispatcher::handle_asok_command();
//剔除用户
void MDSRankDispatcher::evict_clients(const SessionFilter &filter, MCommand *m);
bool MDSRank::evict_client(int64_t session_id, bool wait, bool blacklist, std::stringstream& err_ss,Context *on_killed);
//dump用户session
void MDSRankDispatcher::dump_sessions(const SessionFilter &filter, Formatter *f);
void MDSRankDispatcher::update_log_config();
Session *MDSRank::get_session(Message *m);
void MDSRank::command_scrub_path(Formatter *f, boost::string_view path, vector<string>& scrubop_vec);
void MDSRank::command_tag_path(Formatter *f, boost::string_view path, boost::string_view tag);
void MDSRank::command_flush_path(Formatter *f, boost::string_view path);
void MDSRank::command_flush_journal(Formatter *f);
void MDSRank::command_get_subtrees(Formatter *f);
void MDSRank::command_export_dir(Formatter *f, boost::string_view path, mds_rank_t target);
bool MDSRank::command_dirfrag_split(cmdmap_t cmdmap, std::ostream &ss);
bool MDSRank::command_dirfrag_merge(cmdmap_t cmdmap, std::ostream &ss);
bool MDSRank::command_dirfrag_ls(cmdmap_t cmdmap, std::ostream &ss, Formatter *f);
void MDSRank::dump_status(Formatter *f);
void MDSRank::dump_clientreplay_status(Formatter *f);
void MDSRank::create_logger();
/***************************************************************************/
/*****************************消息分发调度******************************/
bool MDSRankDispatcher::ms_dispatch(Message *m);
bool MDSRank::_dispatch(Message *m, bool new_msg)
{
//如果message不是mds发送过来,则直接返回
if (is_stale_message(m)) {
m->put();
return true;
}
//如果mds处于laggy状态,将消息放入waiting_for_nolaggy数组
if (beacon.is_laggy()) {
dout(10) << " laggy, deferring " << *m << dendl;
waiting_for_nolaggy.push_back(m);
}
//如果消息是新消息并且waiting_for_nolaggy数组不为空, 则放入waiting_for_nolaggy中
else if (new_msg && !waiting_for_nolaggy.empty()) {
dout(10) << " there are deferred messages, deferring " << *m << dendl;
waiting_for_nolaggy.push_back(m);
} else {
if (!handle_deferrable_message(m)) {
dout(0) << "unrecognized message " << *m << dendl;
return false;
}
heartbeat_reset();
}
...
//如果mds处于laggy状态,则直接返回
if (beacon.is_laggy()) {
// We've gone laggy during dispatch, don't do any
// more housekeeping
return true;
}
// done with all client replayed requests?
if (is_clientreplay() &&
mdcache->is_open() &&
replay_queue.empty() &&
beacon.get_want_state() == MDSMap::STATE_CLIENTREPLAY) {
int num_requests = mdcache->get_num_client_requests();
dout(10) << " still have " << num_requests << " active replay requests" << dendl;
if (num_requests == 0)
clientreplay_done();
}
...
update_mlogger();
return true;
}
//延期待处理的消息
bool MDSRank::handle_deferrable_message(Message *m)
{
int port = m->get_type() & 0xff00;
switch (port) {
//cache类型消息,由mdcache处理
case MDS_PORT_CACHE:
ALLOW_MESSAGES_FROM(CEPH_ENTITY_TYPE_MDS);
mdcache->dispatch(m);
break;
//migrator类型消息,由migrator处理
case MDS_PORT_MIGRATOR:
ALLOW_MESSAGES_FROM(CEPH_ENTITY_TYPE_MDS);
mdcache->migrator->dispatch(m);
break;
default:
//client session,slave消息,由server处理
switch (m->get_type()) {
// SERVER
case CEPH_MSG_CLIENT_SESSION:
case CEPH_MSG_CLIENT_RECONNECT:
ALLOW_MESSAGES_FROM(CEPH_ENTITY_TYPE_CLIENT);
// fall-thru
case CEPH_MSG_CLIENT_REQUEST:
server->dispatch(m);
break;
case MSG_MDS_SLAVE_REQUEST:
ALLOW_MESSAGES_FROM(CEPH_ENTITY_TYPE_MDS);
server->dispatch(m);
break;
//heartbeat消息,有balancer处理
case MSG_MDS_HEARTBEAT:
ALLOW_MESSAGES_FROM(CEPH_ENTITY_TYPE_MDS);
balancer->proc_message(m);
break;
case MSG_MDS_TABLE_REQUEST:
ALLOW_MESSAGES_FROM(CEPH_ENTITY_TYPE_MDS);
{
MMDSTableRequest *req = static_cast<MMDSTableRequest*>(m);
if (req->op < 0) {
MDSTableClient *client = get_table_client(req->table);
client->handle_request(req);
} else {
MDSTableServer *server = get_table_server(req->table);
server->handle_request(req);
}
}
break;
//lock消息,由locker处理
case MSG_MDS_LOCK:
case MSG_MDS_INODEFILECAPS:
ALLOW_MESSAGES_FROM(CEPH_ENTITY_TYPE_MDS);
locker->dispatch(m);
break;
//client caps消息,由locker处理
case CEPH_MSG_CLIENT_CAPS:
case CEPH_MSG_CLIENT_CAPRELEASE:
case CEPH_MSG_CLIENT_LEASE:
ALLOW_MESSAGES_FROM(CEPH_ENTITY_TYPE_CLIENT);
locker->dispatch(m);
break;
default:
return false;
}
}
return true;
}
void MDSRank::_advance_queues();
void MDSRank::heartbeat_reset();
/******************************************************************/
/*****************************消息发送******************************/
void MDSRank::send_message(Message *m, Connection *c);
void MDSRank::send_message_mds(Message *m, mds_rank_t mds);
void MDSRank::forward_message_mds(Message *m, mds_rank_t mds);
void MDSRank::send_message_client_counted(Message *m, client_t client);
void MDSRank::send_message_client_counted(Message *m, Connection *connection);
void MDSRank::send_message_client_counted(Message *m, Session *session);
void MDSRank::send_message_client(Message *m, Session *session);
/******************************************************************/
/*****************************类成员相关******************************/
int64_t MDSRank::get_metadata_pool();
MDSTableClient *MDSRank::get_table_client(int t);
MDSTableServer *MDSRank::get_table_server(int t);
utime_t MDSRank::get_laggy_until();
void MDSRank::request_state(MDSMap::DaemonState s);
/*******************************************************************/
/*****************************MDSRank状态相关******************************/
//自杀
void MDSRank::suicide();
//重生
void MDSRank::respawn();
//损坏
void MDSRank::damaged();
void MDSRank::damaged_unlocked();
void MDSRank::handle_write_error(int err)
{
//如果错误为-EBLACKLISTED,则重启MDS
if (err == -EBLACKLISTED) {
derr << "we have been blacklisted (fenced), respawning..." << dendl;
respawn();
return;
}
//如果mds_action_on_write_error大于等于2,则重启MDS
if (g_conf->mds_action_on_write_error >= 2) {
derr << "unhandled write error " << cpp_strerror(err) << ", suicide..." << dendl;
respawn();
} else if (g_conf->mds_action_on_write_error == 1) {
derr << "unhandled write error " << cpp_strerror(err) << ", force readonly..." << dendl;
mdcache->force_readonly();
} else {
// ignore;
derr << "unhandled write error " << cpp_strerror(err) << ", ignore..." << dendl;
}
}
//消息是否来着mds
bool MDSRank::is_stale_message(Message *m);
/********************************************************************/
/*****************************ProgressThread相关******************************/
void *MDSRank::ProgressThread::entry();
void MDSRank::ProgressThread::shutdown();
/***************************************************************************/
/*****************************boot相关******************************/
void MDSRank::boot_start(BootStep step, int r);
void MDSRank::validate_sessions();
void MDSRank::starting_done();
void MDSRank::boot_create();
oid MDSRank::creating_done();
/*****************************boot相关******************************/
/*****************************replay相关******************************/
void MDSRank::calc_recovery_set();
void MDSRank::replay_start();
void MDSRank::_standby_replay_restart_finish(int r, uint64_t old_read_pos);
void MDSRank::standby_replay_restart();
void MDSRank::replay_done();
/*******************************************************************/
/*****************************resolve相关******************************/
void MDSRank::reopen_log();
void MDSRank::resolve_start();
void MDSRank::resolve_done();
/*********************************************************************/
/*****************************reconnect相关******************************/
void MDSRank::reconnect_start();
void MDSRank::reconnect_done();
/***********************************************************************/
/*****************************rejoin相关******************************/
void MDSRank::rejoin_joint_start();
void MDSRank::rejoin_start();
void MDSRank::rejoin_done();
/********************************************************************/
/*****************************clientreplay相关******************************/
void MDSRank::clientreplay_start();
bool MDSRank::queue_one_replay();
void MDSRank::clientreplay_done();
/*************************************************************************/
/*****************************active相关******************************/
void MDSRank::active_start();
/********************************************************************/
/*****************************recovery相关******************************/
oid MDSRank::recovery_done(int oldstate);
/**********************************************************************/
/*****************************creating_相关******************************/
void MDSRank::creating_done();
/***********************************************************************/
/*****************************stopping相关******************************/
void MDSRank::stopping_start();
void MDSRank::stopping_done();
/**********************************************************************/
/*****************************handle_mds_map相关******************************/
void MDSRankDispatcher::handle_mds_map(MMDSMap *m, MDSMap *oldmap);
void MDSRank::handle_mds_recovery(mds_rank_t who);
void MDSRank::handle_mds_failure(mds_rank_t who);
/***************************************************************************/
作者: 李航
个人简介: 多年的底层开发经验,在高性能nginx开发和分布式缓存redis cluster有着丰富的经验,目前从事Ceph工作。
先后在58同城、汽车之家、优酷土豆集团工作。
目前供职于滴滴基础平台-技术专家岗位 主要负责分布式Ceph系统。
个人主要关注的技术领域:高性能Nginx开发、分布式缓存、分布式存储。
如果觉得我的文章对您有用,请点赞。您的支持将鼓励我继续创作!
赞5
添加新评论0 条评论