K8S的mater节点变成not ready状态
[root@k8s-master kubernetes]# kubectl get node
NAME STATUS AGE
127.0.0.1 Ready 18h
k8s-master NotReady 42m
k8s-node1 Ready 1d
k8s-node2 Ready 1d
root@k8s-node1 kubernetes]# kubectl get node
NAME STATUS AGE
127.0.0.1 Ready 18h
k8s-master NotReady 34m
k8s-node1 Ready 1d
k8s-node2 Ready 1d
[root@k8s-node2 kubernetes]# kubectl get node
NAME STATUS AGE
127.0.0.1 Ready 18h
k8s-master NotReady 35m
k8s-node1 Ready 1d
k8s-node2 Ready 1d
理论上get node都不显示master才对?哪里配置有问题?
master上, systemctl restart kubelet看下,
如果不行
journalctl -u kubelet看下日志报什么错
也有可能是master 网络的问题
[root@k8s-master kubernetes]# journalctl -u kubelet
-- Logs begin at Tue 2020-08-18 13:50:47 CST, end at Tue 2020-08-18 16:31:26 CST. --
Aug 18 13:58:08 k8s-master systemd[1]: Started Kubernetes Kubelet Server.
Aug 18 13:58:08 k8s-master kubelet[2585]: Flag --api-servers has been deprecated, Use --kubeconfig instead. Will be removed in a future version.
Aug 18 13:58:08 k8s-master kubelet[2585]: I0818 13:58:08.863977 2585 feature_gate.go:181] feature gates: map[]
Aug 18 13:58:08 k8s-master kubelet[2585]: W0818 13:58:08.864682 2585 server.go:605] Could not load kubeconfig file /var/lib/kubelet/kubeconfig: stat /var/lib/kubelet/kubeconfig: no such file or directory. Using default client conf
Aug 18 13:58:09 k8s-master kubelet[2585]: I0818 13:58:09.262180 2585 docker.go:356] Connecting to docker on unix:///var/run/docker.sock
Aug 18 13:58:09 k8s-master kubelet[2585]: I0818 13:58:09.262212 2585 docker.go:376] Start docker client with request timeout=2m0s
Aug 18 13:58:09 k8s-master kubelet[2585]: E0818 13:58:09.267095 2585 cni.go:163] error updating cni config: No networks found in /etc/cni/net.d
Aug 18 13:58:09 k8s-master kubelet[2585]: I0818 13:58:09.275445 2585 manager.go:143] cAdvisor running in container: "/system.slice/kubelet.service"
Aug 18 13:58:09 k8s-master kubelet[2585]: W0818 13:58:09.293213 2585 manager.go:151] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
Aug 18 13:58:09 k8s-master kubelet[2585]: I0818 13:58:09.303623 2585 fs.go:117] Filesystem partitions: map[/dev/sda1:{mountpoint:/boot major:8 minor:1 fsType:xfs blockSize:0} /dev/mapper/rootvg-root:{mountpoint:/ major:253 minor:0
Aug 18 13:58:09 k8s-master kubelet[2585]: I0818 13:58:09.307865 2585 manager.go:198] Machine: {NumCores:4 CpuFrequency:2294686 MemoryCapacity:4142161920 MachineID:06571e85e6264c8a9c9a098fdd87b052 SystemUUID:3D7E2542-5E2A-F702-FB8F
Aug 18 13:58:09 k8s-master kubelet[2585]: I0818 13:58:09.308427 2585 manager.go:204] Version: {KernelVersion:3.10.0-1127.18.2.el7.x86_64 ContainerOsVersion:CentOS Linux 7 (Core) DockerVersion:1.13.1 CadvisorVersion: CadvisorRevisi
Aug 18 13:58:09 k8s-master kubelet[2585]: I0818 13:58:09.310790 2585 kubelet.go:252] Watching apiserver
Aug 18 13:58:09 k8s-master kubelet[2585]: W0818 13:58:09.313851 2585 kubelet_network.go:69] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
Aug 18 13:58:09 k8s-master kubelet[2585]: I0818 13:58:09.313884 2585 kubelet.go:477] Hairpin mode set to "hairpin-veth"
Aug 18 13:58:09 k8s-master kubelet[2585]: I0818 13:58:09.326968 2585 docker_manager.go:262] Setting dockerRoot to /var/lib/docker
Aug 18 13:58:09 k8s-master kubelet[2585]: I0818 13:58:09.326993 2585 docker_manager.go:265] Setting cgroupDriver to systemd
Aug 18 13:58:09 k8s-master kubelet[2585]: I0818 13:58:09.328314 2585 server.go:770] Started kubelet v1.5.2
Aug 18 13:58:09 k8s-master kubelet[2585]: I0818 13:58:09.328810 2585 server.go:123] Starting to listen on 127.0.0.1:10250
Aug 18 13:58:09 k8s-master kubelet[2585]: E0818 13:58:09.329430 2585 kubelet.go:1146] Image garbage collection failed: unable to find data for container /
Aug 18 13:58:09 k8s-master kubelet[2585]: I0818 13:58:09.331115 2585 fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer
Aug 18 13:58:09 k8s-master kubelet[2585]: I0818 13:58:09.331152 2585 status_manager.go:129] Starting to sync pod status with apiserver
Aug 18 13:58:09 k8s-master kubelet[2585]: I0818 13:58:09.331169 2585 kubelet.go:1715] Starting kubelet main sync loop.
Aug 18 13:58:09 k8s-master kubelet[2585]: I0818 13:58:09.331185 2585 kubelet.go:1726] skipping pod synchronization - [container runtime is down]
Aug 18 13:58:09 k8s-master kubelet[2585]: I0818 13:58:09.332500 2585 volume_manager.go:244] Starting Kubelet Volume Manager
Aug 18 13:58:09 k8s-master kubelet[2585]: I0818 13:58:09.346535 2585 factory.go:305] Registering Docker factory
Aug 18 13:58:09 k8s-master kubelet[2585]: W0818 13:58:09.346568 2585 manager.go:247] Registration of the rkt container factory failed: unable to communicate with Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1
Aug 18 13:58:09 k8s-master kubelet[2585]: I0818 13:58:09.346582 2585 factory.go:54] Registering systemd factory
Aug 18 13:58:09 k8s-master kubelet[2585]: I0818 13:58:09.346779 2585 factory.go:86] Registering Raw factory
Aug 18 13:58:09 k8s-master kubelet[2585]: I0818 13:58:09.346962 2585 manager.go:1106] Started watching for new ooms in manager
Aug 18 13:58:09 k8s-master kubelet[2585]: I0818 13:58:09.349012 2585 oomparser.go:185] oomparser using systemd
Aug 18 13:58:09 k8s-master kubelet[2585]: I0818 13:58:09.349815 2585 manager.go:288] Starting recovery of all containers
Aug 18 13:58:09 k8s-master kubelet[2585]: I0818 13:58:09.434159 2585 kubelet_node_status.go:227] Setting node annotation to enable volume controller attach/detach
Aug 18 13:58:09 k8s-master kubelet[2585]: I0818 13:58:09.438486 2585 kubelet_node_status.go:74] Attempting to register node 127.0.0.1
Aug 18 13:58:09 k8s-master kubelet[2585]: I0818 13:58:09.441564 2585 manager.go:293] Recovery completed
Aug 18 13:58:09 k8s-master kubelet[2585]: I0818 13:58:09.448737 2585 kubelet_node_status.go:113] Node 127.0.0.1 was previously registered
Aug 18 13:58:09 k8s-master kubelet[2585]: I0818 13:58:09.448754 2585 kubelet_node_status.go:77] Successfully registered node 127.0.0.1
Aug 18 13:59:26 k8s-master kubelet[2585]: W0818 13:59:26.698293 2585 reflector.go:319] pkg/kubelet/config/apiserver.go:44: watch of *api.Pod ended with: too old resource version: 131552 (131743)
Aug 18 13:59:26 k8s-master kubelet[2585]: W0818 13:59:26.704430 2585 reflector.go:319] pkg/kubelet/kubelet.go:378: watch of *api.Service ended with: too old resource version: 131353 (131743)
Aug 18 13:59:26 k8s-master kubelet[2585]: W0818 13:59:26.708569 2585 reflector.go:319] pkg/kubelet/kubelet.go:386: watch of *api.Node ended with: too old resource version: 131735 (131743)
Aug 18 13:59:40 k8s-master kubelet[2585]: W0818 13:59:40.021133 2585 conversion.go:110] Could not get instant cpu stats: cumulative stats decrease
Aug 18 14:00:00 k8s-master kubelet[2585]: W0818 14:00:00.104740 2585 conversion.go:110] Could not get instant cpu stats: cumulative stats decrease
Aug 18 14:00:06 k8s-master systemd[1]: Stopping Kubernetes Kubelet Server...
Aug 18 14:00:06 k8s-master systemd[1]: Stopped Kubernetes Kubelet Server.
Aug 18 14:00:09 k8s-master systemd[1]: Started Kubernetes Kubelet Server.
收起