在Kubernetes下Prometheus需要与Kubernetes的API进行交互,从而能够动态的发现Kubernetes中部署的所有可监控的目标资源。
目前主要支持5种服务发现模式,分别是:Node、Service、Pod、Endpoints、Ingress。
为了让Prometheus能够获取到当前集群中所有节点的信息, 我们在Prometheus配置文件中 ,添加如下Job配置:
- job_name: 'kubernetes-nodes'
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
# 这里需要指定用于访问Kubernetes API的ca以及token文件路径
kubernetes_sd_configs:
- role: node
# 通过指定kubernetes_sd_config的模式为node,Prometheus会自动从Kubernetes中发现到所有的node节点并作为当前Job监控的Target实例,发现的节点/metrics接口是默认的kubelet的HTTP接口
对于Ingress,Service,Endpoints, Pod的使用方式也是类似的,这里给出了一个完整Prometheus配置的示例:
apiVersion: v1
data:
prometheus.yml: |-
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: 'kubernetes-nodes'
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
- job_name: 'kubernetes-service'
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: service
- job_name: 'kubernetes-endpoints'
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: endpoints
- job_name: 'kubernetes-ingress'
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: ingress
- job_name: 'kubernetes-pods'
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: pod
kind: ConfigMap
metadata:
name: prometheus-config
更新配置文件, 并重建Prometheus实例:
$ kubectl apply -f prometheus-config.yml
configmap "prometheus-config" configured
$ kubectl get pods
prometheus-69f9ddb588-rbrs2 1/1 Running 0 4m
$ kubectl delete pods prometheus-69f9ddb588-rbrs2
pod "prometheus-69f9ddb588-rbrs2" deleted
$ kubectl get pods
prometheus-69f9ddb588-rbrs2 0/1 Terminating 0 4m
prometheus-69f9ddb588-wtlsn 1/1 Running 0 14s
Prometheus使用新的配置文件重建之后,打开Prometheus UI,通过Service Discovery页面可以查看到当前Prometheus通过Kubernetes发现的所有资源对象。
同时Prometheus会自动将该资源的所有信息,并通过标签的形式体现在Target对象上。