基于 Cilium 和 eBPF 检测容器逃逸
如果运行云原生工作负载均衡设施，则可以更好地保护我们的服务。毕竟，服务经常向公众暴露以及工作负载可能属于不同的租户。在这篇博文中，我将向大家展示访问我们的 Kubernetes 集群的攻击者如何进行容器逃逸：运行 Pod 以获得 root 权限，将 Pod 转义到主机上，并通过不可见的 Pod 和无文件执行来持续攻击。同时，我将向大家展示如何基于 Isovalent Cilium Enterprise 进行攻击检测。
问 > 题
在容器逃逸期间，攻击者打破了主机和容器之间的隔离边界，最终逃逸至 Kubernetes 控制平面或工作节点的地方。在这种情况下，攻击者可以看到在同一主机上运行的其他容器，收集他们的机密信息，在主机文件系统上读写数据，攻击 Kubelet 并提升权限；或者通过部署一个不可见的 Pod 来利用Kubernetes 错误并在环境中持久存在。
在 Kubernetes 环境中应用安全最佳实践可以限制这些类型的攻击，但容器突破仍然是可能的，攻击者可以使用特权 Pod 或利用现有漏洞来获得特权。安全团队需要衡量强化配置是否合适以及应用的保护是否有效。
解决方案
实现这一目标的一种方法是遵循数据驱动方法的可观察性：从 Kubernetes 工作负载和主机收集数据，观察反馈，并做出持续的数据驱动决策以保护Kubernetes 环境。
通过使用 eBPF，安全团队可以直接获得对任何 Kubernetes 工作负载（例如 Pod）的独特可见性。由于 Kubernetes 节点上的 Pod 共享一个内核，因此 Pod 中的每个进程对单个 eBPF 程序都是可见的。这可以提供对节点上运行的每个进程的完整可见性，无论它们是由 systemd 管理的主机上的长期运行进程还是在容器内运行的短期进程。
Cilium
Cilium 使用 eBPF 非常有效地监控 Kubernetes 工作负载内部和主机外部的所有网络和进程行为，并为我们提供对这些行为的 Kubernetes 身份感知和操作系统级进程可见性。
Cilium 在 Kubernetes 环境中以守护进程模式存在。因此，每个 Kubernetes 节 点上都有一个 Cilium 代理运行，它与 Kubernetes API 服务器通信以掌控 Kubernetes Pod 身份、网络策略、服务等。然后根据部署在 Kubernetes 环境中的每个工作负载的身份 , Cilium 安装了一个高效的 eBPF 程序来为这些工作负载做连接性、可观察性和安全性追踪。
丰富的安全事件
Cilium 能够观察并强制执行 Linux 系统内部发生的行为。它可以直接在内核中收集和过滤安全可观察性数据，并将其作为 JSON 事件导出到用户空间和/或通过名为 hubble-enterprise 的 Daemonset 将它们存储在特定的日志文件中。这些 JSON 事件通过 Kubernetes 身份感知信息（包括服务、标签、命名空间、Pod 和容器）以及操作系统级进程可见性数据（包括进程二进制文件、pid、uid、具有完整进程祖先树的父二进制文件）进行了丰富。然后，这些事件可以以各种格式导出并发送到外部系统，例如 SIEM，例如：Elasticsearch、Splunk 或存储在 S3 存储桶中。为简单起见，在这篇博文中，它们将直接从日志文件中使用。
通过 Cilium 利用内核中的实时网络和进程级可见性数据，安全团队能够查看在其 Kubernetes 环境中执行的所有进程，这有助于他们做出持续的数据驱动决策并提高安全性他们系统的姿势。
让我们进入主机命名空间
在此示例中，我们使用具有主机命名空间配置的特权 Pod 来表示容器逃逸攻击。正如我们在此处演示的那样，这在强化的 Kubernetes 环境中是可能的。请注意，有多种方法可以执行突破，例如，攻击者也可以利用漏洞获得特权并逃出容器沙箱。
攻击者执行容器逃逸的第一个也是最简单的步骤是使用特权 Pod 规范启动一个 Pod。注意：Kubernetes 默认允许这样做，并且特权标志授予容器所有可用的内核功能。hostPID 和 hostNetwork 标志将容器置于主机 PID 和网络命名空间中，因此它可以查看所有进程和网络资源并与之交互。可以在以下 yaml 文件中找到一个简单的示例：
[administrator@JavaLangOutOfMemory ~ ] % cat privileged.yaml
apiVersion: v1
kind: Pod
metadata:
name: privileged-the-pod
spec:
hostPID: true
hostNetwork: true
containers:
- name: privileged-the-pod
image: nginx:latest
ports:
- containerPort: 80
securityContext:
privileged: true
然后，让我们执行该特权 Pod 规范：
[administrator@JavaLangOutOfMemory ~ ] % kubectl apply -f privileged.yaml
pod/privileged-the-pod created
[administrator@JavaLangOutOfMemory ~ ] % kubectl get pods
NAME READY STATUS RESTARTS AGE
privileged-the-pod 1/1 Running 0 11s
现在，攻击者拥有一个特权 Pod 并正在运行，该 Pod 授予他们与作为基础节点上的 root 用户相同的权限。 为什么如此强大？ 因为 Pod 已经开始具有这些功能，包括 CAP_SYS_ADMIN，它本质上是 Linux 中的“新根”，还可以访问主机上的所有设备。 此功能与 HostPID 相结合，使攻击者可以访问已部署的所有容器命名空间，因此他们可以与部署特权 Pod 的底层节点上的任何其他进程或文件系统进行交互并加以利用。
通过使用 Cilium，安全团队可以通过执行以下命令获取导出到用户空间的以下 process_exec 事件来检测任何特权容器执行：
[administrator@JavaLangOutOfMemory ~ ] % kubectl logs -n kube-system ds/hubble-enterprise -c export-stdout
{
"process_exec":{
"process":{
"exec_id":"bWluaWt1YmU6MTEzNzkyNjAzMjk3MjoxNzk3OA==",
"pid":17978,
"uid":0,
"cwd":"/",
"binary":"/docker-entrypoint.sh",
"arguments":"/docker-entrypoint.sh nginx -g \\"daemon off;\\"",
"flags":"execve rootcwd clone",
"start_time":"2021-10-13T12:58:31.794Z",
"auid":4294967295,
"pod":{
"namespace":"default",
"name":"privileged-the-pod",
"container":{
"id":"docker://32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158",
"name":"privileged-the-pod",
"image":{
"id":"docker-pullable://nginx@sha256:644a70516a26004c97d0d85c7fe1d0c3a67ea8ab7ddf4aff193d9f301670cf36",
"name":"nginx:latest"
},
"start_time":"2021-10-13T12:58:31Z"
}
},
"docker":"32865cff8fef4a9274e9fa1d",
"parent_exec_id":"bWluaWt1YmU6MTEzNzgyOTM1MzU5NzoxNzk1OA==",
"refcnt":1,
"cap":{
"permitted":[
"CAP_CHOWN",
"DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_LINUX_IMMUTABLE",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_ADMIN",
"CAP_NET_RAW",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_SYS_MODULE",
"CAP_SYS_RAWIO",
"CAP_SYS_CHROOT",
"CAP_SYS_PTRACE",
"CAP_SYS_PACCT",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_NICE",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_MKNOD",
"CAP_LEASE",
"CAP_AUDIT_WRITE",
"CAP_AUDIT_CONTROL",
"CAP_SETFCAP",
"CAP_MAC_OVERRIDE",
"CAP_MAC_ADMIN",
"CAP_SYSLOG",
"CAP_WAKE_ALARM",
"CAP_BLOCK_SUSPEND",
"CAP_AUDIT_READ"
],
"effective":[
"CAP_CHOWN",
"DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_LINUX_IMMUTABLE",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_ADMIN",
"CAP_NET_RAW",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_SYS_MODULE",
"CAP_SYS_RAWIO",
"CAP_SYS_CHROOT",
"CAP_SYS_PTRACE",
"CAP_SYS_PACCT",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_NICE",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_MKNOD",
"CAP_LEASE",
"CAP_AUDIT_WRITE",
"CAP_AUDIT_CONTROL",
"CAP_SETFCAP",
"CAP_MAC_OVERRIDE",
"CAP_MAC_ADMIN",
"CAP_SYSLOG",
"CAP_WAKE_ALARM",
"CAP_BLOCK_SUSPEND",
"CAP_AUDIT_READ"
],
"inheritable":[
"CAP_CHOWN",
"DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_LINUX_IMMUTABLE",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_ADMIN",
"CAP_NET_RAW",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_SYS_MODULE",
"CAP_SYS_RAWIO",
"CAP_SYS_CHROOT",
"CAP_SYS_PTRACE",
"CAP_SYS_PACCT",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_NICE",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_MKNOD",
"CAP_LEASE",
"CAP_AUDIT_WRITE",
"CAP_AUDIT_CONTROL",
"CAP_SETFCAP",
"CAP_MAC_OVERRIDE",
"CAP_MAC_ADMIN",
"CAP_SYSLOG",
"CAP_WAKE_ALARM",
"CAP_BLOCK_SUSPEND",
"CAP_AUDIT_READ"
]
}
},
"parent":{
"exec_id":"bWluaWt1YmU6MTEzNzgyOTM1MzU5NzoxNzk1OA==",
"pid":17958,
"uid":0,
"cwd":"/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158/",
"binary":"/usr/bin/containerd-shim",
"arguments":"-namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf4 eb80d28b94e273d6d1670a6f721a9a1158 -address /var/run/docker/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc -systemd-cgroup",
"flags":"execve clone",
"start_time":"2021-10-13T12:58:31.698Z",
"auid":4294967295,
"parent_exec_id":"bWluaWt1YmU6NDIwODAwMDAwMDA6MjY4MA==",
"refcnt":2
},
"ancestors":[
{
"exec_id":"bWluaWt1YmU6NDIwODAwMDAwMDA6MjY4MA==",
"pid":2680,
"uid":0,
"cwd":"/",
"binary":"/usr/bin/containerd",
"arguments":"--config /var/run/docker/containerd/containerd.toml --log-level info",
"flags":"procFS auid rootcwd",
"start_time":"2021-10-13T12:40:15.948Z",
"auid":0, "
parent_exec_id":"bWluaWt1YmU6NDIwNDAwMDAwMDA6MjY3Mg==",
"refcnt":63
},
{
"exec_id":"bWluaWt1YmU6NDIwNDAwMDAwMDA6MjY3Mg==",
"pid":2672,
"uid":0,
"cwd":"/",
"binary":"/usr/bin/dockerd",
"arguments":"-H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=virtualbox --insecure-registry 10.96.0.0/12",
"flags":"procFS auid rootcwd",
"start_time":"2021-10-13T12:40:15.908Z",
"auid":0,
"parent_exec_id":"bWluaWt1YmU6MjEwMDAwMDAwOjE=",
"refcnt":65
},
{
"exec_id":"bWluaWt1YmU6MjEwMDAwMDAwOjE=",
"pid":1,
"uid":0,
"cwd":"/",
"binary":"/usr/lib/systemd/systemd",
"arguments":"noembed norestore",
"flags":"procFS auid rootcwd",
"start_time":"2021-10-13T12:39:34.078Z",
"auid":0,
"refcnt":101
}
]
},
"node_name":"minikube",
"time":"2021-10-13T12:58:31.794Z"
}
其次，他们也可以看到如下信息：
1、Kubernetes 身份感知信息，例如命名空间：default、pod 名称：privileged-the-pod、container-id 和标签。
2、操作系统级别的可见性信息，例如二进制文件：/docker-entrypoint.sh、
pid：23715、uid：0 和参数：nginx -g \"daemon off。
3、包含 /usr/bin/containerd-shim 作为直接父进程二进制文件的完整进程根树。
4、容器已启动的功能，包括 CAP_NET_RAW 和 CAP_SYS_ADMIN。
第二步，攻击者可以使用 kubectl exec 来获取对特权 Pod 的 shell 访问：
[administrator@JavaLangOutOfMemory ~ ] % kubectl exec -it privileged-the-pod -- /bin/bash
启 动 后 突 然 在 容 器 日 志 中 弹 出 的 外 壳 当 然 是 安 全 团 队 感 兴 趣 的 事 件 。 他 们 可 以 通 过 获 取 以 下 通 过 C i l i u m 导 出 到 用 户 空 间 的 p r o c e s s _ e x e c 事 件 来 检 测 b a s h 执 行 。 在 第 4 行 和 第 1 1 行 之 间 可 以 看 到 进 程 信 息 ， 而 在 第 1 2 行 和 第 2 4 行 之 间 可 以 看 到 K u b e r n e t e s 身 份 感 知 信 息。
{
"process_exec":{
"process":{
"exec_id":"bWluaWt1YmU6MTI5NDM3OTU0NzQ3ODoxOTU5NA==",
"pid":19594,
"uid":0,
"cwd":"/",
"binary":"/bin/bash",
"flags":"execve rootcwd clone",
"start_time":"2021-10-13T13:01:08.248Z",
"auid":4294967295,
"pod":{
"namespace":"default",
"name":"privileged-the-pod",
"container":{
"id":"docker://32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158",
"name":"privileged-the-pod",
"image":{
"id":"docker-pullable://nginx@sha256:644a70516a26004c97d0d85c7fe1d0c3a67ea8ab7ddf4aff193d9f301670cf36",
"name":"nginx:latest"
},
"start_time":"2021-10-13T12:58:31Z"
}
},
"docker":"32865cff8fef4a9274e9fa1d",
"parent_exec_id":"bWluaWt1YmU6MTI5NDMzMzczMTY4NToxOTU4NA==",
"refcnt":1,
"cap":{
"permitted":[
"CAP_CHOWN",
"DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_LINUX_IMMUTABLE",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_ADMIN",
"CAP_NET_RAW",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_SYS_MODULE",
"CAP_SYS_RAWIO",
"CAP_SYS_CHROOT",
"CAP_SYS_PTRACE",
"CAP_SYS_PACCT",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_NICE",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_MKNOD",
"CAP_LEASE",
"CAP_AUDIT_WRITE",
"CAP_AUDIT_CONTROL",
"CAP_SETFCAP",
"CAP_MAC_OVERRIDE",
"CAP_MAC_ADMIN",
"CAP_SYSLOG",
"CAP_WAKE_ALARM",
"CAP_BLOCK_SUSPEND",
"CAP_AUDIT_READ"
],
"effective":[
"CAP_CHOWN",
"DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_LINUX_IMMUTABLE",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_ADMIN",
"CAP_NET_RAW",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_SYS_MODULE",
"CAP_SYS_RAWIO",
"CAP_SYS_CHROOT",
"CAP_SYS_PTRACE",
"CAP_SYS_PACCT",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_NICE",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_MKNOD",
"CAP_LEASE",
"CAP_AUDIT_WRITE",
"CAP_AUDIT_CONTROL",
"CAP_SETFCAP",
"CAP_MAC_OVERRIDE",
"CAP_MAC_ADMIN",
"CAP_SYSLOG",
"CAP_WAKE_ALARM",
"CAP_BLOCK_SUSPEND",
"CAP_AUDIT_READ"
],
"inheritable":[
"CAP_CHOWN",
"DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_LINUX_IMMUTABLE",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_ADMIN",
"CAP_NET_RAW",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_SYS_MODULE",
"CAP_SYS_RAWIO",
"CAP_SYS_CHROOT",
"CAP_SYS_PTRACE",
"CAP_SYS_PACCT",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_NICE",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_MKNOD",
"CAP_LEASE",
"CAP_AUDIT_WRITE",
"CAP_AUDIT_CONTROL",
"CAP_SETFCAP",
"CAP_MAC_OVERRIDE",
"CAP_MAC_ADMIN",
"CAP_SYSLOG",
"CAP_WAKE_ALARM",
"CAP_BLOCK_SUSPEND",
"CAP_AUDIT_READ"
]
}
},
"parent":{
"exec_id":"bWluaWt1YmU6MTI5NDMzMzczMTY4NToxOTU4NA==",
"pid":19584,
"uid":0,
"cwd":"/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158/",
"binary":"/usr/bin/runc",
"arguments":"--root /var/run/docker/runtime-runc/moby --log /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf48eb8 d28b94e273d6d1670a6f721a9a1158/log.json --log-format json --systemd-cgroup exec --process /tmp/runc-process133903661 --console-socket /tmp/pty028492678/pty.sock --detach --pid-file /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf48eb8 d28b94e273d6d1670a6f721a9a1158/a5579f11fb7d75d66213f488bf44e9c37b92c196dee5e94647e6f60c59cf6693.pid 32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158",
"flags":"execve clone",
"start_time":"2021-10-13T13:01:08.202Z",
"auid":4294967295,
"parent_exec_id":"bWluaWt1YmU6MTEzNzgyOTM1MzU5NzoxNzk1OA==",
"refcnt":2
},
"ancestors":[
{
"exec_id":"bWluaWt1YmU6MTEzNzgyOTM1MzU5NzoxNzk1OA==",
"pid":17958,
"uid":0,
"cwd":"/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158/",
"binary":"/usr/bin/containerd-shim",
"arguments":"-namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf4 eb80d28b94e273d6d1670a6f721a9a1158 -address /var/run/docker/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc -systemd-cgroup",
"flags":"execve clone",
"start_time":"2021-10-13T12:58:31.698Z",
"auid":4294967295,
"parent_exec_id":"bWluaWt1YmU6NDIwODAwMDAwMDA6MjY4MA==",
"refcnt":4
},
{
"exec_id":"bWluaWt1YmU6NDIwODAwMDAwMDA6MjY4MA==", "
pid":2680,
"uid":0,
"cwd":"/",
"binary":"/usr/bin/containerd",
"arguments":"--config /var/run/docker/containerd/containerd.toml --log-level info",
"flags":"procFS auid rootcwd",
"start_time":"2021-10-13T12:40:15.948Z",
"auid":0,
"parent_exec_id":"bWluaWt1YmU6NDIwNDAwMDAwMDA6MjY3Mg==",
"refcnt":65
},
{
"exec_id":"bWluaWt1YmU6NDIwNDAwMDAwMDA6MjY3Mg==",
"pid":2672,
"uid":0,
"cwd":"/",
"binary":"/usr/bin/dockerd",
"arguments":"-H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=virtualbox --insecure-registry 10.96.0.0/12",
"flags":"procFS auid rootcwd",
"start_time":"2021-10-13T12:40:15.908Z",
"auid":0,
"parent_exec_id":"bWluaWt1YmU6MjEwMDAwMDAwOjE=",
"refcnt":67
},
{
"exec_id":"bWluaWt1YmU6MjEwMDAwMDAwOjE=",
"pid":1,
"uid":0,
"cwd":"/",
"binary":"/usr/lib/systemd/systemd",
"arguments":"noembed norestore",
"flags":"procFS auid rootcwd",
"start_time":"2021-10-13T12:39:34.078Z",
"auid":0,
"refcnt":106
}
]
},
"node_name":"minikube",
"time":"2021-10-13T13:01:08.248Z"
}
第三步，攻击者可以使用 nsenter 命令进入主机命名空间，在主机上以 root 身份运行 bash 命令。
[administrator@JavaLangOutOfMemory ~ ] % nsenter -t 1 -a bash
bash-5.0#
nsenter 命令在指定的命名空间中执行命令。第一个标志 -t 定义了攻击者想要去的目标命名空间。每台 Linux 机器都运行一个 PID 为 1 的进程，该进程始终在主机命名空间中运行。其他命令行参数定义了攻击者还想进入的其他命名空间，在这种情况下，-a 描述了所有命名空间。
因此，攻击者正在以各种可能的方式从容器中突围，并在主机上以 root 身份运行 bash 命令。
安全团队可以通过选取两个 process_exec 事件来识别此突破。在第一个事件中，他们能够观察到第 8 行中执行的 nsenter 命令，并在第 9 行中使用适当的命名空间参数 -t 1 -a。他们还可以在第 15 行中看到源 Pod 名称，即 privileged-the-pod 以及所有 Kubernetes 身份感知和操作系统级可见性信息：
{
"process_exec":{
"process":{
"exec_id":"bWluaWt1YmU6MTYyNzIwMjkzMjkyMToyMzc0Ng==",
"pid":23746,
"uid":0,
"cwd":"/",
"binary":"/usr/bin/nsenter",
"arguments":"-t 1 -a bash",
"flags":"execve rootcwd clone",
"start_time":"2021-10-13T13:06:41.071Z",
"auid":4294967295,
"pod":{
"namespace":"default",
"name":"privileged-the-pod",
"container":{
"id":"docker://32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158",
"name":"privileged-the-pod",
"image":{
"id":"docker-pullable://nginx@sha256:644a70516a26004c97d0d85c7fe1d0c3a67ea8ab7ddf4aff193d9f301670cf36",
"name":"nginx:latest"
},
"start_time":"2021-10-13T12:58:31Z"
}
},
"docker":"32865cff8fef4a9274e9fa1d",
"parent_exec_id":"bWluaWt1YmU6MTI5NDM3OTU0NzQ3ODoxOTU5NA==",
"refcnt":1,
"cap":{
"permitted":[
"CAP_CHOWN",
"DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_LINUX_IMMUTABLE",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_ADMIN",
"CAP_NET_RAW",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_SYS_MODULE",
"CAP_SYS_RAWIO",
"CAP_SYS_CHROOT",
"CAP_SYS_PTRACE",
"CAP_SYS_PACCT",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_NICE",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_MKNOD",
"CAP_LEASE",
"CAP_AUDIT_WRITE",
"CAP_AUDIT_CONTROL",
"CAP_SETFCAP",
"CAP_MAC_OVERRIDE",
"CAP_MAC_ADMIN",
"CAP_SYSLOG",
"CAP_WAKE_ALARM",
"CAP_BLOCK_SUSPEND",
"CAP_AUDIT_READ"
],
"effective":[
"CAP_CHOWN",
"DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_LINUX_IMMUTABLE",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_ADMIN",
"CAP_NET_RAW",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_SYS_MODULE",
"CAP_SYS_RAWIO",
"CAP_SYS_CHROOT",
"CAP_SYS_PTRACE",
"CAP_SYS_PACCT",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_NICE",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_MKNOD",
"CAP_LEASE",
"CAP_AUDIT_WRITE",
"CAP_AUDIT_CONTROL",
"CAP_SETFCAP",
"CAP_MAC_OVERRIDE",
"CAP_MAC_ADMIN",
"CAP_SYSLOG",
"CAP_WAKE_ALARM",
"CAP_BLOCK_SUSPEND",
"CAP_AUDIT_READ"
],
"inheritable":[
"CAP_CHOWN",
"DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_LINUX_IMMUTABLE",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_ADMIN",
"CAP_NET_RAW",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_SYS_MODULE",
"CAP_SYS_RAWIO",
"CAP_SYS_CHROOT",
"CAP_SYS_PTRACE",
"CAP_SYS_PACCT",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_NICE",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_MKNOD",
"CAP_LEASE",
"CAP_AUDIT_WRITE",
"CAP_AUDIT_CONTROL",
"CAP_SETFCAP",
"CAP_MAC_OVERRIDE",
"CAP_MAC_ADMIN",
"CAP_SYSLOG",
"CAP_WAKE_ALARM",
"CAP_BLOCK_SUSPEND",
"CAP_AUDIT_READ"
]
}
},
"parent":{
"exec_id":"bWluaWt1YmU6MTI5NDM3OTU0NzQ3ODoxOTU5NA==",
"pid":19594,
"uid":0,
"cwd":"/",
"binary":"/bin/bash",
"flags":"execve rootcwd clone",
"start_time":"2021-10-13T13:01:08.248Z",
"auid":4294967295,
"pod":{
"namespace":"default",
"name":"privileged-the-pod",
"container":{
"id":"docker://32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158",
"name":"privileged-the-pod",
"image":{
"id":"docker-pullable://nginx@sha256:644a70516a26004c97d0d85c7fe1d0c3a67ea8ab7ddf4aff193d9f301670cf36",
"name":"nginx:latest"
},
"start_time":"2021-10-13T12:58:31Z"
}
},
"docker":"32865cff8fef4a9274e9fa1d",
"parent_exec_id":"bWluaWt1YmU6MTI5NDMzMzczMTY4NToxOTU4NA==",
"refcnt":2
},
"ancestors":[
{
exec_id":"bWluaWt1YmU6MTI5NDMzMzczMTY4NToxOTU4NA==",
"pid":19584,
"uid":0,
"cwd":"/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158/", "binary":"/usr/bin/runc",
"arguments":"--root /var/run/docker/runtime-runc/moby --log /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf48eb8 d28b94e273d6d1670a6f721a9a1158/log.json --log-format json --systemd-cgroup exec --process /tmp/runc-process133903661 --console-socket /tmp/pty028492678/pty.sock --detach --pid-file /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf48eb8 d28b94e273d6d1670a6f721a9a1158/a5579f11fb7d75d66213f488bf44e9c37b92c196dee5e94647e6f60c59cf6693.pid 32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158",
"flags":"execve clone",
"start_time":"2021-10-13T13:01:08.202Z",
"auid":4294967295,
"parent_exec_id":"bWluaWt1YmU6MTEzNzgyOTM1MzU5NzoxNzk1OA==",
"refcnt":2
},
{
"exec_id":"bWluaWt1YmU6MTEzNzgyOTM1MzU5NzoxNzk1OA==",
"pid":17958,
"uid":0,
"cwd":"/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158/", "binary":"/usr/bin/containerd-shim",
"arguments":"-namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf4 eb80d28b94e273d6d1670a6f721a9a1158 -address /var/run/docker/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc -systemd-cgroup",
"flags":"execve clone",
"start_time":"2021-10-13T12:58:31.698Z",
"auid":4294967295,
"parent_exec_id":"bWluaWt1YmU6NDIwODAwMDAwMDA6MjY4MA==",
"refcnt":4
},
{
"exec_id":"bWluaWt1YmU6NDIwODAwMDAwMDA6MjY4MA==",
"pid":2680,
"uid":0,
"cwd":"/",
"binary":"/usr/bin/containerd",
"arguments":"--config /var/run/docker/containerd/containerd.toml --log-level info",
"flags":"procFS auid rootcwd",
"start_time":"2021-10-13T12:40:15.948Z",
"auid":0,
"parent_exec_id":"bWluaWt1YmU6NDIwNDAwMDAwMDA6MjY3Mg==",
"refcnt":86
},
{
"exec_id":"bWluaWt1YmU6NDIwNDAwMDAwMDA6MjY3Mg==",
"pid":2672,
"uid":0,
"cwd":"/",
"binary":"/usr/bin/dockerd",
"arguments":"-H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=virtualbox --insecure-registry 10.96.0.0/12",
"flags":"procFS auid rootcwd",
"start_time":"2021-10-13T12:40:15.908Z",
"auid":0,
"parent_exec_id":"bWluaWt1YmU6MjEwMDAwMDAwOjE=",
"refcnt":88
},
{
"exec_id":"bWluaWt1YmU6MjEwMDAwMDAwOjE=",
"pid":1,
"uid":0,
"cwd":"/",
"binary":"/usr/lib/systemd/systemd",
"arguments":"noembed norestore",
"flags":"procFS auid rootcwd",
"start_time":"2021-10-13T12:39:34.078Z",
"auid":0,
"refcnt":127
}
]
},
"node_name":"minikube",
"time":"2021-10-13T13:06:41.071Z"
}
通过获取第二个 process_exec 事件，安全团队能够检测主机命名空间上的 bash 执行，该主机命名空间将 nsenter 作为父进程二进制文件。父进程信息可以在第 151 行和第 177 行之间看到，源二进制名称可以在第 8 行看到，即 /usr/bin/bash：
{
"process_exec":{
"process":{
"exec_id":"bWluaWt1YmU6MTYyNzIwNjE3NjIzMjoyMzc0Nw==",
"pid":23747,
"uid":0,
"cwd":"/",
"binary":"/usr/bin/bash",
"flags":"execve rootcwd clone",
"start_time":"2021-10-13T13:06:41.074Z",
"auid":4294967295,
"pod":{
"namespace":"default",
"name":"privileged-the-pod",
"container":{
"id":"docker://32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158",
"name":"privileged-the-pod",
"image":{
"id":"docker-pullable://nginx@sha256:644a70516a26004c97d0d85c7fe1d0c3a67ea8ab7ddf4aff193d9f301670cf36",
"name":"nginx:latest"
},
"start_time":"2021-10-13T12:58:31Z"
}
},
"docker":"32865cff8fef4a9274e9fa1d",
"parent_exec_id":"bWluaWt1YmU6MTYyNzIwMjkzMjkyMToyMzc0Ng==",
"refcnt":1,
"cap":{
"permitted":[
"CAP_CHOWN",
"DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_LINUX_IMMUTABLE",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_ADMIN",
"CAP_NET_RAW",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_SYS_MODULE",
"CAP_SYS_RAWIO",
"CAP_SYS_CHROOT",
"CAP_SYS_PTRACE",
"CAP_SYS_PACCT",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_NICE",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_MKNOD",
"CAP_LEASE",
"CAP_AUDIT_WRITE",
"CAP_AUDIT_CONTROL",
"CAP_SETFCAP",
"CAP_MAC_OVERRIDE",
"CAP_MAC_ADMIN",
"CAP_SYSLOG",
"CAP_WAKE_ALARM",
"CAP_BLOCK_SUSPEND",
"CAP_AUDIT_READ"
],
"effective":[
"CAP_CHOWN",
"DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_LINUX_IMMUTABLE",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_ADMIN",
"CAP_NET_RAW",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_SYS_MODULE",
"CAP_SYS_RAWIO",
"CAP_SYS_CHROOT",
"CAP_SYS_PTRACE",
"CAP_SYS_PACCT",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_NICE",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_MKNOD",
"CAP_LEASE",
"CAP_AUDIT_WRITE",
"CAP_AUDIT_CONTROL",
"CAP_SETFCAP",
"CAP_MAC_OVERRIDE",
"CAP_MAC_ADMIN",
"CAP_SYSLOG",
"CAP_WAKE_ALARM",
"CAP_BLOCK_SUSPEND",
"CAP_AUDIT_READ"
],
"inheritable":[
"CAP_CHOWN",
"DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_LINUX_IMMUTABLE",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_ADMIN",
"CAP_NET_RAW",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_SYS_MODULE",
"CAP_SYS_RAWIO",
"CAP_SYS_CHROOT",
"CAP_SYS_PTRACE",
"CAP_SYS_PACCT",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_NICE",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_MKNOD",
"CAP_LEASE",
"CAP_AUDIT_WRITE",
"CAP_AUDIT_CONTROL",
"CAP_SETFCAP",
"CAP_MAC_OVERRIDE",
"CAP_MAC_ADMIN",
"CAP_SYSLOG",
"CAP_WAKE_ALARM",
"CAP_BLOCK_SUSPEND",
"CAP_AUDIT_READ"
]
}
},
"parent":{
"exec_id":"bWluaWt1YmU6MTYyNzIwMjkzMjkyMToyMzc0Ng==",
"pid":23746,
"uid":0,
"cwd":"/",
"binary":"/usr/bin/nsenter",
"arguments":"-t 1 -a bash",
"flags":"execve rootcwd clone",
"start_time":"2021-10-13T13:06:41.071Z",
"auid":4294967295,
"pod":{
"namespace":"default",
"name":"privileged-the-pod",
"container":{
"id":"docker://32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158",
"name":"privileged-the-pod",
"image":{
"id":"docker-pullable://nginx@sha256:644a70516a26004c97d0d85c7fe1d0c3a67ea8ab7ddf4aff193d9f301670cf36",
"name":"nginx:latest"
},
"start_time":"2021-10-13T12:58:31Z"
}
},
"docker":"32865cff8fef4a9274e9fa1d",
"parent_exec_id":"bWluaWt1YmU6MTI5NDM3OTU0NzQ3ODoxOTU5NA==",
"refcnt":2
},
"ancestors":[
{
"exec_id":"bWluaWt1YmU6MTI5NDM3OTU0NzQ3ODoxOTU5NA==",
"pid":19594,
"uid":0,
"cwd":"/",
"binary":"/bin/bash",
"flags":"execve rootcwd clone",
"start_time":"2021-10-13T13:01:08.248Z",
"auid":4294967295,
"pod":{
"namespace":"default",
"name":"privileged-the-pod",
"container":{
"id":"docker://32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158",
"name":"privileged-the-pod",
"image":{
"id":"docker-pullable://nginx@sha256:644a70516a26004c97d0d85c7fe1d0c3a67ea8ab7ddf4aff193d9f301670cf36",
"name":"nginx:latest"
},
"start_time":"2021-10-13T12:58:31Z"
}
},
"docker":"32865cff8fef4a9274e9fa1d",
"parent_exec_id":"bWluaWt1YmU6MTI5NDMzMzczMTY4NToxOTU4NA==",
"refcnt":3
},
{
"exec_id":"bWluaWt1YmU6MTI5NDMzMzczMTY4NToxOTU4NA==",
"pid":19584,
"uid":0,
"cwd":"/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158/", "binary":"/usr/bin/runc",
"arguments":"--root /var/run/docker/runtime-runc/moby --log /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf48eb8 d28b94e273d6d1670a6f721a9a1158/log.json --log-format json --systemd-cgroup exec --process /tmp/runc-process133903661 --console-socket /tmp/pty028492678/pty.sock --detach --pid-file /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf48eb8 d28b94e273d6d1670a6f721a9a1158/a5579f11fb7d75d66213f488bf44e9c37b92c196dee5e94647e6f60c59cf6693.pid 32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158",
"flags":"execve clone",
"start_time":"2021-10-13T13:01:08.202Z",
"auid":4294967295,
"parent_exec_id":"bWluaWt1YmU6MTEzNzgyOTM1MzU5NzoxNzk1OA==",
"refcnt":3
},
{
"exec_id":"bWluaWt1YmU6MTEzNzgyOTM1MzU5NzoxNzk1OA==",
"pid":17958,
"uid":0,
"cwd":"/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf48eb80d28b94e273d6d1670a6f721a9a1158/", "binary":"/usr/bin/containerd-shim",
"arguments":"-namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/32865cff8fef4a9274e9fa1d80bf4 eb80d28b94e273d6d1670a6f721a9a1158 -address /var/run/docker/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc -systemd-cgroup",
"flags":"execve clone",
"start_time":"2021-10-13T12:58:31.698Z",
"auid":4294967295,
"parent_exec_id":"bWluaWt1YmU6NDIwODAwMDAwMDA6MjY4MA==",
"refcnt":5
},
{
"exec_id":"bWluaWt1YmU6NDIwODAwMDAwMDA6MjY4MA==",
"pid":2680,
"uid":0,
"cwd":"/",
"binary":"/usr/bin/containerd",
"arguments":"--config /var/run/docker/containerd/containerd.toml --log-level info",
"flags":"procFS auid rootcwd",
"start_time":"2021-10-13T12:40:15.948Z",
"auid":0,
"parent_exec_id":"bWluaWt1YmU6NDIwNDAwMDAwMDA6MjY3Mg==",
"refcnt":87
},
{
"exec_id":"bWluaWt1YmU6NDIwNDAwMDAwMDA6MjY3Mg==",
"pid":2672,
"uid":0,
"cwd":"/",
"binary":"/usr/bin/dockerd",
"arguments":"-H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=virtualbox --insecure-registry 10.96.0.0/12",
"flags":"procFS auid rootcwd",
"start_time":"2021-10-13T12:40:15.908Z",
"auid":0,
"parent_exec_id":"bWluaWt1YmU6MjEwMDAwMDAwOjE=",
"refcnt":89
},
{
"exec_id":"bWluaWt1YmU6MjEwMDAwMDAwOjE=",
"pid":1,
"uid":0,
"cwd":"/",
"binary":"/usr/lib/systemd/systemd",
"arguments":"noembed norestore",
"flags":"procFS auid rootcwd",
"start_time":"2021-10-13T12:39:34.078Z",
"auid":0,
"refcnt":128
}
]
},
"node_name":"minikube",
"time":"2021-10-13T13:06:41.074Z"
}
现在，攻击者已经到达了 Kubernetes 集群中某个节点上的主机命名空间，并且正在运行 bash。在此示例中，我们使用了具有 hostPID 关联的特权容器。在现实世界中，这也可能是一个拥有自己进程命名空间的非特权容器，然后设法利用内核漏洞获得特权并突破。他们能做什么？攻击者可以看到在同一控制器节点上运行的容器，收集与它们相关的机密，从主机文件系统读取数据，攻击 kubelet 并提升权限；或者利用 Kubernetes 的特殊行为并通过启动一个不可见的容器来保持突破。假设攻击者选择了最后一个选项。
著作权归作者所有
如果觉得我的文章对您有用，请点赞。您的支持将鼓励我继续创作！
赞1Ctrl+Enter 发表
作者其他文章
评论 0 · 赞 0
评论 4 · 赞 1
评论 4 · 赞 2
评论 6 · 赞 1
评论 4 · 赞 3
添加新评论0 条评论