Pod安全策略是一种机制，用于限制容器在k8s上运行时可执行的操作，例如防止其作为特权容器的运行，和主机网络运行等。

本文内容较多，所以在此做了一个完整的简介说明：

在master节点运行kubeadm init并启用PodSecurityPolicy admission controller；
使用RBAC配置添加一些pod安全策略 - 足够启动CNI和DNS等；
没有它，CNI daemonsets将无法启动
应用您的CNI插件，该插件可以使用以前创建的一个pod安全策略
通过kubeadm join完成向集群添加节点的配置
在向集群添加更多工作负载时，请检查是否需要额外的pod安全策略和RBAC配置

我们将要做什么

以下是在kubeadm上安装运行pod安全策略所采取的步骤：

为master节点的初始化配置pod安全策略 admission controller
为网络层组件配置一些pod安全策略
配置CNI插件 - 这里将使用flannel
然后将使用下面的代码演示一些其他的pod安全策略方案
安装一个具有一些特定要求的nginx-ingress controller - 这只是为了说明添加额外的策略
安装没有特定pod安全策略要求的常规服务 - 基于httpbin.org

环境

将只在单个节点说明，而不是具有高可用集群的多节点集群
Ubuntu：根据kubeadm的要求切换，为UTC配置时区
Docker：假设当前用户在docker组中
kubernetes 1.11.3 - 使用RBAC

master节点配置

按照说明安装kubeadm

安装jq，用它来进行一些json输出处理

sudo apt-get update  
sudo apt-get install -y jq

验证kubeadm的版本

sudo kubeadm version

接着我们将为下面创建的内容创建一个目录，在以下所有说明中我们假设您在此目录中

mkdir ~/psp-inv  
cd ~/psp-inv

kubeadm配置文件

将创建这个文件，并将其用于master上的kubeadm init，使用此内容创建一个kubeadm-config.yaml文件 - 请注意我们必须为flannel指定pod子网为10.244.0.0/16

请注意，此文件对于此演示是最小的，如果您使用更高版本的kubeadm，则可能需要更改apiVersion

apiVersion: kubeadm.k8s.io/v1alpha2  
kind: MasterConfiguration  
apiServerExtraArgs:  
 enable-admission-plugins: PodSecurityPolicy  
controllerManagerExtraArgs:  
address: 0.0.0.0  
kubernetesVersion: v1.11.3  
networking:  
podSubnet: 10.244.0.0/16  
schedulerExtraArgs:  
address: 0.0.0.0

Master init

根据上面命令输出的说明获取您自己的kubeconfig文件副本，如果要将工作节点添加到集群，请注意加入以下信息，并检查master节点的状态

NAME STATUS ROLES AGE VERSION
pmcgrath-k8s-master NotReady master 1m v1.11.3

这说明节点还没有准备好，因为它正在等待CNI，检查一下pods

No resources found.

所以，如果我们没有启用pod security policy admission control，通常看不到任何正在运行的pod，检查 docker

docker container ls --format '{{ .Names }}'
k8s\_kube-scheduler\_kube-scheduler-pmcgrath-k8s-master\_kube-system\_a00c35e56ebd0bdfcd77d53674a5d2a1\_0
k8s\_kube-controller-manager\_kube-controller-manager-pmcgrath-k8s-master\_kube-system\_fd832ada507cef85e01885d1e1980c37\_0  
k8s\_etcd\_etcd-pmcgrath-k8s-master\_kube-system\_16a8af6b4a79e9b0f81092f85eab37cf\_0  
k8s\_kube-apiserver\_kube-apiserver-pmcgrath-k8s-master\_kube-system\_db201a8ecaf8e99623b425502a6ba627\_0  
k8s\_POD\_kube-controller-manager-pmcgrath-k8s-master\_kube-system\_fd832ada507cef85e01885d1e1980c37\_0  
k8s\_POD\_kube-scheduler-pmcgrath-k8s-master\_kube-system\_a00c35e56ebd0bdfcd77d53674a5d2a1\_0  
k8s\_POD\_kube-apiserver-pmcgrath-k8s-master\_kube-system\_db201a8ecaf8e99623b425502a6ba627\_0  
k8s\_POD\_etcd-pmcgrath-k8s-master\_kube-system\_16a8af6b4a79e9b0f81092f85eab37cf\_0

所有容器都在运行，但这不是通过kubectl获取的信息,还需检查事件

会看到类似这样的错误：pods “kube-proxy-“ is forbidden: no providers available to validate pod request

配置pod安全策略

当前已经配置了：

任何工作负载都可以使用默认的pod安全策略，它没有什么特权，但应该适用于大多数工作负载
将创建一个RBAC ClusterRole
将为所有经过身份验证的用户创建RBAC ClusterRoleBinding
我授予kube-system命名空间中的节点和所有服务帐户访问权限的特权pod安全策略
考虑这个权限应该被限制在这个命名空间内
应该只在此命名空间中运行k8s组件
将创建一个RBAC ClusterRole
将在kube-system命名空间中创建RBAC RoleBinding

使用此内容创建default-psp-with-rbac.yaml文件

apiVersion: policy/v1beta1  
kind: PodSecurityPolicy  
metadata:  
 annotations:  
 apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default'  
 apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default'  
 seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default'  
 seccomp.security.alpha.kubernetes.io/defaultProfileName: 'docker/default'  
 name: default  
spec: allowedCapabilities: \[\] # default set of capabilities are implicitly allowed  
 allowPrivilegeEscalation: false  
 fsGroup:  
 rule: 'MustRunAs'  
 ranges:  
 # Forbid adding the root group.  
\- min: 1  
 max: 65535  
 hostIPC: false  
 hostNetwork: false  
 hostPID: false  
 privileged: false  
 readOnlyRootFilesystem: false  
 runAsUser:  
 rule: 'MustRunAsNonRoot'  
 seLinux:  
 rule: 'RunAsNonRoot'  
 supplementalGroups:  
 rule: 'RunAsNonRoot'  
 ranges:  
 # Forbid adding the root group.  
\- min: 1  
 max: 65535  
 volumes:  
\- 'configMap'  
\- 'downwardAPI'  
\- 'emptyDir'  
\- 'persistentVolumeClaim'  
\- 'projected'  
\- 'secret'  
 hostNetwork: false  
 runAsUser:  
 rule: 'RunAsAny'  
 seLinux:  
 rule: 'RunAsAny'  
 supplementalGroups:  
 rule: 'RunAsAny'  
 fsGroup:  
 rule: 'RunAsAny'
\---
\# Cluster role which grants access to the default pod security policy 
apiVersion: rbac.authorization.k8s.io/v1  
kind: ClusterRole  
metadata:  
 name: default-psp  
rules:  
\- apiGroups:  
 - policy  
 resourceNames:  
 - default  
 resources:  
 - podsecuritypolicies  
 verbs:  
 - use\---
\# Cluster role binding for default pod security policy granting all authenticated users access
apiVersion: rbac.authorization.k8s.io/v1  
kind: ClusterRoleBinding  
metadata:  
 name: default-psp  
roleRef:  
 apiGroup: rbac.authorization.k8s.io  
 kind: ClusterRole  
 name: default-psp  
subjects:  
\- apiGroup: rbac.authorization.k8s.io  
 kind: Group  
 name: system:authenticated

使用此内容创建privileged-psp-with-rbac.yaml文件

# Should grant access to very few pods, i.e. kube-system system pods and possibly CNI pods  
apiVersion: policy/v1beta1  
kind: PodSecurityPolicy  
metadata:  
 annotations:  
 # See https://kubernetes.io/docs/concepts/policy/pod-security-policy/#seccomp  
 seccomp.security.alpha.kubernetes.io/allowedProfileNames: '\*'  
 name: privileged  
spec:  
 allowedCapabilities:  
\- '\*'  
 allowPrivilegeEscalation: true  
 fsGroup:  
 rule: 'RunAsAny'  
 hostIPC: true  
 hostNetwork: true  
 hostPID: true  
 hostPorts:  
\- min: 0  
 max: 65535  
 privileged: true  
 readOnlyRootFilesystem: false  
 runAsUser:  
 rule: 'RunAsAny'  
 seLinux:  
 rule: 'RunAsAny'  
 supplementalGroups:  
 rule: 'RunAsAny'  
 volumes:  
\- '\*'
\---
\# Cluster role which grants access to the privileged pod security policy 
apiVersion: rbac.authorization.k8s.io/v1  
kind: ClusterRole  
metadata:  
 name: privileged-psp  
rules:  
\- apiGroups:  
 - policy  
 resourceNames:  
 - privileged  
 resources:  
 - podsecuritypolicies  
 verbs:  
 - use\---
\# Role binding for kube-system - allow nodes and kube-system service accounts - should take care of CNI i.e. flannel running in the kube-system namespace 
\# Assumes access to the kube-system is restricted  
apiVersion: rbac.authorization.k8s.io/v1  
kind: RoleBinding  
metadata:  
 name: kube-system-psp  
 namespace: kube-system  
roleRef:  
 apiGroup: rbac.authorization.k8s.io  
 kind: ClusterRole  
 name: privileged-psp  
subjects:  
\# For the kubeadm kube-system nodes  
\- apiGroup: rbac.authorization.k8s.io  
 kind: Group  
 name: system:nodes  
\# For all service accounts in the kube-system namespace  
\- apiGroup: rbac.authorization.k8s.io  
 kind: Group  
 name: system:serviceaccounts:kube-system

使用RBAC配置应用上述pod安全策略

kubectl apply -f default-psp-with-rbac.yaml  
kubectl apply -f privileged-psp-with-rbac.yaml

校验

一段时间后，网络层pods将处于运行状态，coredns pods将挂起-等待CNI

kubectl get pods --all-namespaces --output wide --watch

在配置CNI之前,网络层pods将再次启动失败，因为节点仍未准备就绪

安装flannel**

只有当前存在有特权 pod安全策略才能完成此操作，并且kube-system 中的flannel服务帐户将能够使用，如果使用不同的CNI插件，则应使用其安装说明，可能需要修改用于kubeadm init的kubeadm-config.yaml文件中的podSubnet。

延伸链接

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

再次校验

kubectl get pods --all-namespaces --output wide --watch

所有pods最终将进入运行状态，包括coredns pod(s)

kubectl get nodes

节点现已准备就绪。

允许master上的工作负载

如果想要启动工作节点，可以正常使用kubeadm join命令，利用kubeadm init输出，在这里跳过。

这些工作节点上没有特别需要加入集群pod安全策略，允许master节点上的工作负载，因为我们只是尝试在单个节点集群上进行验证。

kubectl taint nodes --all node-role.kubernetes.io/master-

nginx ingress

使用下方链接的清单

创建一个新的命名空间和单个实例ingress controller，足以说明额外的pod安全策略。

由于命名空间尚不存在，因此我们可以创建，以便引用服务帐户

kubectl create namespace ingress-nginx

创建一个pod安全策略

本pod安全策略基于deployment清单，使用此内容创建文件nginx-ingress-psp-with-rbac.yaml

apiVersion: policy/v1beta1  
kind: PodSecurityPolicy  
metadata:  
 annotations:  
 # Assumes apparmor available  
 apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default'  
 apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default'  
 seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default'  
 seccomp.security.alpha.kubernetes.io/defaultProfileName: 'docker/default'  
 name: ingress-nginx  
spec:  
 # See nginx-ingress-controller deployment at https://github.com/kubernetes/ingress-nginx/blob/master/deploy/mandatory.yaml  
 # See also https://github.com/kubernetes-incubator/kubespray/blob/master/roles/kubernetes-apps/ingress\_controller/ingress\_nginx/templates/psp-ingress-nginx.yml.j2  
 allowedCapabilities:  
\- NET\_BIND\_SERVICE  
 allowPrivilegeEscalation: true  
 fsGroup:  
 rule: 'MustRunAs'  
 ranges:  
\- min: 1  
 max: 65535  
 hostIPC: false  
 hostNetwork: false  
 hostPID: false  
 hostPorts:  
\- min: 80  
 max: 65535  
 privileged: false  
 readOnlyRootFilesystem: false  
 runAsUser:  
 rule: 'MustRunAsNonRoot'  
 ranges:  
\- min: 33  
 max: 65535  
 seLinux:  
 rule: 'RunAsAny'  
 supplementalGroups:  
 rule: 'MustRunAs'  
 ranges:  
 # Forbid adding the root group.  
\- min: 1  
 max: 65535  
 volumes:  
\- 'configMap'  
\- 'downwardAPI'  
\- 'emptyDir'  
\- 'projected'  
\- 'secret'

\---

apiVersion: rbac.authorization.k8s.io/v1

  
kind: Role  
metadata:  
 name: ingress-nginx-psp  
 namespace: ingress-nginx  
rules:  
\- apiGroups:  
 - policy  
 resourceNames:  
 - ingress-nginx  
 resources:  
 - podsecuritypolicies  
 verbs:  
 - use\---

apiVersion: rbac.authorization.k8s.io/v1

  
kind: RoleBinding  
metadata:  
 name: ingress-nginx-psp  
 namespace: ingress-nginx  
roleRef:  
 apiGroup: rbac.authorization.k8s.io  
 kind: Role  
 name: ingress-nginx-psp  
subjects:  
\# Lets cover default and nginx-ingress-serviceaccount service accounts  
\# Could have altered default-http-backend deployment to use the same service acccount to avoid granting the default service account access  
\- kind: ServiceAccount  
 name: default  
\- kind: ServiceAccount  
 name: nginx-ingress-serviceaccount

即创建

kubectl apply -f nginx-ingress-psp-with-rbac.yaml

创建nginx-ingress工作负载

删除controller-publish-service arg，因为在这里不需要

curl -s https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml | sed '/--publish-service/d' | kubectl apply -f -

检查pods

kubectl get pods --namespace ingress-nginx --watch

现在可以看到pod安全策略附加了注释

kubectl get pods --namespace ingress-nginx --selector app.kubernetes.io/name=ingress-nginx -o json | jq -r '.items\[0\].metadata.annotations."kubernetes.io/psp"'

Httpbin.org工作负载

部署一个工作负载，默认的pod安全策略就足够了，使用此内容创建一个httpbin.yaml文件

apiVersion: apps/v1  
kind: Deployment  
metadata:  
 labels:  
 app.kubernetes.io/name: httpbin  
 name: httpbin  
spec:  
 replicas: 1  
 selector:  
 matchLabels:  
 app.kubernetes.io/name: httpbin  
 template:  
 metadata:  
 labels:  
 app.kubernetes.io/name: httpbin  
 spec:  
 containers:  
\- args: \["-b", "0.0.0.0:8080", "httpbin:app"\]  
 command: \["gunicorn"\]  
 image: docker.io/kennethreitz/httpbin:latest  
 imagePullPolicy: Always  
 name: httpbin  
 ports:  
\- containerPort: 8080  
 name: http  
 restartPolicy: Always
\---

apiVersion: extensions/v1beta1

  
kind: Ingress  
metadata:  
 annotations:  
 kubernetes.io/ingress.class: "nginx"  
 labels:  
 app.kubernetes.io/name: httpbin  
 name: httpbin  
spec:  
 rules:  
 - host: my.httpbin.com  
 http:  
 paths:  
 - path:  
 backend:  
 serviceName: httpbin  
 servicePort: 8080\---

apiVersion: v1

  
kind: Service  
metadata:  
 labels:  
 app.kubernetes.io/name: httpbin  
 name: httpbin  
spec:  
 ports:  
 - name: http  
 port: 8080  
 selector:  
 app.kubernetes.io/name: httpbin

创建命名空间并在工作负载里运行

kubectl create namespace demo  
kubectl apply --namespace demo -f httpbin.yaml

检查pod是否存在以及是否使用了默认策略

kubectl get pods --namespace demo  kubectl get pods --namespace demo --selector app.kubernetes.io/name=httpbin -o json | jq -r '.items\[0\].metadata.annotations."kubernetes.io/psp"'

测试工作负载

通过调用 ingress controller pod实例

\# Get nginx ingress controller pod IP  
nginx\_ip=$(kubectl get pods --namespace ingress-nginx --selector app.kubernetes.io/name=ingress-nginx --output json | jq -r .items\[0\].status.podIP)
\# Test ingress and out httpbin workload
curl -H 'Host: my.httpbin.com' http://$nginx\_ip/get

如果打乱它，可以重置并重启

\# Note: Will loose PKI also which is fine here as kubeadm master init will re-create  
sudo kubeadm reset
\# Should flush iptable rules after a kubeadm reset, see https://blog.heptio.com/properly-resetting-your-kubeadm-bootstrapped-cluster-nodes-heptioprotip-473bd0b824aa 
sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X

参考文档

安全建议

GCE政策

添加小助手微信，加入【容器魔方】技术社群。

创作场景

在 Kubeadm 中使用 pod 安全策略