Problem
I'd like to issue certs to many different developers (different subjects) all within the dev group, and have them all have access to create and modify things within the dev namespace, but not touch anything outside it, and definitely not see secrets outside it. I suspect the roles, role bindings, etc. I'm creating in step 2 below are not correct, can anyone suggest corrections?
Attempt
- Deployed Kubernetes with API Server flags to support "RBAC,AlwaysAllow" authorization modes, set RBAC super user, and enable RBAC API via
--runtime-config
. - Created a namespace, role, and role binding with the intent that (a) service accounts and system components can effectively still have "AlwaysAllow" access, and (b) any entity in group
dev
can access anything in namespacedev
using this YAML file. NOTE: contents of this link have changed, see YAML files I got working at bottom of question. - Updated Kubernetes to only allow "RBAC" authorization mode.
- Generated client TLS data where the certificate subject flag (for openssl) was
-subj "/[email protected]/O=dev"
. - Generated a kubeconfig file following this template.
Actual Result
I get the following errors when running: kubectl -v 8 --kubeconfig=/tmp/dev-kube-config.yml create -f /tmp/busybox.yml
:
I1219 16:12:37.584657 44323 loader.go:354] Config loaded from file /tmp/dev-kube-config.yml
I1219 16:12:37.585953 44323 round_trippers.go:296] GET https://api.kubernetes.click/api
I1219 16:12:37.585968 44323 round_trippers.go:303] Request Headers:
I1219 16:12:37.585983 44323 round_trippers.go:306] Accept: application/json, */*
I1219 16:12:37.585991 44323 round_trippers.go:306] User-Agent: kubectl/v1.5.1+82450d0 ( darwin/amd64) kubernetes/82450d0
I1219 16:12:38.148994 44323 round_trippers.go:321] Response Status: 403 Forbidden in 562 milliseconds
I1219 16:12:38.149056 44323 round_trippers.go:324] Response Headers:
I1219 16:12:38.149070 44323 round_trippers.go:327] Content-Type: text/plain; charset=utf- 8
I1219 16:12:38.149081 44323 round_trippers.go:327] Content-Length: 17
I1219 16:12:38.149091 44323 round_trippers.go:327] Date: Tue, 20 Dec 2016 00:12:38 GMT
I1219 16:12:38.149190 44323 request.go:904] Response Body: Forbidden: "/api"
I1219 16:12:38.149249 44323 request.go:995] Response Body: "Forbidden: \"/api\""
I1219 16:12:38.149567 44323 request.go:1151] body was not decodable (unable to check for Status): Object 'Kind' is missing in 'Forbidden: "/api"'
...
I1219 16:12:38.820672 44323 round_trippers.go:296] GET https://api.kubernetes. click/swaggerapi/api/v1
I1219 16:12:38.820702 44323 round_trippers.go:303] Request Headers:
I1219 16:12:38.820717 44323 round_trippers.go:306] User-Agent: kubectl/v1.5.1+82450d0 ( darwin/amd64) kubernetes/82450d0
I1219 16:12:38.820731 44323 round_trippers.go:306] Accept: application/json, */*
I1219 16:12:38.902256 44323 round_trippers.go:321] Response Status: 403 Forbidden in 81 milliseconds
I1219 16:12:38.902306 44323 round_trippers.go:324] Response Headers:
I1219 16:12:38.902327 44323 round_trippers.go:327] Content-Type: text/plain; charset=utf- 8
I1219 16:12:38.902345 44323 round_trippers.go:327] Content-Length: 31
I1219 16:12:38.902363 44323 round_trippers.go:327] Date: Tue, 20 Dec 2016 00:12:38 GMT
I1219 16:12:38.902456 44323 request.go:904] Response Body: Forbidden: "/swaggerapi/api/v1"
I1219 16:12:38.902512 44323 request.go:995] Response Body: "Forbidden: \"/swaggerapi/api/v1\""
F1219 16:12:38.903025 44323 helpers.go:116] error: error validating "/tmp/busybox.yml": error validating data: the server does not allow access to the requested resource; if you choose to ignore these errors, turn validation off with --validate=false
Expected Result
Expected to create busybox pod in dev
namespace.
Additional details:
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1+82450d0", GitCommit:"82450d03cb057bab0950214ef122b67c83fb11df", GitTreeState:"not a git tree", BuildDate:"2016-12-14T04:09:31Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.6", GitCommit:"e569a27d02001e343cb68086bc06d47804f62af6", GitTreeState:"clean", BuildDate:"2016-11-12T05:16:27Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
GitHub issue: https://github.com/kubernetes/kubernetes/issues/38997
- Mailing list post: https://groups.google.com/forum/#!topic/kubernetes-dev/6TBTu1AC2L8
EDIT: Working solution based on answer and comments
Based on Jordan's answer below, I upgraded to Kubernetes v1.5.1 and then got the following two YAML files to construct the namespace and all the correct RBAC resources so that everything works as desired:
system-access.yml
(because the out-of-the-box cluster roles and cluster role bindings didn't seem to work):
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1alpha1
metadata:
name: system:node--kubelet
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:node
subjects:
- kind: User
name: kubelet
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1alpha1
metadata:
name: cluster-admin--kube-system:default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: default
namespace: kube-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1alpha1
metadata:
name: system:node-proxier--kube-proxy
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:node-proxier
subjects:
- kind: User
name: kube-proxy
kind: Namespace
apiVersion: v1
metadata:
name: dev
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1alpha1
metadata:
namespace: dev
name: dev-all
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1alpha1
metadata:
name: dev-role-dev-all-members
namespace: dev
subjects:
- kind: Group
name: dev
- kind: Group
name: system:serviceaccounts:dev
roleRef:
kind: Role
name: dev-all
apiGroup: "rbac.authorization.k8s.io"
First, you need to allow access to the URLs kubectl uses for API discovery and validation (swagger, listings of API groups and resource types, etc).
The easiest way to do that is to load the default bootstrap cluster roles and cluster role bindings:
That will create a
system:discovery
ClusterRole and bind all users (authenticated and unauthenticated) to it, allowing them to access swagger and API group information.Second, you shouldn't include the dev service account in the
all
cluster role binding. That would allow that service account (and anyone with access to secrets in the dev namespace containing the dev service account credentials) cluster wide access