FLUX CD · SECTION 4 · LECTURE 7

Onboarding tenants and enforcing restrictions: the admin team

Section 4, Lecture 7, from the Flux CD course by DevCloudLab.

Correction: at about 7:00 the video says the patches need the Flux CD Kustomization resource. That is backwards. The patches live in a plain (vanilla) Kustomize file, and the Flux CD Kustomization named dev is the object those patches change. Key Concepts below has the correct explanation.

What you’ll learn

Generate tenant RBAC (namespace, service account, RoleBinding) with flux create tenant
Create Git and Helm repository sources scoped to a tenant’s namespace with flux create source
Wire a tenant’s Flux Kustomization to its own service account and Git repository with flux create kustomization
Patch a tenant’s Kustomization path per environment through a cluster-level tenants.yaml
Verify a deployed tenant application by ingress hostname and HTTP status code

Replace <your-gitlab-username> with your own GitLab username everywhere it appears below. Use your own admin repository (the one you built in the previous lectures) and your own copy of the weather app repository. The finished weather app repository, gitlab.com/abohmeed/myweatherapp, is public, so you can fork it if you did not build your own in the previous lecture.

Lecture Summary

This lecture covers how an admin team uses Flux CD to onboard development teams into a multi-cluster environment while enforcing access restrictions. The key concept is using Kubernetes RBAC combined with Flux CD Kustomization to:

Create isolated namespaces and service accounts for each tenant (development team)
Grant limited permissions to each tenant’s service account
Deploy tenant applications from external Git repositories using Flux CD
Separate concerns between infrastructure (admin team) and applications (development teams)

Key Concepts

Multi-Tenancy in Flux CD

Multi-tenancy allows multiple teams to share Kubernetes clusters while remaining isolated from each other. Each team gets:

A dedicated namespace
A service account with limited permissions
Access to deploy their own applications through Flux CD

Role-Based Access Control (RBAC)

RBAC in Kubernetes controls who can do what. In this scenario:

The dev team’s service account has admin permissions only in the “apps” namespace
They cannot access other namespaces or cluster-wide resources
The admin team maintains control of the infrastructure

Flux CD Kustomization Resource

The Kustomization resource from kustomize.toolkit.fluxcd.io/v1 tells Flux CD:

Which Git repository to watch for changes
Which path in that repository to apply (the per-environment patches that change this path live in plain Kustomize files, not in the Flux resource)
How often to reconcile (sync) changes

This is different from vanilla Kustomize (kustomize.config.k8s.io/v1beta1), which is the tool that shapes and patches manifests. In this lecture the patches live in plain Kustomize files, and the Flux Kustomization named dev is the object those patches change.

Step-by-Step Walkthrough

1. Create Directory Structure

mkdir -p ./tenants/base/dev
mkdir -p ./tenants/staging
mkdir -p ./tenants/production

This creates:

tenants/base/dev/: Base manifests shared across environments
tenants/staging/: Staging cluster specific configuration
tenants/production/: Production cluster specific configuration

2. Generate Tenant RBAC

flux create tenant dev --with-namespace=apps --export > ./tenants/base/dev/rbac.yaml

This command:

Tells Flux CD to create a tenant named “dev”
Constrains it to the “apps” namespace. The flag is --with-namespace, not --namespace (plain --namespace is the CLI’s own request-scope flag, and using it here fails with ✗ with-namespace is required)
Exports the YAML instead of applying it to the cluster
Redirects output to a file

Generated rbac.yaml contents (Flux v2.9.4):

---
apiVersion: v1
kind: Namespace
metadata:
  labels:
    toolkit.fluxcd.io/tenant: dev
  name: apps
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    toolkit.fluxcd.io/tenant: dev
  name: dev
  namespace: apps
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    toolkit.fluxcd.io/tenant: dev
  name: dev-reconciler
  namespace: apps
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: gotk:apps:reconciler
- kind: ServiceAccount
  name: dev
  namespace: apps

Three objects, and no ClusterRole of their own: Flux references the cluster-admin role that Kubernetes already ships. The second subject, gotk:apps:reconciler, is a user subject that flux create tenant adds. The identity Flux actually acts as when it reconciles on behalf of this tenant is the dev service account, because the tenant’s Kustomization sets serviceAccountName: dev.

Key point: cluster-admin is a cluster-wide role, but it is granted here by a RoleBinding inside the apps namespace, not a ClusterRoleBinding. That one choice is what confines the dev team’s permissions to their own namespace.

3. Create Git Repository Source

flux create source git dev \
  --namespace=apps \
  --url=https://gitlab.com/<your-gitlab-username>/myweatherapp \
  --branch=main \
  --secret-ref=gitlab-auth \
  --export > ./tenants/base/dev/sync.yaml

This:

Creates a GitRepository resource named “dev”
Points to the dev team’s weather app repository
Uses the “gitlab-auth” secret for credentials
Places it in the “apps” namespace

4. Create Helm Repository Source

flux create source helm gitlab \
  --namespace=apps \
  --url=oci://registry.gitlab.com/<your-gitlab-username>/myweatherapp \
  --secret-ref=gitlab-auth \
  --export >> ./tenants/base/dev/sync.yaml

This appends (using >>) a HelmRepository resource that:

Points to the GitLab OCI (Open Container Initiative) registry
Uses the same credentials secret
Allows Helm charts to be pulled from GitLab’s registry

5. Create Kustomization Resource

flux create kustomization dev \
  --namespace=apps \
  --service-account=dev \
  --source=GitRepository/dev \
  --path="./" \
  --export >> ./tenants/base/dev/sync.yaml

This appends a Kustomization resource that:

Links the “dev” Git repository as the source
Uses the “dev” service account to deploy resources
Watches the root path ./ by default (to be overridden per environment)

Generated sync.yaml (all three resources combined):

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: dev
  namespace: apps
spec:
  interval: 1m0s
  ref:
    branch: main
  secretRef:
    name: gitlab-auth
  url: https://gitlab.com/<your-gitlab-username>/myweatherapp
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: gitlab
  namespace: apps
spec:
  interval: 1m0s
  secretRef:
    name: gitlab-auth
  type: oci
  url: oci://registry.gitlab.com/<your-gitlab-username>/myweatherapp
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: dev
  namespace: apps
spec:
  interval: 1m0s
  path: ./
  prune: false
  serviceAccountName: dev
  sourceRef:
    kind: GitRepository
    name: dev

prune is false here because --prune was not passed to flux create kustomization. The cluster-level tenants Kustomization further down does set prune: true, which is the one that garbage-collects tenants removed from Git.

6. Create Base Kustomization File

cd ./tenants/base/dev
kustomize create --autodetect
cd -

This generates kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- rbac.yaml
- sync.yaml

This vanilla Kustomize file simply lists the manifests that belong together.

7. Create Staging Cluster Patch

cat > ./tenants/staging/dev-patch.yaml << 'EOF'
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: dev
  namespace: apps
spec:
  path: ./kustomize/staging
EOF

This creates a JSON Merge Patch that overrides the path field in the base Kustomization resource. It tells Flux CD to read the dev team’s staging manifests from ./kustomize/staging instead of the root.

8. Create Staging Kustomization File

cat > ./tenants/staging/kustomization.yaml << 'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../base/dev
patches:
  - path: dev-patch.yaml
EOF

This vanilla Kustomize file:

References the base manifests in ../base/dev
Applies the patch dev-patch.yaml to override the path

The patch lives here, in the plain Kustomize file, because vanilla Kustomize is the tool that changes manifests. The Flux Kustomization dev does not carry the patch; it is the object being patched. Flux then delivers the result to the cluster through GitOps.

9. Create Staging Cluster Configuration

cat > ./clusters/staging/tenants.yaml << 'EOF'
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: tenants
  namespace: flux-system
spec:
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./tenants/staging
  prune: true
EOF

This Flux CD Kustomization resource:

Lives in the flux-system namespace (admin space)
Watches the tenants/staging directory in the admin repo
Reconciles every 5 minutes
Automatically prunes resources that are removed from Git

10. Commit, Push and Let Flux Create the Namespace

The lecture commits as it goes (after rbac.yaml, after the base files, and now after the staging files). Make sure everything is pushed, then have Flux apply it:

git add ./tenants/ ./clusters/staging/
git commit -m "Configure staging cluster tenant deployment"
git push

flux reconcile kustomization flux-system --with-source -n flux-system
kubectl get ns apps

kubectl get ns apps must list the namespace before you continue. The Secret in the next step lives in apps, and until Flux has created that namespace the command fails with namespaces "apps" not found. That is why the Secret is created last.

11. Create Authentication Secret

flux create secret git gitlab-auth \
  --url=https://gitlab.com/<your-gitlab-username>/myweatherapp.git \
  --username=<your-gitlab-username> \
  --password=<your-personal-access-token> \
  --namespace=apps

This command:

Creates a Kubernetes Secret in the “apps” namespace
Stores GitLab credentials for authentication
The secret is not stored in Git (for security)
Flux CD uses this secret to clone the dev team’s repository and pull Helm charts

The weather service needs a second secret in the same namespace: its RapidAPI key.

kubectl -n apps create secret generic api-key \
  --from-literal=values.yaml="apikey: <your-rapidapi-key>"

Use your own key from rapidapi.com: subscribe to the WeatherAPI.com API and paste the key in place of <your-rapidapi-key>. The dev team’s repository commits only a placeholder; the weatherapp-weather HelmRelease reads this secret through valuesFrom, so the real key is created against the cluster and never stored in Git (the same rule as gitlab-auth).

12. Reconcile and Verify Staging

flux reconcile kustomization tenants -n flux-system
flux reconcile source git dev -n apps
flux reconcile kustomization dev -n apps
kubectl get helmrelease -n apps

After a few seconds you should see three Helm releases, one per weather app chart.

13. Repeat for Production

Most of the work is already done. Switch to the production cluster first:

kubectl config use-context kind-production

Copy the staging patch and kustomization file, then change the patch’s path to ./kustomize/production. The kustomization file needs no changes:

cp ./tenants/staging/dev-patch.yaml ./tenants/production/dev-patch.yaml
cp ./tenants/staging/kustomization.yaml ./tenants/production/kustomization.yaml

Production patch (tenants/production/dev-patch.yaml):

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: dev
  namespace: apps
spec:
  path: ./kustomize/production

Production cluster configuration (clusters/production/tenants.yaml):

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: tenants
  namespace: flux-system
spec:
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./tenants/production
  prune: true

Commit, push, and let Flux create the apps namespace on production:

git add ./tenants/production/ ./clusters/production/
git commit -m "Configure production cluster tenant deployment"
git push

flux reconcile kustomization flux-system --with-source -n flux-system
kubectl get ns apps

Secrets are not in Git, so the production cluster does not have them yet. Once apps exists there, create both Secrets again, exactly as on staging:

flux create secret git gitlab-auth \
  --url=https://gitlab.com/<your-gitlab-username>/myweatherapp.git \
  --username=<your-gitlab-username> \
  --password=<your-personal-access-token> \
  --namespace=apps

kubectl -n apps create secret generic api-key \
  --from-literal=values.yaml="apikey: <your-rapidapi-key>"

Then reconcile and check the releases:

flux reconcile kustomization tenants -n flux-system
flux reconcile source git dev -n apps
flux reconcile kustomization dev -n apps
kubectl get helmrelease -n apps

Complete File Structure

After all steps, your repository should have:

.
├── clusters/
│   ├── staging/
│   │   └── tenants.yaml
│   └── production/
│       └── tenants.yaml
└── tenants/
    ├── base/
    │   └── dev/
    │       ├── rbac.yaml
    │       ├── sync.yaml
    │       └── kustomization.yaml
    ├── staging/
    │   ├── dev-patch.yaml
    │   └── kustomization.yaml
    └── production/
        ├── dev-patch.yaml
        └── kustomization.yaml

Testing the Deployment

Verify GitRepository and HelmRepository

# List all sources in the apps namespace
flux get all -n apps

# Check detailed status of the git repository
flux get source git -n apps

# Check detailed status of the helm repository
flux get source helm -n apps

Verify Helm Releases

# List all Helm releases created by the dev team
kubectl get helmrelease -n apps

# Get detailed information
kubectl describe helmrelease -n apps

Access the Application

The KinD clusters already publish their ingress controllers on the host: staging on port 8080, production on port 8081. Do not run kubectl port-forward onto those ports: they are already bound, and the command fails with address already in use.

The dev team’s ingress rules match on a hostname, so a request to plain localhost matches no rule and nginx answers with its default 404. Check what the rule matches on:

kubectl config use-context kind-staging
kubectl get ingress -n apps

You should see host weatherapp.staging (and weatherapp.production on the other cluster). Add one line to the /etc/hosts file of the machine running your browser, pointing both names at the machine running the clusters (127.0.0.1 if that is the same machine, otherwise its IP address):

sudo sh -c 'echo "127.0.0.1 weatherapp.staging weatherapp.production" >> /etc/hosts'

Then open:

staging: http://weatherapp.staging:8080
production: http://weatherapp.production:8081

To prove the route without a browser (302 is the app redirecting to /login, 404 means the Host header matched no ingress rule):

curl -s -o /dev/null -w '%{http_code}\n' -H 'Host: weatherapp.staging' http://localhost:8080/

Important Differences: Flux CD vs Vanilla Kustomize

Aspect	Vanilla Kustomize	Flux CD Kustomization
API Version	`kustomize.config.k8s.io/v1beta1`	`kustomize.toolkit.fluxcd.io/v1`
Purpose	Template and patch manifests	GitOps automation and deployment
When to Use	Organizing and structuring files	Continuous reconciliation with Git
Reconciliation	Manual (`kustomize build`)	Automatic (every interval)
Prune	Not applicable	Can automatically delete resources

In this lecture:

Vanilla Kustomize files organize the base and environment-specific patches
Flux CD Kustomization resources automate the deployment and continuous sync