preview environments with argocd

reviewing a pull request from a diff is one thing. running the actual change in an isolated environment before merging is something else entirely. preview environments give every pull request its own namespace, its own deployment, and its own url - so changes can be tested against real infrastructure instead of screenshots and speculation.

argocd makes this possible through applicationsets. rather than defining each environment by hand, a single applicationset watches a git repository for open pull requests and generates an argocd application for each one. when the pull request is merged or closed, the application and everything it deployed is removed automatically.

the short version

an applicationset with a pull request generator creates one argocd application per open pull request. each application deploys into its own namespace using values derived from the pr metadata - branch name, pr number, commit sha. when the pr closes, argocd deletes the application and its resources.

how the pull request generator works

the pull request generator polls a git hosting provider for open pull requests on a configured repository. for each open pr, it produces a set of parameters that can be used in the applicationset template.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: preview-environments
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - pullRequest:
        github:
          owner: organisation
          repo: repository
          tokenRef:
            secretName: github-token
            key: token
        requeueAfterSeconds: 60
  template:
    metadata:
      name: "preview-{{ .number }}"
    spec:
      project: previews
      source:
        repoURL: https://github.com/organisation/repository.git
        targetRevision: "{{ .head_sha }}"
        path: deploy/helm
        helm:
          valuesObject:
            image:
              tag: "{{ .head_short_sha }}"
            ingress:
              host: "pr-{{ .number }}.preview.example.com"
      destination:
        server: https://kubernetes.default.svc
        namespace: "preview-{{ .number }}"
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

each open pull request produces an application named preview-<pr-number> that deploys into a namespace with the same name. the helm values override the image tag to the pr’s commit sha and set the ingress hostname to a predictable url.

requeueAfterSeconds controls how often argocd polls the github api for changes. 60 seconds is a reasonable default. lower values increase api calls without meaningful improvement - the bottleneck is usually the image build, not the polling interval.

goTemplate: true enables go templating, which gives access to the full set of pr parameters: {{ .number }}, {{ .branch }}, {{ .head_sha }}, {{ .head_short_sha }}, and {{ .labels }}.

filtering pull requests

not every pull request needs a preview environment. draft prs, documentation changes, and dependabot bumps are usually not worth deploying. labels are the simplest filtering mechanism:

generators:
  - pullRequest:
      github:
        owner: organisation
        repo: repository
        tokenRef:
          secretName: github-token
          key: token
        labels:
          - preview
      requeueAfterSeconds: 60

adding labels: ["preview"] restricts the generator to pull requests with that label. this keeps the cluster from spinning up environments for every branch and gives developers explicit control over when a preview is created.

for more complex filtering, argocd supports multiple label selectors and branch matching with regex filters on branchMatch.

scoping resource permissions

preview environments should be scoped to their own argocd project. this limits what the generated applications can do and where they can deploy:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: previews
  namespace: argocd
spec:
  description: preview environments for pull requests
  sourceRepos:
    - https://github.com/organisation/repository.git
  destinations:
    - namespace: "preview-*"
      server: https://kubernetes.default.svc
  namespaceResourceWhitelist:
    - group: ""
      kind: "*"
    - group: apps
      kind: Deployment
    - group: networking.k8s.io
      kind: Ingress
  orphanedResources:
    warn: true

the destinations field restricts preview applications to namespaces matching preview-*. this prevents a misconfigured template from deploying into production namespaces. namespaceResourceWhitelist limits the resource types that preview applications can create - there’s no reason a preview should be creating clusterroles or persistent volumes.

namespace lifecycle

CreateNamespace=true in the sync options tells argocd to create the namespace if it doesn’t exist. when the pull request closes and argocd deletes the application, the namespace and all its resources are removed with it - provided the application has the correct finalizer.

argocd adds the resources-finalizer.argocd.argoproj.io finalizer to applications by default. this ensures that deleting the application also deletes the resources it created. without it, the application object is removed but the deployments, services, and namespace stay behind.

verify the finalizer is present:

kubectl get application preview-42 -n argocd -o jsonpath='{.metadata.finalizers}'

if the output is empty, resources will be orphaned on deletion.

wiring up the image build

the applicationset handles deployment, but something needs to build and push the container image for each pull request. a ci pipeline triggered on pr events handles this:

# .github/workflows/preview.yaml
name: build preview image
on:
  pull_request:
    types: [opened, synchronize]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - uses: docker/build-push-action@v6
        with:
          push: true
          tags: ghcr.io/organisation/repository:${{ github.event.pull_request.head.sha }}

the image is tagged with the full commit sha of the pr head. this matches the head_sha parameter that the applicationset uses. when a new commit is pushed to the branch, ci builds a new image and argocd picks up the updated sha on its next poll cycle.

dns and ingress

each preview environment needs a routable url. a wildcard dns record pointing *.preview.example.com to the cluster’s ingress controller is the simplest approach. combined with the ingress host from the applicationset template, each pr gets its own endpoint without any dns management per environment.

*.preview.example.com → ingress controller load balancer

shared dependencies

most applications need a database, a cache, or other backing services. preview environments have two options.

dedicated instances per preview. each namespace gets its own database. this gives full isolation but increases resource consumption and startup time. for lightweight databases like sqlite or embedded postgres, this works. for heavier dependencies it’s often not practical.

shared instances with logical isolation. all previews connect to the same database server but use separate databases or schemas named after the pr. the application’s helm values inject the database name:

helm:
  valuesObject:
    database:
      name: "preview_{{.number}}"
    image:
      tag: "{{.head_short_sha}}"

a job or init container creates the database on deploy and drops it on teardown. this balances isolation with resource efficiency, but requires the application to support dynamic database names.

resource limits

preview environments should be constrained. without limits, a handful of preview deployments can consume enough resources to affect production workloads on the same cluster. apply resource quotas to the preview namespace through the helm chart:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: preview-quota
spec:
  hard:
    requests.cpu: "500m"
    requests.memory: 512Mi
    limits.cpu: "1"
    limits.memory: 1Gi
    pods: "10"

this caps each preview at a reasonable ceiling. if previews need more resources than this allows, that’s worth investigating - a preview should be running a minimal replica count with reduced resource allocation, not mirroring production sizing.

a LimitRange in the namespace provides defaults for pods that don’t specify their own limits, preventing a single unconstrained pod from consuming the entire quota.

what to watch out for

stale environments. if argocd can’t reach the github api - token expired, rate limiting, network issue - it stops seeing closed pull requests and the corresponding environments are not cleaned up. monitor the applicationset controller logs for api errors and set up alerts on applicationset reconciliation failures.

github api rate limits. each poll cycle makes api calls proportional to the number of open pull requests. with a low requeueAfterSeconds and many open prs across multiple applicationsets, the token can hit github’s rate limit. use a dedicated github app token instead of a personal access token - app tokens have higher rate limits.

sync waves and ordering. if the helm chart uses sync waves to order resource creation - database migration job before deployment, for example - ensure the preview chart respects the same ordering. a common mistake is adding the migration job to the production chart but not the preview values.

secret management. preview environments need credentials for image pulls, database access, and external services. avoid copying production secrets into preview namespaces. use separate credentials with limited scope, or external secret operators that provision per-namespace secrets from a vault.

namespace deletion hangs. if a namespace contains resources with finalizers that can’t be resolved - a persistent volume claim waiting for a volume to detach, a custom resource whose controller is not running - the namespace deletion blocks indefinitely. kubectl get namespace preview-42 -o json and check status.conditions when this happens.

cleanup automation

as an additional safety net, a cronjob can sweep preview namespaces that have outlived their pull requests. this catches environments that slipped through because of api errors or race conditions during deletion. run it daily or after hours to avoid interfering with active work.

references

[1] argo cd documentation. “applicationset pull request generator.”
argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators-Pull-Request

[2] argo cd documentation. “applicationset go template.”
argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/GoTemplate

[3] argo cd documentation. “projects.”
argo-cd.readthedocs.io/en/stable/user-guide/projects

[4] argo cd documentation. “automated sync policy.”
argo-cd.readthedocs.io/en/stable/user-guide/auto_sync

[5] kubernetes documentation. “resource quotas.”
kubernetes.io/docs/concepts/policy/resource-quotas

[6] github documentation. “rate limits for github apps.”
docs.github.com/en/apps/creating-github-apps/registering-a-github-app/rate-limits-for-github-apps