How to Implement Job Objects in Kubernetes

I used to run one-off data migration scripts as bare Pods, back before I understood why that was a bad idea. The problem became obvious the first time a migration pod crashed halfway through — nothing restarted it, nothing tracked whether it actually completed, and I found out it had failed only because someone noticed the migration hadn’t taken effect two days later. That’s the exact failure mode Job objects exist to prevent.

What Is a Job?

A Job is a Kubernetes controller that manages Pods intended to run to completion, rather than run indefinitely like a Deployment. It creates one or more Pods, tracks their success, and retries on failure according to a configurable policy — guaranteeing that a specified number of Pods terminate successfully.

Use Jobs for:

• Database migrations
• Batch data processing
• Report generation
• One-time cleanup scripts
• Any task with a defined “done” state

Don’t use Jobs for long-running services — that’s what Deployments are for.

How Jobs Work Internally

The Job controller watches for Job objects and creates Pods based on the spec. It tracks pod completions using labels it automatically injects (job-name, controller-uid). When a Pod succeeds, the controller counts it toward .spec.completions. When a Pod fails, depending on .spec.backoffLimit, the controller creates a replacement Pod.

Unlike Deployments, Jobs don’t use a rolling update strategy — there’s no “desired steady state” to reconcile against, just a completion target to reach.

Basic Job Example

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
spec:
  template:
    spec:
      containers:
      - name: migration
        image: myregistry.io/db-migrator:1.2.0
        command: ["python", "migrate.py"]
        env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: url
        resources:
          requests:
            cpu: "250m"
            memory: "256Mi"
          limits:
            cpu: "500m"
            memory: "512Mi"
      restartPolicy: Never
  backoffLimit: 4

Key detail: restartPolicy inside a Job’s pod template must be Never or OnFailure — never Always, since that’s incompatible with the concept of “running to completion.”

Apply and watch:

kubectl apply -f db-migration.yaml
kubectl get jobs

NAME            COMPLETIONS   DURATION   AGE
db-migration    1/1           14s        20s

Check pod logs to confirm what actually happened:

kubectl logs job/db-migration

Parallel Jobs

Jobs support running multiple pods in parallel toward a completion count:

apiVersion: batch/v1
kind: Job
metadata:
  name: image-processing
spec:
  completions: 10
  parallelism: 3
  template:
    spec:
      containers:
      - name: processor
        image: myregistry.io/image-processor:2.0.0
        command: ["python", "process.py"]
        resources:
          requests:
            cpu: "500m"
            memory: "1Gi"
      restartPolicy: OnFailure
  backoffLimit: 6

This runs up to 3 Pods concurrently until 10 total completions are reached — useful for embarrassingly parallel workloads like batch image resizing or file conversion.

Indexed Jobs (Work Queue Pattern)

For cases where each Pod needs to process a distinct chunk of work (e.g., “process shard 0 through shard 9”), use completionMode: Indexed:

apiVersion: batch/v1
kind: Job
metadata:
  name: sharded-export
spec:
  completions: 10
  parallelism: 5
  completionMode: Indexed
  template:
    spec:
      containers:
      - name: exporter
        image: myregistry.io/data-exporter:1.0.0
        command: ["/bin/sh", "-c"]
        args:
        - "python export.py --shard=$JOB_COMPLETION_INDEX"
      restartPolicy: Never
  backoffLimit: 3

Each Pod gets a unique JOB_COMPLETION_INDEX environment variable (0 through 9), letting each replica self-identify which chunk of work to process — no external coordination needed.

Controlling Retries: backoffLimit

backoffLimit caps how many times the Job controller retries a failed Pod before marking the Job itself as failed. The backoff between retries increases exponentially (10s, 20s, 40s…) capped at 6 minutes.

spec:
  backoffLimit: 3

Check status after exhausting retries:

kubectl describe job db-migration

Conditions:
  Type     Status  Reason
  ----     ------  ------
  Failed   True    BackoffLimitExceeded

Setting a Deadline: activeDeadlineSeconds

Prevent a runaway or stuck Job from consuming resources indefinitely:

spec:
  activeDeadlineSeconds: 300
  backoffLimit: 3

If the Job’s total active runtime exceeds 300 seconds, it’s terminated and marked as failed with reason DeadlineExceeded, regardless of backoffLimit.

Automatic Cleanup: ttlSecondsAfterFinished

Without cleanup, completed Jobs and their Pods linger in kubectl get jobs forever, cluttering the namespace.

spec:
  ttlSecondsAfterFinished: 3600

The Job (and its Pods) are automatically garbage-collected an hour after completion — extremely useful for CronJob-spawned Jobs that would otherwise accumulate.

CronJobs: Scheduled Job Execution

CronJobs wrap Jobs with a schedule, using standard cron syntax:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-report
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      ttlSecondsAfterFinished: 86400
      template:
        spec:
          containers:
          - name: report-generator
            image: myregistry.io/report-gen:1.1.0
            command: ["python", "generate_report.py"]
          restartPolicy: OnFailure
      backoffLimit: 2

concurrencyPolicy: Forbid prevents a new run from starting if the previous one hasn’t finished — critical for jobs that aren’t safe to run concurrently, like anything touching shared state.

kubectl get cronjobs
kubectl get jobs --selector=job-name

Real-World Use Case: CI/CD Pipeline Integration

Jobs are commonly triggered from CI/CD pipelines for deployment-time tasks like schema migrations, run before rolling out a new application version:

# .gitlab-ci.yml snippet or equivalent
deploy:
  script:
    - kubectl apply -f migration-job.yaml
    - kubectl wait --for=condition=complete --timeout=300s job/db-migration
    - kubectl apply -f deployment.yaml

kubectl wait --for=condition=complete --timeout=300s job/db-migration

This blocks the pipeline until the migration Job genuinely succeeds — or fails fast if it doesn’t — before the actual application rollout proceeds. Much safer than assuming a migration script “probably worked.”

Helm Hook Pattern

Helm supports Jobs as lifecycle hooks, commonly used for pre-install/pre-upgrade migrations:

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ .Release.Name }}-migration
  annotations:
    "helm.sh/hook": pre-upgrade,pre-install
    "helm.sh/hook-weight": "0"
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
  template:
    spec:
      containers:
      - name: migration
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
        command: ["python", "migrate.py"]
      restartPolicy: Never
  backoffLimit: 2

hook-delete-policy cleans up old migration Jobs before new ones are created, avoiding naming collisions on repeated deploys.

Monitoring Jobs with Prometheus

The kube-state-metrics exporter surfaces Job status as Prometheus metrics:

kube_job_status_succeeded{job_name="db-migration"} 1
kube_job_status_failed{job_name="db-migration"} 0
kube_job_failed{job_name="nightly-report"} 1

A typical alerting rule:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: job-failure-alerts
spec:
  groups:
  - name: jobs
    rules:
    - alert: JobFailed
      expr: kube_job_status_failed > 0
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Job {{ $labels.job_name }} has failed"

Troubleshooting

# Full status and events
kubectl describe job <job-name>

# Logs from the most recent pod
kubectl logs job/<job-name>

# List all pods created by a job
kubectl get pods --selector=job-name=<job-name>

# Check why a job's pods are pending
kubectl describe pod <pod-name>

Common failure signatures:

• BackoffLimitExceeded — the container itself is failing; check application logs, not just Kubernetes events.
• Pods stuck Pending — usually resource requests exceeding available capacity, or scheduling constraints (affinity/taints) that no node satisfies.
• Job appears to “hang” with no pods created — check for a bad completions/parallelism combination, or an admission webhook silently rejecting the pod spec.

Suspending and Resuming Jobs

Jobs support a suspend field, useful for holding work without deleting the Job definition — for example, pausing a batch pipeline during a maintenance window without losing its configuration or completion progress tracking:

apiVersion: batch/v1
kind: Job
metadata:
  name: nightly-batch-process
spec:
  suspend: true
  parallelism: 4
  completions: 20
  template:
    spec:
      containers:
      - name: processor
        image: myregistry.io/batch-processor:1.0.0
      restartPolicy: OnFailure

kubectl patch job nightly-batch-process -p '{"spec":{"suspend":false}}'

When suspended, any active Pods are deleted, but the Job’s completion count is preserved — flipping suspend back to false resumes work toward the same completion target rather than starting over.

Job Ownership and Garbage Collection with CronJobs

Understanding how CronJob-spawned Jobs relate to their Pods matters for cleanup and debugging. Each CronJob run creates a distinct Job object (named with a timestamp-based suffix), which in turn owns its Pods via ownerReferences. Deleting a Job cascades to delete its Pods by default:

kubectl get jobs -l job-name --sort-by=.metadata.creationTimestamp
kubectl delete job nightly-report-28234560

successfulJobsHistoryLimit and failedJobsHistoryLimit on the CronJob spec control how many old Job objects are retained automatically — set these deliberately rather than relying on defaults, since a high-frequency CronJob without limits can accumulate hundreds of completed Job objects over time, adding unnecessary load to kubectl get operations and etcd storage.

spec:
  successfulJobsHistoryLimit: 5
  failedJobsHistoryLimit: 10

Keeping more failed history than successful history is a common deliberate choice — failed runs are what you actually want to go back and inspect; successful ones rarely need forensic review.

Common Mistakes

• Using restartPolicy: Always — invalid for Jobs and will be rejected by the API server.
• No activeDeadlineSeconds on Jobs that might hang indefinitely, tying up compute resources.
• No ttlSecondsAfterFinished, leading to thousands of completed Job objects cluttering kubectl get jobs and etcd over time.
• Forgetting concurrencyPolicy on CronJobs whose work isn’t safe to run concurrently — leads to race conditions on shared resources.
• Not checking exit codes properly — the Job controller determines success/failure purely by container exit code, so a script that swallows errors and exits 0 will be marked “succeeded” even if the actual work failed.

Summary

Job objects are the correct primitive for anything that needs to run to completion rather than indefinitely — migrations, batch processing, scheduled reports, and CI/CD pipeline tasks. Combine backoffLimit for retry control, activeDeadlineSeconds for runaway protection, ttlSecondsAfterFinished for automatic cleanup, and completionMode: Indexed for parallel sharded work. Wrapped in a CronJob, they become a reliable, Kubernetes-native replacement for external cron infrastructure.

References

• Kubernetes Documentation: Jobs
• Kubernetes Documentation: CronJob
• Kubernetes API Reference: Job
• Helm Documentation: Chart Hooks
• CNCF Kubernetes Project