How to Set Up Jenkins for Docker Projects

There’s a difference between “building one Docker image in a pipeline” and actually running a whole Docker-based project through Jenkins – multi-container apps, Compose stacks, integration tests that need several services running together, and image lifecycle management across a whole team. This article covers that broader setup: getting Jenkins genuinely fluent in Docker as your project’s primary runtime, not just as a build artifact format. I’ll walk through how I structure this for teams whose entire local dev and CI environment is Docker-based from the ground up.

What “Docker Projects” Really Means Here

A Docker project, in this context, is any codebase where Docker (and often Docker Compose) is the primary way services are built, tested, and run – think a microservices app with an API, a database, a cache, and a frontend, all defined in a docker-compose.yml. Jenkins’ job here isn’t just “build one image,” it’s orchestrating the whole multi-container lifecycle: build all services, spin up the stack, run integration tests against it, tear it down cleanly, and push whichever images changed.

Step 1: Prerequisites

• Jenkins agent(s) with Docker and Docker Compose available (see the dedicated “Jenkins in Docker Containers” article for controller/agent Docker setup options).
• The Docker Pipeline Plugin and Docker Compose Build Step Plugin (or just shell out to docker compose directly, which is often more flexible than plugin-specific syntax).

java -jar jenkins-cli.jar -s http://localhost:8080/ install-plugin docker-workflow -restart

Step 2: Structure the Project’s Compose File for CI

A typical multi-service project:

# docker-compose.yml
version: "3.9"
services:
  api:
    build: ./api
    depends_on:
      - db
      - redis
    environment:
      - DATABASE_URL=postgres://postgres:postgres@db:5432/app
      - REDIS_URL=redis://redis:6379

  frontend:
    build: ./frontend
    depends_on:
      - api

  db:
    image: postgres:16
    environment:
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_DB=app

  redis:
    image: redis:7-alpine

I recommend a separate docker-compose.ci.yml overlay for CI-specific overrides (disabling volumes that assume local host paths, adding healthchecks) rather than reusing the exact dev Compose file unmodified:

# docker-compose.ci.yml
services:
  db:
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 5s
      retries: 10
  api:
    depends_on:
      db:
        condition: service_healthy

Step 3: Write the Jenkinsfile

pipeline {
    agent any

    environment {
        COMPOSE_PROJECT_NAME = "myapp-${env.BUILD_NUMBER}"
        REGISTRY = 'myregistry.com/myorg'
    }

    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }

        stage('Build All Services') {
            steps {
                sh 'docker compose -f docker-compose.yml -f docker-compose.ci.yml build'
            }
        }

        stage('Start Stack') {
            steps {
                sh 'docker compose -f docker-compose.yml -f docker-compose.ci.yml up -d'
                sh 'docker compose ps'
            }
        }

        stage('Run Integration Tests') {
            steps {
                sh '''
                    docker compose exec -T api npm run test:integration
                '''
            }
        }

        stage('Tag and Push Changed Images') {
            when { branch 'main' }
            steps {
                script {
                    def commitSha = env.GIT_COMMIT.take(7)
                    ['api', 'frontend'].each { service ->
                        sh "docker tag ${env.COMPOSE_PROJECT_NAME}-${service}:latest ${REGISTRY}/${service}:${commitSha}"
                        sh "docker push ${REGISTRY}/${service}:${commitSha}"
                    }
                }
            }
        }
    }

    post {
        always {
            sh 'docker compose -f docker-compose.yml -f docker-compose.ci.yml logs > compose-logs.txt || true'
            archiveArtifacts artifacts: 'compose-logs.txt', allowEmptyArchive: true
            sh 'docker compose -f docker-compose.yml -f docker-compose.ci.yml down -v --remove-orphans'
        }
    }
}

The COMPOSE_PROJECT_NAME including BUILD_NUMBER is important – it namespaces containers, networks, and volumes per build, so parallel builds on the same agent don’t collide with each other.

Step 4: Only Build/Push Images That Actually Changed

For monorepo-style Docker projects, rebuilding every service on every commit wastes time. Detect changed paths and build selectively:

stage('Detect Changed Services') {
    steps {
        script {
            def changedFiles = sh(script: "git diff --name-only HEAD~1 HEAD", returnStdout: true).trim()
            env.BUILD_API = changedFiles.contains('api/') ? 'true' : 'false'
            env.BUILD_FRONTEND = changedFiles.contains('frontend/') ? 'true' : 'false'
        }
    }
}

stage('Build API') {
    when { environment name: 'BUILD_API', value: 'true' }
    steps {
        sh 'docker compose build api'
    }
}

Step 5: Running Tests Inside vs Against Containers

There are two patterns for integration testing, and picking the right one matters:

• Run tests inside a service container (docker compose exec -T api npm run test:integration) – simplest, good when the test runner and app share the same environment.
• Run tests from a separate test-runner container that hits the stack over the network – better isolation, and lets you use a different language/toolchain for tests than for the app itself:

stage('Run Integration Tests') {
    steps {
        sh '''
            docker compose -f docker-compose.test.yml run --rm test-runner
        '''
    }
}

# docker-compose.test.yml
services:
  test-runner:
    build: ./tests
    depends_on:
      - api
    environment:
      - API_BASE_URL=http://api:3000

Step 6: Cleaning Up Reliably

Because Compose stacks create networks, volumes, and containers, cleanup has to be unconditional – I always put it in a post { always { ... } } block, never as a regular pipeline step, so a failed test stage doesn’t leave orphaned containers hogging agent resources:

post {
    always {
        sh 'docker compose down -v --remove-orphans || true'
        sh 'docker system prune -f --filter "label=com.docker.compose.project=${COMPOSE_PROJECT_NAME}" || true'
    }
}

Integrating with the Wider Toolchain

• Git/GitHub – trigger on PRs for the full build-test-teardown cycle, and only push images to the registry on merges to main.
• Kubernetes – for teams whose production target is Kubernetes rather than Compose, this pipeline’s job is really just “produce well-tested images”; deployment then hands off to a separate Helm/kubectl stage or a GitOps tool like Argo CD.
• Terraform/Ansible – provision the registry and any supporting infrastructure (networking, secrets stores) ahead of time; keep infra provisioning out of the application pipeline itself.
• Security tools – scan every built image with Trivy or Grype as a required stage before the “Tag and Push” step, exactly as in a single-image pipeline.
• Slack – report which services were rebuilt and pushed on each run, since in a multi-service project “the build passed” is more useful when it says which services changed.

Monitoring and Troubleshooting

• Services fail to become healthy before tests run – rely on Compose healthcheck and depends_on: condition: service_healthy rather than arbitrary sleep calls, which are the single most common source of flaky Docker Compose CI pipelines.
• Port conflicts between parallel builds on the same agent – avoid publishing fixed host ports in the CI overlay file; let Compose assign ephemeral ports, or run each build in an isolated network namespace.
• Disk space exhaustion from accumulated images/volumes – schedule regular docker system prune and consider using ephemeral, disposable agents (Kubernetes pods, as covered in the Jenkins-on-Kubernetes article) so cleanup happens for free when the agent disappears.
• Tests pass locally but fail in CI – check for hardcoded localhost references in test code that should instead reference the Compose service name (e.g., api, not localhost), since containers on a Compose network resolve each other by service name, not by host loopback.

Security Best Practices

• Never bake secrets (database passwords, API keys) into images; inject them via environment variables sourced from Jenkins Credentials at container start.
• Scan every service’s built image, not just the “main” application image – a vulnerable base image in a supporting service is still a supply-chain risk.
• Use --remove-orphans on every teardown to avoid stale containers with outdated code silently lingering and being reused by mistake.
• Restrict which branches are allowed to push to the shared registry namespace, to avoid accidental or malicious pushes from feature branches.

Best Practices

• Keep a CI-specific Compose overlay file separate from your local dev Compose file, since the two environments have genuinely different needs (healthchecks, no bind mounts, no fixed host ports).
• Namespace Compose projects per build (COMPOSE_PROJECT_NAME) to enable safe parallel builds on shared agents.
• Detect and build only changed services in monorepo setups to keep pipeline times reasonable as the project grows.
• Always tear down unconditionally in a post { always { ... } } block – leaked containers are one of the most common causes of “works sometimes, fails randomly” CI behavior.

FAQs

Do I need Kubernetes for a Docker Compose-based project, or is Compose enough for CI? Compose is perfectly fine for CI testing purposes even if production runs on Kubernetes – CI just needs to validate that services work together, not mirror production infrastructure exactly.

How do I handle database migrations in this kind of pipeline? Add a dedicated stage or an init container in the Compose stack that runs migrations before the test stage starts, gated behind the database’s healthcheck.

Can this same pipeline structure work with Podman instead of Docker? Yes, Podman supports Compose-compatible syntax (podman compose or podman-compose), and most of this pipeline structure translates directly.

What’s the best way to speed up rebuilds across pipeline runs? Use Docker layer caching (--cache-from with a previously pushed image, or BuildKit’s inline cache) so unchanged layers don’t rebuild from scratch every run.

Should I run this whole pipeline on every single commit? For fast feedback, run unit tests and lightweight builds on every commit, and reserve the full multi-service integration test stack for PRs and merges to shared branches, to keep short-cycle feedback fast.

Summary

Setting up Jenkins for a genuinely Docker-native project means treating Compose as a first-class part of your pipeline, not an afterthought – build all services, start the stack with proper healthchecks, run integration tests against real inter-service networking, and tear everything down unconditionally. Layer in selective builds for monorepos, per-build namespacing for safe parallelism, and mandatory vulnerability scanning before anything gets pushed, and you end up with a pipeline that actually reflects how your multi-container application behaves in the real world, not just how one isolated image builds.

References

• Docker Compose documentation: https://docs.docker.com/compose/
• Docker Compose CI/CD guidance: https://docs.docker.com/compose/how-tos/production/
• Docker Pipeline Plugin documentation: https://plugins.jenkins.io/docker-workflow/
• Trivy vulnerability scanner: https://aquasecurity.github.io/trivy/
• Jenkins Pipeline Syntax reference: https://www.jenkins.io/doc/book/pipeline/syntax/