Offensive Security Researcher & Quantum Cryptography Analyst

How to Set Up Jenkins for Docker Projects

Abdul Wahab Junaid
How to Set Up Jenkins for Docker Projects

There’s a difference between “building one Docker image in a pipeline” and actually running a whole Docker-based project through Jenkins – multi-container apps, Compose stacks, integration tests that need several services running together, and image lifecycle management across a whole team. This article covers that broader setup: getting Jenkins genuinely fluent in Docker as your project’s primary runtime, not just as a build artifact format. I’ll walk through how I structure this for teams whose entire local dev and CI environment is Docker-based from the ground up.

What “Docker Projects” Really Means Here

A Docker project, in this context, is any codebase where Docker (and often Docker Compose) is the primary way services are built, tested, and run – think a microservices app with an API, a database, a cache, and a frontend, all defined in a docker-compose.yml. Jenkins’ job here isn’t just “build one image,” it’s orchestrating the whole multi-container lifecycle: build all services, spin up the stack, run integration tests against it, tear it down cleanly, and push whichever images changed.

Step 1: Prerequisites

java -jar jenkins-cli.jar -s http://localhost:8080/ install-plugin docker-workflow -restart

Step 2: Structure the Project’s Compose File for CI

A typical multi-service project:

# docker-compose.yml
version: "3.9"
services:
  api:
    build: ./api
    depends_on:
      - db
      - redis
    environment:
      - DATABASE_URL=postgres://postgres:postgres@db:5432/app
      - REDIS_URL=redis://redis:6379

  frontend:
    build: ./frontend
    depends_on:
      - api

  db:
    image: postgres:16
    environment:
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_DB=app

  redis:
    image: redis:7-alpine

I recommend a separate docker-compose.ci.yml overlay for CI-specific overrides (disabling volumes that assume local host paths, adding healthchecks) rather than reusing the exact dev Compose file unmodified:

# docker-compose.ci.yml
services:
  db:
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 5s
      retries: 10
  api:
    depends_on:
      db:
        condition: service_healthy

Step 3: Write the Jenkinsfile

pipeline {
    agent any

    environment {
        COMPOSE_PROJECT_NAME = "myapp-${env.BUILD_NUMBER}"
        REGISTRY = 'myregistry.com/myorg'
    }

    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }

        stage('Build All Services') {
            steps {
                sh 'docker compose -f docker-compose.yml -f docker-compose.ci.yml build'
            }
        }

        stage('Start Stack') {
            steps {
                sh 'docker compose -f docker-compose.yml -f docker-compose.ci.yml up -d'
                sh 'docker compose ps'
            }
        }

        stage('Run Integration Tests') {
            steps {
                sh '''
                    docker compose exec -T api npm run test:integration
                '''
            }
        }

        stage('Tag and Push Changed Images') {
            when { branch 'main' }
            steps {
                script {
                    def commitSha = env.GIT_COMMIT.take(7)
                    ['api', 'frontend'].each { service ->
                        sh "docker tag ${env.COMPOSE_PROJECT_NAME}-${service}:latest ${REGISTRY}/${service}:${commitSha}"
                        sh "docker push ${REGISTRY}/${service}:${commitSha}"
                    }
                }
            }
        }
    }

    post {
        always {
            sh 'docker compose -f docker-compose.yml -f docker-compose.ci.yml logs > compose-logs.txt || true'
            archiveArtifacts artifacts: 'compose-logs.txt', allowEmptyArchive: true
            sh 'docker compose -f docker-compose.yml -f docker-compose.ci.yml down -v --remove-orphans'
        }
    }
}

The COMPOSE_PROJECT_NAME including BUILD_NUMBER is important – it namespaces containers, networks, and volumes per build, so parallel builds on the same agent don’t collide with each other.

Step 4: Only Build/Push Images That Actually Changed

For monorepo-style Docker projects, rebuilding every service on every commit wastes time. Detect changed paths and build selectively:

stage('Detect Changed Services') {
    steps {
        script {
            def changedFiles = sh(script: "git diff --name-only HEAD~1 HEAD", returnStdout: true).trim()
            env.BUILD_API = changedFiles.contains('api/') ? 'true' : 'false'
            env.BUILD_FRONTEND = changedFiles.contains('frontend/') ? 'true' : 'false'
        }
    }
}

stage('Build API') {
    when { environment name: 'BUILD_API', value: 'true' }
    steps {
        sh 'docker compose build api'
    }
}

Step 5: Running Tests Inside vs Against Containers

There are two patterns for integration testing, and picking the right one matters:

stage('Run Integration Tests') {
    steps {
        sh '''
            docker compose -f docker-compose.test.yml run --rm test-runner
        '''
    }
}

# docker-compose.test.yml
services:
  test-runner:
    build: ./tests
    depends_on:
      - api
    environment:
      - API_BASE_URL=http://api:3000

Step 6: Cleaning Up Reliably

Because Compose stacks create networks, volumes, and containers, cleanup has to be unconditional – I always put it in a post { always { ... } } block, never as a regular pipeline step, so a failed test stage doesn’t leave orphaned containers hogging agent resources:

post {
    always {
        sh 'docker compose down -v --remove-orphans || true'
        sh 'docker system prune -f --filter "label=com.docker.compose.project=${COMPOSE_PROJECT_NAME}" || true'
    }
}

Integrating with the Wider Toolchain

Monitoring and Troubleshooting

Security Best Practices

Best Practices

FAQs

Do I need Kubernetes for a Docker Compose-based project, or is Compose enough for CI? Compose is perfectly fine for CI testing purposes even if production runs on Kubernetes – CI just needs to validate that services work together, not mirror production infrastructure exactly.

How do I handle database migrations in this kind of pipeline? Add a dedicated stage or an init container in the Compose stack that runs migrations before the test stage starts, gated behind the database’s healthcheck.

Can this same pipeline structure work with Podman instead of Docker? Yes, Podman supports Compose-compatible syntax (podman compose or podman-compose), and most of this pipeline structure translates directly.

What’s the best way to speed up rebuilds across pipeline runs? Use Docker layer caching (--cache-from with a previously pushed image, or BuildKit’s inline cache) so unchanged layers don’t rebuild from scratch every run.

Should I run this whole pipeline on every single commit? For fast feedback, run unit tests and lightweight builds on every commit, and reserve the full multi-service integration test stack for PRs and merges to shared branches, to keep short-cycle feedback fast.

Summary

Setting up Jenkins for a genuinely Docker-native project means treating Compose as a first-class part of your pipeline, not an afterthought – build all services, start the stack with proper healthchecks, run integration tests against real inter-service networking, and tear everything down unconditionally. Layer in selective builds for monorepos, per-build namespacing for safe parallelism, and mandatory vulnerability scanning before anything gets pushed, and you end up with a pipeline that actually reflects how your multi-container application behaves in the real world, not just how one isolated image builds.

References

Exit mobile version