How to Use Jenkins for Multi-branch Pipelines

The first time I had to manually create a new Jenkins job for every feature branch, I knew there had to be a better way. Multibranch pipelines solved that problem completely — Jenkins now discovers branches on its own, spins up a pipeline for each, and tears it down when the branch disappears. If you’re still hand-creating jobs per branch, this article will save you a lot of clicking.

What Is a Multibranch Pipeline

A multibranch pipeline is a Jenkins job type that automatically scans a source repository (or a GitHub/GitLab organization) and creates a separate pipeline for every branch that contains a Jenkinsfile. Delete the branch, and Jenkins automatically discards the corresponding pipeline. This maps naturally onto Git workflows like GitFlow or trunk-based development with feature branches.

Jenkins Architecture in a Multibranch Context

• Controller: Runs the branch-scanning logic (periodically or via webhook) and manages the folder-like structure of per-branch pipelines.
• Branch indexing: A background process that checks the repository for new, updated, or deleted branches and syncs Jenkins job state to match.
• Per-branch pipeline: Each discovered branch gets its own independent build history, status, and Jenkinsfile execution — effectively a lightweight, disposable job.
• Agents: Still execute the actual build steps; multibranch just changes how jobs are organized and discovered, not how builds run.

Prerequisites

• Jenkins (2.4+ LTS)
• A Git-based source repository (GitHub, GitLab, Bitbucket) with branch protection or PR workflows
• A Jenkinsfile committed at the root of each branch you want built
• Appropriate credentials for Jenkins to read (and optionally write status checks to) the repository

Step 1: Install Required Plugins

• Pipeline: Multibranch (bundled with most Jenkins installs by default)
• Git Plugin
• GitHub Branch Source Plugin (for GitHub) or GitLab Branch Source Plugin (for GitLab)
• Pipeline: Stage View (nice-to-have, visualizes stage timing per branch)

Step 2: Create a Multibranch Pipeline Job

1. From the Jenkins dashboard, click New Item.
2. Enter a name, select Multibranch Pipeline, and click OK.
3. Under Branch Sources, click Add source and choose GitHub (or Git, GitLab, Bitbucket).
4. Provide the repository URL and credentials (a personal access token or SSH key stored in Jenkins Credentials).
5. Under Behaviors, configure what counts as a “branch” — you can include/exclude by name pattern, discover pull requests, or discover tags.
6. Set the Build Configuration to “by Jenkinsfile” and specify the path if it’s not at the repo root.
7. Under Scan Multibranch Pipeline Triggers, enable periodically if not otherwise run as a fallback, but prefer webhook-based triggering for instant scans.

Step 3: Configure Webhook-Based Discovery

Polling is slow and wasteful. Instead, configure a webhook so GitHub notifies Jenkins the instant a branch is pushed or a PR is opened:

In your GitHub repo: Settings > Webhooks > Add webhook

• Payload URL: https://your-jenkins-url/github-webhook/
• Content type: application/json
• Events: “Pushes” and “Pull requests”

With the GitHub Branch Source Plugin installed, Jenkins automatically registers itself for organization-level webhooks if you use a GitHub Organization folder instead of a single multibranch pipeline — useful if you manage many repos.

Step 4: Write a Branch-Aware Jenkinsfile

The real power of multibranch pipelines comes from writing a single Jenkinsfile that behaves differently depending on which branch triggered it:

pipeline {
    agent any

    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }

        stage('Install & Test') {
            steps {
                sh 'npm ci'
                sh 'npm test'
            }
        }

        stage('Deploy to Dev') {
            when {
                branch 'develop'
            }
            steps {
                sh './deploy.sh dev'
            }
        }

        stage('Deploy to Staging') {
            when {
                branch 'release/*'
            }
            steps {
                sh './deploy.sh staging'
            }
        }

        stage('Deploy to Production') {
            when {
                branch 'main'
            }
            steps {
                input message: 'Deploy main to production?'
                sh './deploy.sh production'
            }
        }

        stage('Feature Branch Checks Only') {
            when {
                not {
                    anyOf {
                        branch 'main'
                        branch 'develop'
                        branch 'release/*'
                    }
                }
            }
            steps {
                echo 'Feature branch — running tests only, no deployment.'
            }
        }
    }
}

The when { branch '...' } directive is the key mechanism: the same Jenkinsfile runs on every branch, but stages conditionally activate based on branch name patterns.

Handling Pull Requests

Enable Discover pull requests from origin (and from forks, if you accept external contributions) under Branch Sources > Behaviors. Each open PR then gets its own ephemeral pipeline run, perfect for gating merges on CI status:

stage('PR Validation') {
    when {
        changeRequest()
    }
    steps {
        sh 'npm run lint'
        sh 'npm test -- --coverage'
    }
}

changeRequest() evaluates true only when the build was triggered by a pull request, letting you run PR-specific checks like coverage diffing without affecting regular branch builds.

Shared Libraries to Avoid Duplication

As the number of branches (and repos) grows, copy-pasted Jenkinsfile logic becomes a maintenance headache. Jenkins Shared Libraries let you centralize common pipeline logic:

@Library('my-shared-library') _

pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                standardBuild()
            }
        }
        stage('Deploy') {
            when { branch 'main' }
            steps {
                standardDeploy('production')
            }
        }
    }
}

Define standardBuild() and standardDeploy() once in a separate Git repo configured under Manage Jenkins > System > Global Pipeline Libraries, and every multibranch pipeline across your organization can reuse them.

Managing Orphaned Branches

By default, Jenkins keeps pipeline history even after a branch is deleted, until it’s explicitly cleaned up. Configure automatic cleanup under Orphaned Item Strategy in the multibranch job configuration — set a retention period so deleted branches’ build history is discarded after, say, 7 days, keeping the Jenkins UI from becoming a graveyard of stale branches.

Real-World Workflow

1. A developer creates feature/checkout-redesign and pushes a commit.
2. GitHub webhook fires; Jenkins discovers the branch within seconds and runs the “Feature Branch Checks Only” path — lint and tests, no deploy.
3. They open a PR into develop; Jenkins runs the changeRequest() validation stage and reports status back to GitHub as a required check.
4. PR merges into develop; Jenkins runs the full pipeline and deploys to the dev environment automatically.
5. Weeks later, release/2.4 is cut; Jenkins deploys it to staging.
6. main is updated via a fast-forward merge; Jenkins pauses at the manual approval gate before deploying to production.

Best Practices

• Keep the Jenkinsfile as thin as possible — push logic into shared libraries so branch-specific behavior stays readable.
• Use branch naming conventions (feature/*, release/*, hotfix/*) so when conditions stay simple and predictable.
• Set sensible orphaned-item retention to avoid unbounded disk usage from old branch build artifacts.
• Protect main and release/* branches in GitHub/GitLab so only PR-merged, CI-validated code lands there.
• Use parallel stages for independent checks (lint, unit tests, security scan) to keep feedback fast on every branch.

Troubleshooting

• New branch not appearing in Jenkins: Trigger a manual “Scan Repository Now” from the job page, and verify the webhook is actually firing (check GitHub’s webhook delivery log for errors).
• when { branch } not matching as expected: Remember branch name patterns are glob-style, not regex, unless you explicitly use branch pattern: '...', comparator: 'REGEXP'.
• PR builds not triggering: Confirm “Discover pull requests” behavior is enabled and, for forked PRs, that the appropriate trust policy is set (build only after approval, for security).
• Shared library changes not taking effect: Multibranch pipelines cache library versions per build; confirm you’re referencing the right branch/tag of the shared library (@Library('my-lib@main')).

Parallelizing Checks Across Branches for Faster Feedback

As a multibranch setup scales to dozens of active branches, keeping each individual pipeline fast matters as much as the discovery mechanism itself. Structure the common checks — lint, unit tests, security scan — as parallel stages so feedback on a feature branch arrives in minutes, not tens of minutes:

stage('Quality Gates') {
    parallel {
        stage('Lint') {
            steps { sh 'npm run lint' }
        }
        stage('Unit Tests') {
            steps { sh 'npm test' }
        }
        stage('Security Scan') {
            steps { sh 'npm audit --audit-level=high' }
        }
    }
}

Combine this with the agent { kubernetes { ... } } pattern (if your Jenkins runs on Kubernetes) so each parallel branch of the pipeline gets its own container, rather than competing for a single agent’s resources.

Restricting Deployment Stages by Contributor Trust

For organizations that accept pull requests from forks, it’s worth being deliberate about what a fork-originated PR pipeline is allowed to do. Discovery settings under Behaviors let you require that PRs from forks only build after being explicitly approved by a maintainer, and deployment stages should never run for fork-based PRs at all — only the read-only test/lint stages:

stage('Deploy Preview Environment') {
    when {
        allOf {
            changeRequest()
            not { changeRequest(fork: true) }
        }
    }
    steps {
        sh './deploy-preview.sh'
    }
}

This not { changeRequest(fork: true) } condition ensures deployment credentials are never exposed to a pipeline triggered by an untrusted external contribution, while still giving internal branches and PRs full CI/CD coverage.

Branch Indexing Performance at Scale

Once an organization folder manages dozens of repositories, periodic branch indexing across all of them can become slow and resource-intensive on the controller. A few things that help:

• Rely primarily on webhook-triggered scans rather than frequent periodic polling, reserving periodic scans as an infrequent fallback (e.g., once daily) rather than the primary discovery mechanism.
• Use the Discover branches behavior’s exclusion patterns to skip branches that will never need CI (archive/*, experimental/*) rather than indexing and immediately ignoring them.
• Consider splitting a very large organization folder into several smaller ones if controller load from indexing becomes a bottleneck.

FAQs

Can one multibranch pipeline job manage builds for multiple repositories? No — a multibranch pipeline maps to one repository. For multiple repos, use a GitHub Organization folder, which creates a multibranch pipeline per repo automatically.

How is a multibranch pipeline different from a regular pipeline job with parameters? A parameterized pipeline is one job with manually chosen inputs; a multibranch pipeline dynamically creates and destroys jobs per branch automatically based on repository state.

Can different branches use completely different Jenkinsfiles? Yes, since each branch has its own file at the configured path — a main branch’s Jenkinsfile can be entirely different from a develop branch’s, though keeping them consistent via when conditions is usually cleaner.

Does multibranch support Bitbucket and GitLab, or just GitHub? Yes, via the respective Branch Source plugins (GitLab Branch Source, Bitbucket Branch Source), with equivalent webhook and PR/MR discovery support.

Summary

Multibranch pipelines eliminate the manual overhead of managing one Jenkins job per branch. With webhook-driven discovery, a single branch-aware Jenkinsfile, and shared libraries for common logic, you get automatic CI/CD coverage across every feature branch, release branch, and pull request — with zero manual job creation as your repository grows.

References

• Jenkins Multibranch Pipeline Documentation
• GitHub Branch Source Plugin
• Jenkins Shared Libraries Documentation
• Jenkins Pipeline Syntax Reference