Offensive Security Researcher & Quantum Cryptography Analyst

How to Configure Jenkins for Python Projects

Abdul Wahab Junaid
How to Configure Jenkins for Python Projects

Python projects have their own quirks when it comes to CI/CD — virtual environments, dependency management, multiple interpreter versions, and testing frameworks like pytest all need to play nicely together inside a Jenkins pipeline. I’ve configured Jenkins for everything from small Flask apps to larger Django and data science projects, and in this guide, I’ll share the full setup process, from installation to a production-ready pipeline.

Why Jenkins Works Well for Python

Jenkins doesn’t care what language your project is written in — it just executes shell commands and reports results. That flexibility means Python projects integrate just as smoothly as any other, provided you configure the right interpreter versions, virtual environments, and dependency caching.

Jenkins Architecture Context

For Python projects, I recommend running builds on agents that either have Python pre-installed or use the Pyenv Pipeline Plugin / ShiningPanda Plugin to manage multiple Python versions. Isolating each build in its own virtual environment prevents dependency conflicts between different projects sharing the same agent.

Step 1: Install Jenkins

If not already installed:

sudo apt update
sudo apt install openjdk-17-jdk -y
curl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key | sudo tee \
  /usr/share/keyrings/jenkins-keyring.asc > /dev/null
echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc]" \
  https://pkg.jenkins.io/debian-stable binary/ | sudo tee \
  /etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt update
sudo apt install jenkins -y
sudo systemctl start jenkins

Step 2: Install Python on the Jenkins Agent

sudo apt install python3 python3-pip python3-venv -y
python3 --version
pip3 --version

If you need multiple Python versions (e.g., 3.9, 3.10, 3.11), consider installing pyenv on the agent:

curl https://pyenv.run | bash
pyenv install 3.11.4
pyenv install 3.10.11

Step 3: Install Relevant Jenkins Plugins

Step 4: Connect Your Git Repository

Create a new Pipeline job and point it to your repository, either via manual configuration or “Pipeline script from SCM” if you’re storing a Jenkinsfile in your repo (recommended).

Step 5: Write a Jenkinsfile for a Python Project

Here’s a complete example using a virtual environment and pytest:

pipeline {
    agent any

    environment {
        VENV_DIR = '.venv'
    }

    stages {
        stage('Checkout') {
            steps {
                git branch: 'main', url: 'https://github.com/yourusername/your-python-app.git'
            }
        }

        stage('Set Up Virtual Environment') {
            steps {
                sh 'python3 -m venv $VENV_DIR'
                sh '. $VENV_DIR/bin/activate && pip install --upgrade pip'
            }
        }

        stage('Install Dependencies') {
            steps {
                sh '. $VENV_DIR/bin/activate && pip install -r requirements.txt'
            }
        }

        stage('Lint') {
            steps {
                sh '. $VENV_DIR/bin/activate && flake8 . --max-line-length=100'
            }
        }

        stage('Run Tests') {
            steps {
                sh '. $VENV_DIR/bin/activate && pytest --junitxml=reports/results.xml --cov=app --cov-report=xml'
            }
        }

        stage('Build Package') {
            steps {
                sh '. $VENV_DIR/bin/activate && python setup.py sdist bdist_wheel'
                archiveArtifacts artifacts: 'dist/*', fingerprint: true
            }
        }
    }

    post {
        always {
            junit 'reports/results.xml'
        }
        failure {
            echo 'Build or tests failed. Check console output for details.'
        }
    }
}

A few notes on this setup:

Step 6: Managing Requirements and Caching

For faster builds, consider caching pip downloads between runs:

stage('Install Dependencies') {
    steps {
        sh '. $VENV_DIR/bin/activate && pip install --cache-dir=/var/jenkins_cache/pip -r requirements.txt'
    }
}

Just make sure the cache directory persists across builds (i.e., it’s not wiped by a “clean workspace” step) and that multiple concurrent jobs don’t corrupt a shared cache — using per-agent cache directories usually avoids this.

Step 7: Using Docker for Fully Isolated Python Builds

Rather than relying on the agent’s installed Python version, many teams containerize the build itself:

pipeline {
    agent {
        docker { image 'python:3.11-slim' }
    }
    stages {
        stage('Install & Test') {
            steps {
                sh 'pip install -r requirements.txt'
                sh 'pytest --junitxml=reports/results.xml'
            }
        }
    }
    post {
        always {
            junit 'reports/results.xml'
        }
    }
}

This ensures every build uses an identical, reproducible Python environment regardless of what’s installed on the underlying host.

Step 8: Django/Flask-Specific Considerations

For web frameworks, you’ll often need a test database. A common pattern uses Docker Compose or a service container:

stage('Run Django Tests') {
    steps {
        sh '. $VENV_DIR/bin/activate && python manage.py test'
    }
}

If your tests depend on PostgreSQL or Redis, spin those up as sidecar containers using the Docker Pipeline Plugin, or point to a dedicated test database service already running in your infrastructure.

Step 9: Deployment Stage

Once tests pass, deployment might involve pushing a Docker image, deploying to a PaaS, or using tools like Ansible:

stage('Deploy') {
    steps {
        sh 'docker build -t yourdockerhub/python-app:${BUILD_NUMBER} .'
        sh 'docker push yourdockerhub/python-app:${BUILD_NUMBER}'
        sh 'ansible-playbook -i inventory/production deploy.yml'
    }
}

Real-World Workflow Example

  1. Developer pushes to a feature branch → Jenkins runs lint + unit tests in an isolated virtual environment.
  2. PR merges to main → full test suite runs, coverage report generated.
  3. Docker image built and pushed to a private registry.
  4. Ansible playbook deploys the new image to a staging server.
  5. After manual approval, the same image gets promoted to production.

Testing Across Multiple Python Versions with a Matrix

If your library needs to support several Python versions, Jenkins’ Declarative matrix block lets you test all of them in parallel within a single pipeline:

pipeline {
    agent none
    stages {
        stage('Test Matrix') {
            matrix {
                axes {
                    axis {
                        name 'PYTHON_VERSION'
                        values '3.9', '3.10', '3.11'
                    }
                }
                stages {
                    stage('Test') {
                        agent { docker { image "python:${PYTHON_VERSION}-slim" } }
                        steps {
                            sh 'pip install -r requirements.txt'
                            sh 'pytest'
                        }
                    }
                }
            }
        }
    }
}

This runs the same test suite across three separate Python versions simultaneously, catching version-specific compatibility issues before release.

Data Science and Notebook-Based Projects

For teams running data science pipelines, Jenkins can also automate notebook execution and validation using papermill to run Jupyter notebooks non-interactively:

stage('Run Notebook') {
    steps {
        sh '. $VENV_DIR/bin/activate && papermill analysis.ipynb output.ipynb'
    }
}

This is a handy pattern for validating that data pipelines and exploratory notebooks still execute correctly as dependencies or underlying data sources change.

Troubleshooting Common Issues

Working with Poetry Instead of pip/venv

Some Python teams prefer Poetry for dependency management over plain pip and venv. The pipeline adjustments are minor:

stage('Install Dependencies') {
    steps {
        sh 'pip install poetry'
        sh 'poetry install'
    }
}
stage('Run Tests') {
    steps {
        sh 'poetry run pytest --junitxml=reports/results.xml'
    }
}

Poetry’s lock file (poetry.lock) plays the same role as requirements.txt, ensuring reproducible installs across every build.

Security Best Practices

FAQs

Q: Should I use ShiningPanda Plugin or just plain shell commands with venv? Plain shell commands with venv are simpler and more portable, especially with Pipeline jobs. ShiningPanda is more useful for Freestyle projects needing UI-based Python version selection.

Q: How do I test against multiple Python versions in one pipeline? Use a matrix block in a Declarative Pipeline, or parallel stages each using a different Docker image tag (e.g., python:3.9, python:3.10, python:3.11).

Q: Can Jenkins handle Conda environments instead of venv? Yes — just replace the venv creation and activation commands with the equivalent conda create and conda activate commands.

Q: How do I speed up dependency installation? Use a persistent pip cache directory, or better yet, use a private PyPI mirror/proxy like devpi or a Nexus repository for your organization.

Summary

Configuring Jenkins for Python projects mainly comes down to properly managing virtual environments, dependency installation, and test reporting. Whether you use plain venv, Docker-based isolation, or a plugin like ShiningPanda, the goal is the same: consistent, repeatable builds that catch issues before they reach production.

References

Exit mobile version