Offensive Security Researcher & Quantum Cryptography Analyst

How to Use Jenkins with Splunk for Log Analysis

Abdul Wahab Junaid
How to Use Jenkins with Splunk for Log Analysis

Anyone who has debugged a failed deployment at midnight knows the pain of scrolling through raw Jenkins console output looking for the one line that matters. After doing this too many times, I started piping Jenkins logs into Splunk, and it completely changed how I troubleshoot pipelines. Instead of grepping through text on a build page, I get dashboards, alerts, and searchable history across every job. This article walks through exactly how to connect the two.

Why Combine Jenkins and Splunk

Jenkins is excellent at running pipelines but mediocre at long-term log retention and analysis. Splunk, on the other hand, is built specifically for ingesting, indexing, and searching massive volumes of log data. Together they give you:

Jenkins Architecture Recap

Jenkins generates several categories of logs worth sending to Splunk:

Understanding this separation matters because each log type has a different volume and urgency profile — audit logs are low-volume but high-importance, while console logs are high-volume and mostly useful for troubleshooting a specific run.

Prerequisites

Step 1: Set Up the Splunk HTTP Event Collector (HEC)

In Splunk Web, go to Settings > Data Inputs > HTTP Event Collector > New Token. Give it a name like jenkins-hec, select or create an index (e.g., jenkins_logs), and enable indexer acknowledgment if you need delivery guarantees. Copy the generated token — you’ll need it in Jenkins.

Make sure HEC is globally enabled under Global Settings, and note the port (default 8088).

Step 2: Install the Splunk Plugin for Jenkins

In Jenkins, go to Manage Jenkins > Plugins > Available Plugins and search for “Splunk Plugin”. Install it and restart Jenkins if prompted.

Then configure it globally: Manage Jenkins > System > Splunk Configuration:

Step 3: Enable Logging in Your Pipeline

For declarative pipelines, add the Splunk logging step so console output is streamed as the build runs, not just archived after the fact:

pipeline {
    agent any

    options {
        // Splunk plugin listens on the build's console output stream
    }

    stages {
        stage('Build') {
            steps {
                echo 'Starting build...'
                sh 'mvn clean install'
            }
        }
        stage('Test') {
            steps {
                sh 'mvn test'
            }
        }
    }

    post {
        always {
            splunkins consoleLogEnabled: true,
                       gzipDisabled: false,
                       maxEventsBatchSize: '1000'
        }
    }
}

The splunkins post step (provided by the Splunk plugin) forwards the console log to your configured HEC index once the build finishes.

Step 4: Forward System-Level Jenkins Logs

Console logs cover pipeline output, but you also want the Jenkins service logs (/var/log/jenkins/jenkins.log) in Splunk. The cleanest way is the Splunk Universal Forwarder installed directly on the Jenkins host:

wget -O splunkforwarder.tgz "https://download.splunk.com/products/universalforwarder/releases/latest/linux/splunkforwarder-linux.tgz"
tar xvzf splunkforwarder.tgz -C /opt
/opt/splunkforwarder/bin/splunk start --accept-license
/opt/splunkforwarder/bin/splunk add forward-server <splunk-indexer>:9997
/opt/splunkforwarder/bin/splunk add monitor /var/log/jenkins/jenkins.log -index jenkins_logs

This ensures both application-level (build) and infrastructure-level (service) logs land in the same index for correlation.

Step 5: Build Splunk Dashboards and Alerts

Once data is flowing, a few searches I keep pinned:

Failed builds in the last 24 hours:

index=jenkins_logs "Finished: FAILURE"
| stats count by job_name
| sort -count

Average build duration trend:

index=jenkins_logs sourcetype=jenkins:console
| rex field=_raw "Build duration: (?<duration>\d+)"
| timechart avg(duration) by job_name

Alert on repeated failures for the same job (potential flaky pipeline):

Create a saved search with the failed-builds query above, schedule it every 15 minutes, and set an alert action to fire when count > 3 for the same job_name, sending a Slack or email notification.

Correlating Deployments with Application Errors

The real payoff comes when you tag deployment events and cross-reference them against application logs already in Splunk. Add a marker event from your Jenkinsfile right after a deploy stage:

stage('Notify Splunk of Deployment') {
    steps {
        sh '''
            curl -k https://splunk-hec-host:8088/services/collector/event \
              -H "Authorization: Splunk $HEC_TOKEN" \
              -d '{"event": "deployment", "sourcetype": "jenkins:deploy", "fields": {"job":"'"$JOB_NAME"'","build":"'"$BUILD_NUMBER"'","status":"success"}}'
        '''
    }
}

Then in Splunk, overlay deployment markers on an application error-rate timechart to instantly see whether a spike in errors correlates with a specific release.

Real-World Workflow

A workflow I rely on in production:

  1. Jenkins pipeline runs and streams console logs to Splunk in real time via the plugin.
  2. A deployment marker event is sent to Splunk immediately after a successful deploy.
  3. A Splunk dashboard panel shows error rate for the 30 minutes before and after each deployment marker.
  4. If error rate crosses a threshold post-deploy, a Splunk alert fires and pings the on-call channel — often before a human even notices.

Security Considerations

Troubleshooting

Building a Jenkins Health Dashboard in Splunk

Beyond individual alerts, a single dashboard that summarizes CI/CD health at a glance is one of the most useful things I’ve built with this integration. A few panels worth including:

Build success rate over the last 7 days (by job):

index=jenkins_logs sourcetype=jenkins:console
| eval status=if(match(_raw, "Finished: SUCCESS"), "success", "other")
| stats count by job_name, status
| eventstats sum(count) as total by job_name
| where status="success"
| eval success_rate=round((count/total)*100, 1)
| table job_name, success_rate

Longest-running jobs (candidates for optimization):

index=jenkins_logs sourcetype=jenkins:console
| rex field=_raw "Build duration: (?<duration>\d+)"
| stats avg(duration) as avg_duration by job_name
| sort -avg_duration
| head 10

Deployment frequency (a core DORA metric):

index=jenkins_logs sourcetype="jenkins:deploy" status=success
| timechart span=1d count by job

Pin these to a shared dashboard so engineering leads can see deployment frequency, change failure rate, and build stability trends without digging through individual job pages — turning Jenkins/Splunk data into the kind of metrics that actually inform process decisions.

Retention and Cost Management

Splunk indexing costs scale with data volume, and Jenkins console logs can get noisy fast, especially from verbose dependency installs or test runners. A few things that keep costs sane:

FAQs

Is the Splunk plugin required, or can I just use a Universal Forwarder for everything? A Universal Forwarder alone works but loses the build-level metadata (job name, build number, result) the plugin attaches automatically, which makes searches much less useful.

Can I use Splunk Cloud instead of self-hosted Splunk? Yes, the HEC endpoint and plugin configuration work identically; just use your Splunk Cloud HEC URL and token.

Does this add significant latency to builds? The splunkins post step runs after the build completes and is generally fast, but very large console logs can add a few seconds — enable gzip compression to reduce transfer time.

Can I search Jenkins job configuration changes in Splunk too? Yes, if you also forward Jenkins’ audit trail (via the Audit Trail plugin writing to a log file that the Universal Forwarder monitors).

Summary

Wiring Jenkins into Splunk turns scattered console logs into a searchable, alertable, correlatable data source. Between the Splunk plugin for build-level events and a Universal Forwarder for system logs, you get full visibility into both what Jenkins is doing and how it affects the systems it deploys to — which is exactly what you need when something breaks at 2 AM.

References

Exit mobile version