Deploying Jenkins on HashiCorp Nomad with Terraform

I’ve been playing with HashiCorp Nomad recently, so made a 5-node Raspberry Pi cluster to experiment with. One of the things I wanted to be able to do with it was have an easy way to build and deploy Docker images for the ARM architecture. Enter Jenkins!

Jenkins is as venerable as it is old. It has fallen out of fashion more recently and has a historic bad reputation for being difficult to maintain and run. That hasn’t stopped a large open source community improving it though, and along with modern deployment techniques I believe it is as powerful and useful as ever. Certainly enough for building some ARM-Docker images on my Raspberry Pis reliably and repeatably!

As mentioned in my previous post about the cluster itself, i’m using NFS storage to maintain state for Jenkins. Configuration, plugins and job history live here so that I have can have Jenkins run in a Nomad/Docker job without worrying about state. To do this the NFS volume is mounted on each Raspberry Pi, so it does not matter where Nomad schedules the job on launch.

Nomad has the jenkins_volume defined as “local” (its actually NFS) storage on each node in the client configuration:

client {
  enabled = true
  meta {
  }
  host_volume "jenkins_volume" {
    path = "/nfs/nomad/jenkins"
    read_only = false
  }
}

This is launched on the Pi cluster as a Nomad job using Terraform. The TF code:

# Configure the Nomad provider
provider "nomad" {
  address = "http://nomad.service.consul:4646"
}

# Register the Jenkins Nomad job
resource "nomad_job" "jenkins" {
  jobspec = file("${path.module}/jobs/jenkins.hcl")
}

The Nomad job specification is as follows:

job "jenkins" {
  datacenters = ["DC0"]
  type="service"
  group "jenkins-group" {
    volume "jenkins_volume" {
      type = "host"
      source = "jenkins_volume"
      read_only = false
    }
    task "jenkins-task" {
      driver = "docker"
      service {
        port = "http"
        name = "jenkins"
      }
      service {
        port = "jnlp"
        name = "jenkins-jnlp"
      }
      resources {
        cpu    = 2000
        memory = 512
        network {
          port "http" { static = 8082 }
          port "jnlp" { static = 50000 }
          mbits = 10
        }
      }
      volume_mount {
        volume      = "jenkins_volume"
        destination = "/var/jenkins_home"
      }
      config {
        image = "glharris/rpi-jenkins:latest"
        port_map = {
          http = 8080
          jenkins_jnlp = 50000
        }
        dns_servers = ["${attr.unique.network.ip-address}"]
      }
    }
  }
}

Now running terraform apply:

Jenkins is now accessible on http://jenkins.service.consul:8082

This can then be used to run some more terraform to configure Jenkins jobs themselves. I’ll add another post on the specifics of the Jenkins configuration:

I have deployed two Jenkins pipelines. One to build and push an ARM-architecture Jenkins server Docker image, and another for the Jenkins agent: