Custom GitLab Runner Executor for Nomad: Running GitLab CI/CD Jobs in Nomad
As organizations continue to embrace DevOps and CI/CD methodologies, the need for flexible and efficient job runners is ever-increasing. GitLab CI/CD has gained significant popularity for automating the software development process, and HashiCorp’s Nomad has emerged as a powerful orchestrator for managing containerized applications and tasks.I wrote a custom GitLab Runner executor implementation called “nomad-gitlab-runner” part of “nomad-tools” in pypi, which enables GitLab CI/CD jobs to run within Nomad.
Introduction to nomad-gitlab-runner
Nomad-GitLab-Runner is a specialized executor designed to integrate GitLab CI/CD pipelines with HashiCorp’s Nomad, extending the capabilities of GitLab Runner to run jobs in a Nomad cluster. This integration brings several advantages to the table, including improved job isolation, scaling, and enhanced resource management. Let’s delve into the key features of nomad-gitlab-runner:
Execution modes:
- Raw Execution (
raw_exec): In this mode, each GitLab Runner job corresponds to a single Nomad task. The executor is executed usingnomad alloc execwithin the task spawned in Nomad. Custom entrypoint scripts can be provided for fine-grained control. This mode is suitable for running simple scripts and commands. - Exec Execution (
exec): Similar toraw_exec, each GitLab Runner job corresponds to a Nomad task. This mode is designed for advanced customization and supports the execution of tasks with specific niceness levels, OOM settings, taskset, and user switching. It also has a fallback to theshshell ifbashis unavailable. - Docker Execution (
docker): In this mode, the Nomad job contains multiple tasks, similar to GitLab Runner’s Docker executor. It is ideal for running jobs with containerized environments. One task handles cloning the repository and managing artifacts, while the main task executes the job image entrypoint. Custom entrypoint wrapper settings are provided through the Docker configuration.
Service Support
Nomad-GitLab-Runner supports specifying services in the .gitlab-ci.yml file when using the docker mode. Each service is represented as a separate task in the Nomad job, and tasks share the same network stack in bridge mode. A waiter task is created to wait for services to respond, similar to how GitLab Runner’s Docker executor performs service health checks.
To enable service support, a few prerequisites must be met. The waiter helper image must have /var/run/docker.sock mounted, and Nomad must support the ‘bridge’ Docker network driver.
Configuration and Usage
To set up Nomad-GitLab-Runner, you need to configure your GitLab Runner and Nomad, as well as your .gitlab-ci.yml and ACL policies for Nomad. Below are some example configuration files to get you started:
- GitLab Runner Configuration (
/etc/gitlab-runner/config.toml): This configuration file specifies the executor as “custom” and sets various execution and cleanup commands.
toml
[[runners]] id = 27898742 executor = "custom" [runners.custom] config_exec = "nomad-gitlab-runner" config_args = ["config"] prepare_exec = "nomad-gitlab-runner" prepare_args = ["prepare"] run_exec = "nomad-gitlab-runner" run_args = ["run"] cleanup_exec = "nomad-gitlab-runner" cleanup_args = ["cleanup"]
- Nomad Configuration (
/etc/gitlab-runner/nomad.yaml): This configuration file provides settings for Nomad, such as tokens, addresses, and specific runner overrides.
yaml
default: NOMAD_TOKEN: "1234567" NOMAD_ADDR: "http://127.0.0.1:4646" NOMAD_NAMESPACE: "gitlabrunner" 27898742: mode: "docker" CPU: 2048 MemoryMB: 2048 MemoryMaxMB: 2048 docker: image: "alpine:latest" services_privileged: true override: task_config: cpuset_cpus: "2-8"
- Example
.gitlab-ci.yml: This YAML file demonstrates how to use thedockermode with GitLab CI/CD, including service configurations.
yaml
default: image: docker:24.0.5 services: - docker:24.0.5-dind docker_dind_auto: variables: DOCKER_TLS_CERTDIR: "/certs" script: - docker info - docker run -ti --rm alpine echo hello world docker_dind_alloc: variables: DOCKER_CERT_PATH: "/alloc/client" DOCKER_HOST: tcp://docker:2376 DOCKER_TLS_CERTDIR: "/alloc" DOCKER_TLS_VERIFY: 1 script: - docker info - docker run -ti --rm alpine echo hello world
- Nomad ACL Policy Example: This policy defines the necessary permissions for creating jobs and executing ‘raw_exec’ tasks.
hcl
namespace "gitlabrunner" { policy = "write" capabilities = ["alloc-node-exec"] }
Conclusion
Nomad-GitLab-Runner bridges the gap between GitLab CI/CD and HashiCorp Nomad, providing a powerful and flexible solution for running CI/CD jobs. With support for different execution modes and service configurations, it offers a versatile approach to executing tasks within a Nomad cluster. By integrating GitLab and Nomad, organizations can achieve greater scalability and resource efficiency while maintaining the flexibility to run jobs in various environments. Whether you’re looking to enhance your CI/CD pipeline or optimize resource utilization, Nomad-GitLab-Runner is a tool worth exploring.