How to update Nomad service configuration?

There is a surprising, not only for me, feature of Nomad that requires a little of understanding.

When a job is running a Nomad and you update the job specifications, generally two classes of actions can happen. There are “destructive job updates” and non-destructive ones. The destructive, cause the currently running processes to be killed and new ones spawned. Not necessarily in this order, Nomad may first run the new instances of processes, wait for them to be healthy, and then kill the old ones, in what is called in Nomad as Blue/Green Deployments. You can see if a job update will result in destructive update by using Nomad plan.

There are also non-destructive job updates and there are very little of those. Generally, updating service health check is a non-destructive job update, and this is the common example I see.

Updating job “meta” is a destructive update, only because the meta is exported for the process as NOMAD_META_<name> variables. It is not really possible to refresh environment variables of a process. To update the meta exported variables, the process has to be restarted.

The current preferred way of distributing program configuration with the service are templates. Templates themselves are templated, and when the result of rendering the template changes, Nomad will detect that and execute an user configured action depending on the change_mode configuration. That’s great, you can call nginx -s reload when your nginx configuration changes. No, not really.

When the result of rendering the template changes, then yes. But if manually change the template, this is considered a destructive update, and will in turn ignore the change_mode configuration and restart your service (related: issue #6151 issue #17155 Updating a Nomad job template for running tasks without restarting the containers Nomad task constantly restarts due to uncontrolled template rerender ).

The solution here is rather what looks like a workaround. The template itself should be something static that does not change. Instead, the source of the information of the template should change whenever you make configuration change. The lately introduced Nomad variables look ideally to use here.

The template itself reference a path in the nomad variable, like:

   template {
        destination = "local/nginx.conf"
        data = "{{with nomadVar \"nomad/jobs/job_id\"}}{{index \"nginx.conf\".}}{{end}}"
        change_mode = "script"
        change_script {
            command = "sh"
            args = ["-c", "nginx -t reload"]
        }
    }

Then, having a locally stored nginx.conf you can load it into Nomad variables using command line whenever you change the file.

cat nginx.conf | jq -R '{".nginx.conf": .}' | nomad var put -in json nomad/jobs/job_id

Trying to write a small Bash script to handle multiple files and automate it, sprinkled to life now unfinished nomad-vardir project. Maybe one day I’ll revisit and finish it.

Leave a Comment

Scroll to Top