How to watch all Nomad allocations of a job?

My main problem with Hashicorp Nomad is that there are missing user-friendly tools to work with it. See at docker – there are so many useful docker commands and wrappers. Nomad doesn’t have them. So let’s write them.

This is how nomad-watch as part of pypi packages nomad-tools came to be. It existed actually for a long time, under different names and being a shell wrapper, that checks if an job has allocation and spawns nomad alloc logs --follow processes in the background. And kills them on exit. You can install the tools with pipx install nomad-tools.

Another issue was running one-off commands. The way Nomad works, it is a scheduler. You post a job. That job is “to be scheduled”. Then it get’s scheduled. Then executed. This execution – called an allocation in Nomad – can fail. It can fail because configuration is wrong, or it can fail because it can’t run the command, or it can finally fail when the command exited with non-zero exit status. Then I required to collect the result of the execution – if it succeeded or not. I used what we programmers always use – created “DONE” files from the job and checked for the existence of these files with some timeout.

One day I decided I need more. I need an actual tool that will work and will watch every allocation that I can restart and kill and it will leave nothing behind. That will “just work” and I wouldn’t have to care about.

This sprinkled into existence nomad-watch run. It takes a job name, runs it, waits for it to be scheduled, wait for it to be running, wait for it to be finished, wait for it to be cleanup ed, and exits with the command exit status (or other non-zero exit status if any stage failed).

The overall goal was to mimic docker-compose. The following shows a short comparison:

docker compose nomad-watch
docker compose up nomad-watch –stop run ./jobname.nomad.hcl
docker compose -d up nomad-watch start ./jobname.nomad.hcl
docker compose down nomad-watch stop jobname
docker compose -f -n 10 logs nomad-watch -f -n 10 job jobname
docker compose logs nomad-watch –no-follow job jobname

There are also additional modes that are mostly for “waiting” for the job to finish some stage. nomad-watch starting jobname waits for a job started with nomad run ./jobname.nomad.hcl to be “started” – have at least one allocation running. nomad-watch stopping jobname waits for a job to be dead – have no running allocation.

nomad-watch is not meant to completely replace nomad, like docker compose does with docker. It is a tool specifically for “watching” nomad jobs, something that I miss in the standard nomad tool. Why I could expand it to wrap all the nomad commands – and, it would be a great thing to do, as some nomad parsing options are really confusing – it is too much work for me at this time.

Hopefully, nomad-watch will gain some recognition. It has helped me numerous times in debugging. While working on an issue I often want to see logs from a running job. For examplefabio logs all proxy requests on my server. Then I type:

nomad-watch -f job fabio

With this command I get a stream of fabio logs directly on my terminal. I can filter those logs with my favourite standard utilities.

Happy watching!

Leave a Comment

Scroll to Top