Nomads at Hashicorp Nomad

HashiCorp Nomad tool is a great tool, advertised and promoted as:

A simple and flexible scheduler and orchestrator to deploy and manage containers and non-containerized applications across on-prem and clouds at scale.

I do not want to review Nomad by itself and give it a bad, or good, review and scale it on some 0-5 scale. It is what it is. It found its niche that it fills, in companies with a lot of long-running services and those who can’t switch to Kubernetes. From my perspective Nomad is “simple” scheduler.

I am not coming from these companies. I have mixed workload – I have stuff I want to run temporary, depending on the time of day, I have dependency between hundreds of batch jobs, and I have services to run temporary depending on time of day, and then, on the end of the list, I have 24×7 services that finally in those Nomad specializes in. What I want to do, is to dive into missing and existing features that are frustrating and surprising for me, which I do not understand nor could find the history behind for me questionable decisions to implement them.

Priority is hit man score

Each job in Nomad has a priority field. You might think – the job with the highest priority goes first? No. Och god, no. I can’t imagine how many times my colleagues asked about it. This is the most surprising element of Nomad scheduler. A simple sorting of pending jobs over priority is enough to satisfy the minimum of expectancy of users to have jobs queued in some priority order, especially when a job property named “priority” exists. That is just all false. Nope, not in Nomad.

The documentation states in https://developer.hashicorp.com/nomad/docs/job-specification/job#priority :

priority (int: 50) – Specifies the job priority which is used to prioritize scheduling and access to resources. Must be between 1 and job_max_priority inclusively, with a larger value corresponding to a higher priority. If value 0 is provided this will fallback to job_default_priority. Priority only has an effect when job preemption is enabled. It does not have an effect on which of multiple pending jobs is run first.

That last, extremely important sentence, was not there when I started working with Nomad. I had a whole utility build specially around the priority order of scheduling inside Nomad. Sadly, the whole utility had to be rewritten. On that later.

In the end, priority in Nomad is not priority. Priority property of job is a score that is taken into account when preempting a different job. If a different job is already running, and a new job comes along that there is no place for, and it has that score high enough, the running job is stopped and the new job takes it place. This is not “priority”. This is “hit man score” – how much of a probability does your job have to kill a different one.

No users

Nomad has tokens. These tokens are just UUID numbers that nomad stores in a database. But Nomad does not store who has started a job. Just these tokens. And, do you want to create a separate token for each, you end up with a database. How do you want to manage it?

The Vault as an OIDC for Nomad does not work successfully for me, at least in our environment.

Short memory, but short

Nomad has an idea of garbage collector, not only in go language it is written, but in the jobs themselves. This affects stopped jobs – no longer running, no longer scheduled to be running. Once a job is stopped, after some predefined time, the job is removed.

No, the job is removed might sound not that bad, but it is bad. And by bad, I mean all the logs, all the data that the job has ever run is gone. Getting used to it was hard, as when a service was running on Friday and broke on Friday, by Monday not only we did not know it ever existed, got no alerts, but all the logs in Nomad were gone. Puff. Job not there. No logs to analyze. You are alone.

This is not that big of a deal to send an email, or execute a custom script when a job terminates. Or starts. Or something happens. Overall, the answer is Nomad does not include native methods to trigger alerts and alerts are expected to be fulfilled by an external process.

So you have to not only write your service, you have to write a service that will watch that service running, and write a service that will monitor if nomad is healthy, and if it is not, send an alert. That all is simple in itself for any programmer, but… It takes time.

The lack of API expressibility

With MySQL, to get all the jobs running on a specific list of hosts, you could do select * from allocations where hostname in (host1, host2). That would be great. But not in Nomad.

HashiCorp developed their own database. And not one, but two. Firstly, we have raft consensus protocol. Raft – it’s great. I have no idea how it works. What it does, it makes sure I have one maser and multiple followers, and each follower can become a master when it fails. It works. It’s stable.

What I lack is expressiveness. Anything more to express than simple query that I am able to take with HTTP API Nomad exposes. What I end up doing in 99% of cases, is that I have to do thousands of API calls, basically download the whole current state of Nomad and execute the joins myself in python. Thousands of API calls result in inconsistent state and long execution time.

Lack of job ordering.

There are different types of workflows. But there are workflows that want to run services – and Nomad seems to be for that. But batch workflows of computation jobs is not Nomad strength. More, it is a weakness.

There are init tasks, but – again!- they lack expressiveness. You can just start one job – most probably a service like MySQL – before one other. In case of computation jobs, I want to run 10000 batch processes, wait till they finish, then run another 10000, wait till they are finished, then run another 10000…. and so on. This is impossible to do in Nomad, and once again, has to be achieved by external tools.

Solutions, solutions, solutions

I started to have a feeling after working with Nomad for about 6 months. Often I or my colleagues come with a simple thing to do that we expect Nomad should be capable of doing. There is mostly one answer from Nomad we get – bad luck or can't do or no-no.

Nomad is simple. I might say, too simple. The complexity of Nomad has been thrown into state consistency and raft protocol, not into the simple down-to-earth features we people require and use. Hopefully the number of my feature requests on GitHub will change it.

Overall, we are programmers. By the end of the day, stuff has to work, and we come up with ideas to make them work. Then, we make them idea to life.

Database of jobs

Firstly, to solve the lack of API expressibility and history, I wrote a nomad-database-exporter. It is a simple python script that listens on Nomad events, and jam them into PostgreSQL database. The database has some tables for what Nomad stores – allocations, jobs, evaluations, nodes. All tables have same 3 columns – UUID, changed_date and JSON data.

PostgreSQL has great support for JSON data. Each time something happens, the JSON data in the database are updated, or a new entry is created. The changed_date is the last date of change, and is used for cleanup. UUID are not unique, and they can clash, but I assume they won’t, or will seldom clash enough that I won’t notice. With that PostgreSQL database, I can finally do what I need to do – history of jobs in Grafana over last 30 days. and write any queries I imagine.

Schedulers scheduler

Then, there is lack of priority and no users. So we have built our own solution on top of Nomad. And it took time. And failures. It has its own priority queue and calculation, its own database of jobs to run and own logins and users. This is basically a scheduler on top of Nomad scheduler.

And this, here, defeats the point of Nomad altogether. The lack of expressiveness in small steps resulted in you have to have an external tool for that which resulted in writing these external tools, which resulted in implementing a whole external scheduler on top of Nomad. Then why do we need Nomad in the first place? Using our own tool, we could be just scheduling our jobs ourselves without using Nomad at all.

Future

Currently, the answer is technical debt – the more I worked on the topic, the more debt arose, the more was dependent on Nomad. I am not happy with the result. But, on the other hand, Nomad does what it does best – keeps consistency and starts and kills the jobs when needed. At least it can do that.

I’ll see what will happen in the future. Maybe we will run away from Nomad right into SLURM. SLURM has all the features you can imagine, and then some more. What is does not have is a good marking team flashing it before your eyes now and then. Kubernetes is too big, it requires the whole machine to be for Kubernetes, and who would want to move the whole workflow to Kubernetes all at once.

For now, on my custom server, I also have Nomad. And a service that I wrote that watches over Nomad to check if any service stopped. And a service that sends an alert when needed. And a service that…

Leave a Comment

Scroll to Top