Recently I published my first PyPI package available at nomad-tools.
There are several tools there that I decided to keep together for the purpose that they keep getting interconnected and sharing the common database. I also was missing several very simple tools for managing Nomad services. Much simpler, very basic, that I have required to work with. I was finding myself writing short scripts with nomad status ... | jq .some.parsing which I could write in python and serialize in a script.
The main utility that I really wanted to work on was nomad-watch. I did not particularly like it. I have written at several scripts that were working very similarly to it.
At first, when debugging, I wanted to see the logs of the Nomad job in terminal. All the logs. Not only stdout or(!) stderr of a single allocation. All the allocations and all the tasks and all the logs, and I would then filter it later with something that I know like grep. nomad exec logs gives only one allocation. What started as while nomad status ...; do for i in some time changed into a python script. Then I decided to start a fully fledged project from it.
stderr stdout; do nomad exec logs & done; done
The other times, I wanted to update a service instance. I wanted to “know” if the updated version of the service “started” successfully or not. What started as manual work and spawning F5 in the browser waiting for the consul to register a service and seeing the redirecting in action, was then a Bash script to check curl exit status in a loop. I wanted something better that would accommodate also that.
So nomad-watch sprinkled to life. It has two major modes that I use – nomad-watch job <thejob> to “watch” the logs of a single job. Watch if it fails, or blocks, or there are no resources to run it, or it blew up, or god knows what has happened to it.
The other “mode” is nomad-watch start the_job_file.nomad.hcl, which will first start the job and then wait for it to start. What “start” means, it’s really hard to define. That was really trouble creating this tool. There are multiple hand-crafted gears stacked together and switches to detect what does it mean to “start” a job. I hope I did good enough explanation in the documentation.
The other side are race condition and the poor state of the Nomad API. Nomad event stream returns different, partial data compared to the query. But that’s not a problem. The real problem is with the structure.
When you have MySQL, you can execute lock the transaction and execute a single query over all tables. Nomad, it’s basically a database. With no history (that’s for another project). It stores all currently running executions of processes, what it calls “deployments”, and nodes information. That is simple, a relational database. Simple. The problem with Nomad is exactly that. It is not MySQL.
I can’t “lock transaction” in Nomad to have a consistent view across all the database. You also can’t execute a SELECT query with a JOIN of another table to get data from two tables at once. And when you execute two queries you are going to get inconsistent view – something might have been changed between them. The half-part glued together solution is to use Nomad stream.
But, firstly, the Nomad stream gives you previous events that were in the buffer, where the size of the buffer is configured at Nomad startup time. And stream gives you new events – you still have to query the current state of the database.
To be honest, writing this abomination wasn’t simple. The multithreaded watching database to serve consistent view of what is in Nomad database. Firstly, the Nomad event stream listener is started. Started, that’s all. Then, after it is started (and buffered), the queries are executed to get current state of the database. Whatever will change in between, will be reflected in the event stream. The current state is then “loaded” inside the database and updated with the event stream running forever. That way I can keep consistent and clear view of the Nomad database inside a python program and keep reacting to change in the event stream.
I am not happy with how the class Db in the source code looks like. Ideally, I think I would want to create a command line tool for it, so it is easily extensible to other languages of other programmers choice. Maybe I’ll do it, just for fun.
In another post I’ll talk a bit about other tools that I added to nomad-tools project.