Bash argparse implementation stories

Bash argparse implementation

I have written a argparse-ish parsing tool in pure Bash. It is fast, it is reliable, it works.

There have been multiple implementation of the parsing of command line arguments. There have been drawbacks and some benefits of each. This article serves as a documentation to me what works and what did not and why.

Long time ago

I am not happy with how Bash handles arguments parsing for a long time. Effectively, people:

  • write their own while [ $# -ne 0 ]; do case $1 in loops and implement argument parsing themselves
  • use getopts

These methods lack what I really wanted a command line tools to do – generate help messages and bash completion. These needed to be written by hand and kept synchronized with the implementation. Same code had to be repeated multiple times in many places. I did not like that.

Shorter time ago

There came libraries. The Bash command line parsing libraries that I know of are:

Stupid implementation of L_argparse

My first try at L_argparse in Bash in 2021 was just a wrong idea. I was passing the arguments from Bash into Python, parsing them in Python, then serializing the result with shlex.quote from Python into Bash.

This was a bad solution. It did not finish the goal that I had and was very slow. Calling Python takes time, a lot of time, Python is slow to load up. The passing resulted in sluggish behavior.

I tried many times to make getopts parse long options.

I tried to write a wrapper around getopt, but I did not like calling an external program to do the argument parsing, because it was a slow and a limited idea. There is no way to represent positional arguments in getopt properly, I would have to write a parser anyway. There is also help generation, completion generation. Completion requires parsing the arguments anyway.

The first implementation of L_argparse.

The first Bash L_argparse implementation was just a testing ground. It used -- to separate groups of L_argparse arguments and double -- -- to separate actual arguments to parse.

The gist is there. It heavily used associative arrays to represent all the internal state. The whole L_asa_* library in L_lib was invented just for L_argparseto work. (The name asa comes from “Associative Array”. Still better than assarray.)

The variable _L_parser represented the current parser in use. It is an associative array of several keys, and the elements where serialized versions of associative arrays from declare -p of them.

Parsing an option like -o consisted of getting the _L_parser[-o] value and loading it up with declare -A _L_optspec=${_L_parser[-o]}. After the sub-associative array was extracted it was possible to work on it. Changing any value required to declare -p it again, so changing was only done when parsing the L_argparse arguments, not when parsing user arguments.

The variable _L_mainsettings represented the parser “main settings” – i.e. constructor options of Python argparse.ArgumentParser() call. It contains properties like description or epilog of the parser.

I also noticed that using -- -- was not great visually and added support for :: :::: and then for -- ----. The parsing “started working”. I was able to write some rudimentary tests and get some results from them. Proper work was being done to figure out edge cases and how to encode them.

How to represent that an option takes a writable file? How to write Bash completion? Completion still requires parsing all arguments, even if some of them can be meaningless, errors can be ignored and if there can be a hint of validity of later option, completion should be attempted. I tried to write both defensive and non-defensive code, two code paths, for non-completion and completion ways.

The gist of this implementation is still there. Many functions names are still there. The internal API stood the time. The way the internal state was represented was slow and inefficient. The serialization of associative variables into strings to be loaded later with declare was very slow. The act of “loading” the with declare effectively prevented from accessing multiple options at the same time.

The work was progressing to create better options parsing implementation, Fish and Zsh completion and more features.

Sub-parsers implementation

Then the work began to be able to at least “represent” how sub-parsers specification could be written in Bash. In Python, it is much easier as it has data structures that actually can contain other data structures as values. This became the biggest obstacle in Bash.

How to represent a graph containing structures in Bash? How to represent a double linked list? How to represent a structure containing an array of structures? The fight began, and the stage was set.

I settled on { } to represent nesting – the notion of separating a sub-parser settings from the parent parser. I also settled that there will be no sub calls, just a single API entry L_argparse. This is a design decision and it is not great. Passing improper arguments to L_argparse often results in very gibberish error messages to the developer. You do not know where you did the mistake.

The nested structures could be represented with nested declare -p calls. The slowness of declare became much apparent with sub-parsers. The idea was that the child parser could inherit some of the state from the parent parser. This was really hard, as to access the parent you have to load it with declare, save the information, then load the child with declare and assign result.

With this representation there was no way to “go back” and visit the parent parser. The parent parser was needed to print sensible help messages. Thus, there was a chain of visited parser stored and each time a sub-parser was loaded it was added to the chain, so you could navigate “up” in the stack of parsers.

Then it hit me.

Flatten out structures

There is no need for associative arrays to represent an array of structures. The structure members can be “flatten out” with a prefix and each variable can be an array. This resulted in many local variables.

Now the _L_opt structures is represented as an array of multiple variables. Effectively _L_opt[3].help becomes _L_opt_help[3]. This requires some constraint and thinking while writing the code, but the code is fast. To navigate or iterate over options becomes really simple, just keeping the number of options in a separate variable. Any option can be accessed randomly.

To implement the list, the variable _L_opt__parseri contains the index, integer of the parser that owns this option. Similarly, parser have _L_parser__parenti that contains the index of the parser above it.

This significantly speed up the parsing and variables handling. It allowed starting working on function subparsers implementation.

How to parse list of arguments by multiple functions?

getopts sets the current index of parsing arguments in OPTIND. To parse positional arguments, user code has to shift $((OPTIND-1)).

I found that to be ineffective when dealing with multiple nested functions. It is simpler instead of OPTIND, which you have to track, just return the number of arguments left to parse.

some_parser() {
   while (($#)); do
     ...
     shift
   done
   _L_g_args_left=$#
}
call_some_parser() {
   local _L_g_args_left
   some_parser
   shift $(($#-_L_g_args_left))
   parse_rest_of_args "$@"
}
call_some_parser "$@"

This just sets one variable after the parsing. The code does not have to keep the index of current variable, and can use shift on any depth. Just a single variable to set.

However, this turned out to be highly ineffective.

I profiled the Bash script, and it turns out creating positional arguments "$@" and shift were hot paths in the code. They tool the longest time, as reported by timestamps in DEBUG trap or by set -x lines. The act of calling a func "$@" with a lot of arguments was taking a lot of time, and a simple shift $a_lot was also very slow.

When parsing arguments by multiple nested functions, it is much more efficient to use a global array to assign the arguments, and use an index variable to keep the current position.

some_parser() {
   for ((; argi < ${#args[@]}; argi++)); do
      ...
   done
}
call_some_parser() {
   local args=("$@") argsi=0
   some_parser
   parse_rest_of_args
}
call_some_parser "$@"

This code is confusing – it uses global state. Global state as far as the chain of functions is concerned. This global state clusters the mind and slows the programming. However, it is fast for Bash. I suspect the act of expanding "$@" with a lot of arguments just uses a lot of allocations.

Shell Completion

Shell completion bottom line comes to interprocess communication between the current user shell and the completion program.

When executed L_argparse specification... ---- --L_argparse_get_completion args... instead of executing anything the program outputs a list of elements to complete given arguments. Then this list is parsed by Fish or Zsh or Bash shell to properly out fill out completion information.

The outputted list consists of lines separated by newlines. Each line consists_is_correct of 3 elements separated by byte number \35 – group separator byte in ASCII table. These lines are then parsed by the shell.

This causes bottom line the parser to go two ways. An invalid option results in program termination with an error message. However, during completion, an invalid option results in skipping the option until the last option is found.

Additionally, this presented really hard challenge – completion of subparser functions.

Subparser functions

The Python argparse module allows for add_subparsers subparsers implementation. When a specific word is given as an argument, specific subparser is chosen for parsing. This was implemented with an array of parsers and an additional parser field that registers the child<->parent structure between parsers.

After consideration, I have chosen { } to signify parsers in the L_argparse parsing specification.

L_argparse \
  { \
     subparser1 \
     -- --option-of-subparser1 \
  } \
  { \
     subparser2 \
     -- --option-of-subaprser2 \
  } \
  ---- "$@"

In case of subparser “given” by the user in the subparsing specification there is no case of code execution. However, what I really wanted is to store the subparser in normal written separate functions.

CMD_subparser1() {
   L_argparse -- --option-of-subparser1 ---- "$@"
}
CMD_subparser2() {
   L_argparse -- --option-of-subparser2 ---- "$@"
}
CMD_invalid() {
   echo "will this execute on code completion?"
}
L_argparse \
   -- call=function prefix=CMD_ \
   ---- "$@"

The above code defines an L_argparse parser with 3 subparser defined then in functions. This is simple to do when parsing arguments. However on code completion, the CMD_invalid function may not be executed, as it is only code completion not real code execution.

On code completion, the subparser function is executed with context preserved by upper L_argparse execution. That way they “know” they are inside code completion routine and call exit 0 before executing user code. However, CMD_invalid above does not call L_argparse and as a library writer, I cannot do anything about it.

To battle such cases, a property of function subparser was introduced subcall. When subcall=1 the call to the subparser function is always made during completion, and we trust every single CMD_* function starts with a L_argparse call on top of it.

However, with subcall=auto, the code “detects” if the function starts with L_argparse with introspection. It is fairly easy to get the function definition text from bash just by calling declare -F func. Before calling the function during shell argument completion, first the function definition is obtained, and it is checked against a regex if the function makes a call to L_argparse. This is not a great solution, and it does prevent many designs, however it worked “good enough” for most common cases I worked with.

Subparser function context

There is some magic in the local Bash built-in call. It allows creating nested “global” variables. In contrast to other programming languages, the local variables are local to the function and all child functions that happen to be called. This means that you can temporary “hide” global value.

func2() {
   local LC_ALL=C
   echo "IFS=$IFS LC_ALL=$LC_ALL"
}
func1() {
   local IFS=' '
   func2
}

The “hiding” features open a great can of possibilities and abuse. You can create a stack of same variables with different values. These variables will be auto-cleaned up by Bash upon all function returns.

The “nested” L_argparse call in the CMD_subparser1 above “knows” about the parser definition of the main parser. The _L_parser_* and _L_opt_* and other L_argparse variable values are preserved. This way, when calling example subparser1 --help the subparser “knows” it is being chained in the list and properly reports the usage message as example subparser1 [---options...].

Subprarsers also inherit some properties from parent parsers, they can do that because the _L_parser_* variables have not been cleared. The call to CMD_subparser1 is done, and has to be done for this feature as implemented to work, inside `L_argparse()~ function call. This nests to any kind of nested subparsers.

Conclusion

The journey of creating a Bash argparse-ish parsing tool has been long and arduous. With multiple implementations and setbacks, it has been a challenge to create a tool that is both fast and reliable. However, the end result has been worth it.

The current implementation uses a novel approach to represent nested structures, using a combination of arrays and variables to create a flexible and efficient parsing system.

The tool has been tested extensively and has proven to be robust and reliable. It has also been used in real-world projects, where it has performed admirably.

While there is still room for improvement, the current implementation is a significant improvement over previous versions. It is fast, reliable, and easy to use.

In the future, I plan to continue working on this tool, adding new features and improving its performance. I am excited to see where this journey will take us.

In conclusion, the Bash L_argparse parsing tool is a powerful and flexible tool that has been a game-changer for command line argument parsing. I am proud of what we have accomplished and look forward to continuing to improve and expand this tool.

Leave a Comment

Scroll to Top