Debugging Cmd-Queue Powered Scripts
-----------------------------------

When running a cmd-queue based script it is important to understand that it is
primarilly a mechanism for building a sequence of commands to accomplish a
task. The execution of those commands is secondary.

Idomatically, most cmd-queue scripts will have the options: ``--run`` and
``--print_commands``. By setting  ``--run=0`` and  ``--print_commands=1`` it
will simply show you the commands that it would execute in order. You can
manually execute these commands step-by-step to diagnose problems.

Also there is usually a ``--backend`` option. Often this is set to
``--backend=tmux``, but if you use ``--backend=serial`` it will run everything
in the foreground terminal session allowing you to see everything happen one at
a time. This is very similar to running commands one-by-one, but there are
minor differences so its important to be aware of both strategies.

When using the tmux cmd-queue backend, if jobs are failing you can use tmux to
inspect the status of the jobs. Use ``tmux a`` to attach to one of the tmux
sessions, and you can use ``CTRL-b`` followed by ``s`` to show an interactive
overview of all sessions. You can navigate to inspect what each session is
doing. You will also notice that each tmux session has an associate script that
it sources at the very beginning. You can look at that to see the details of
what is happening in each tmux session (Note: there is a lot of bash
boilerplate in these files, and most of it can be ignored: look for the part of
the code with the ``# command`` comment).