Cluster jobs
Four sbatch templates ship with the package. Copy one out with:
cp "$(python -c 'from importlib.resources import files; print(files("research_helpers")/"templates"/"sweep.sbatch")')" .
Template |
Description |
|---|---|
|
Runs one array task per slice of a planned grid. See parameter sweeps. |
|
Runs one block of a queue file per array element, for many sweeps at once. See many sweeps through few arrays. |
|
Runs one script as a single job. |
|
Diagnostics for when a job cannot find the interpreter. |
Each has an EDIT THIS BLOCK fence around the parts that require configuration—account, partition, module
loads, environment name.
What the templates handle
Thread oversubscription. Every template sets OMP_NUM_THREADS, OPENBLAS_NUM_THREADS,
MKL_NUM_THREADS, and NUMEXPR_NUM_THREADS to the cores the scheduler granted. Without
this, a task granted one core still sees the whole node through BLAS and spawns a thread per core.
The interpreter. They call $CONDA_PREFIX/bin/python by absolute path rather than trusting
PATH. This prevents a common occurrence where after an apparently successful activation the
PATH is wrong and python falls through to the base install leading to failed jobs at the
first third-party import. sweep.sbatch and queue.sbatch share this setup line for line, and a
test keeps the two identical.
Re-queuing. The two array templates set --no-requeue. When a node stops answering, a re-queue
can start a second copy of a task while the first is still running, and both append to the same
part file from two nodes. That corrupts it beyond repair (see
elements that ran twice). A failed task is resubmitted instead,
and resumes from its last recorded combination.
Forcing a re-run. Set SWEEP_NO_RESUME=1 on the sbatch line to recompute combinations
already recorded. After a change to the code or the environment, a resumed task that finds
everything done exits cleanly, exactly like a success.
Single jobs
Used to do work that is not a grid: a diagnostic battery over a whole dataset, a one-off scaling measurement, a build step too large for a login node. Each job has its own arguments and writes its own single output file, so there is nothing for a manifest to describe.
sbatch job.sbatch scripts/evaluate.py --preset full --out runs/full.json
sbatch -t 0-12:00 --mem-per-cpu=32G job.sbatch scripts/evaluate.py --out runs/big.json
Resources are overridden at submit time, so one template covers jobs of any size.
Note
The template also has a commented block for checking the environment before the work starts.
Resubmitting
sweep.sbatch takes a sparse array, and slices come from the manifest rather than from the width of
the array submitted, so resubmitting two killed tasks reproduces exactly the slices they had:
sbatch --array=17,42 sweep.sbatch runs/my-sweep
status prints that range already collapsed. See status.