Using Snakemake on Esrum

This page describes how to best use Snakemake on the Esrum cluster. As this includes a number of suggested settings (described below), a basic profile file is provided below to automatically set these.

Snakemake can either be run directly, where all rules (i.e. tasks) are run on the same system as Snakemake itself, or Snakemake can be configured to use Slurm to run the individual rules, allowing these to be run on any compute node on Esrum. The choice between the two options boils down to the following considerations:

  • If the steps in your Snakemake pipeline are very short, then you should run your pipeline in a regular sbatch script, via srun, or in an interactive session. This is because Slurm adds some overhead to jobs, which for very short rules may result in a significant increase in the total runtime.

  • If your steps run for a longer time or take up significant amount of resources, then you should enable Slurm support when running your pipeline. This ensures that you only reserve resources for your rules while they are running and enables you to run more rules simultaneously than can fit on one compute node.

  • If any of your rules make use of GPUs, then you must enable Slurm support when running Snakemake. This ensures that GPUs are only reserved while they are actively being used, which we require since GPUs are a very limited resource on Esrum. See below for how to reserve GPUs for your rules.

To summarize:

  • Short jobs, few resources: Run Snakemake in a Slurm job.

  • Longer jobs, more resources, or GPUs: Run Snakemake with Slurm support enabled.

For most bioinformatics pipelines, the most efficient choice is to run Snakemake with Slurm support enabled.

Running Snakemake with Slurm support

How you enable Slurm support in Snakemake depends on the version of Snakemake that you use. You can check this via snakemake --version, if you are unsure:

$ snakemake --version
9.6.0

The following two subsections describe how to use Slurm depending on the version of Snakemake you use. For the above example, you would follow the instructions in the second subsection.

Note also that you must run Snakemake on the head node when using the --slurm option. This is required for Snakemake to be able to interact with Slurm. Furthermore, you should be running it a tmux or screen session to ensure that Snakemake keeps running after you log out. See the Persistent sessions with tmux page for more information.

Warning

Some older tutorials may suggest setting Slurm options via --cluster or similar command-line option. These tutorials are outdated, and should not be followed.

Snakemake version 7 or older

To run Snakemake 7 or older with Slurm support enabled, simply pass the options --slurm and --jobs N, where the N is the maximum number of jobs you want to queue simultaneously. For example,

$ module load snakemake/7.30.1
$ snakemake --slurm --jobs 32

This command will run your pipeline via Slurm and queue at most 32 jobs at once. Note that we do not need to specify the maximum number of CPUs (via --cores), since Slurm will take care that (see below).

See the Snakemake profile section below describes how to set these settings automatically, as well as other useful settings.

Snakemake version 8 or newer

To run Snakemake 8 or newer with Slurm support, you must pass the options --executor slurm, --default-resources, and --jobs N, where the N is the maximum number of jobs you want to queue simultaneously. For example,

$ module load snakemake/9.9.0
$ snakemake snakemake --executor slurm --default-resources --jobs 32

The --default-resources option ensures that Snakemake calculates resource requirements automatically, for tasks where you have not specified this yourself.

See the Snakemake profile section below describes how to set these settings automatically, as well as other useful settings.

Tip

If you install Snakemake yourself, then you need to also install the snakemake-executor-plugin-slurm plugin. For more information, see the Snakemake slurm plugin page.

Requesting CPUs

Snakemake will automatically request a number of CPUs corresponding to the number of threads used by a rule:

1rule my_rule:
2    input: ...
3    output: ...
4    threads: 8

Snakemake will in other words reserve 8 CPUs for the above rule when submitting it through Slurm.

Requesting memory

Snakemake will by default estimate the amount of memory needed for a rule based on the size of the input data (max(2*input.size_mb, 1000)), which translates to two times the size of the input but no less than 1000 MB.

This is, however, frequently less than the Slurm default of ~16 GB per CPU reserved, and we therefore recommend overriding this default using the --default-resources option:

$ snakemake --default-resources mem_mb_per_cpu=15948

This corresponds to the behavior of sbatch and srun.

Should a job require more memory than the default ~16 GB per CPU, then you can request additional memory using the resources section of your rule:

1rule my_rule:
2    input: ...
3    output: ...
4    resources:
5        mem_mb: 64 * 1024

The mem_mb specifies a total amount of memory to reserve in MB and the above example therefore requests 64 GB for this specific rule.

Using the GPU / high-memory nodes

Running a job on the GPU / high-memory nodes is accomplished by specifying that you want to use the gpuqueue by adding slurm_partition="gpuqueue" to the resources section of your rule. Once you have done so, you can reserve GPUs using the slurm_extra resource:

1rule gpu_example:
2    input: "my_input.dat"
3    output: "my_output.dat"
4    shell: "my-command {input} > {output}"
5    resources:
6        # Run this rule on the GPU queue
7        slurm_partition="gpuqueue",
8        # Reserve 1 GPU for this job
9        slurm_extra="--gres=gpu:1",

If you need memory rather than GPUs, then omit the slurm_extra resource and instead specify the amount of RAM needed in MB, using the mem_mb resource as described above:

1rule high_mem_example:
2    input: "my_input.dat"
3    output: "my_output.dat"
4    shell: "my-command {input} > {output}"
5    resources:
6        # Run this rule on the GPU queue
7        slurm_partition="gpuqueue",
8        # Reserve 3 TB of memory (specified in MB)
9        mem_mb=3 * 1024 * 1024,

Warning

Do not reserve GPUs if you do not need to use them; we only have a few GPUs, so we will terminate jobs found to be unnecessarily reserving GPU resources.

Using environment modules

Snakemake can automatically load environment required by a rule. This requires either that the --use-envmodules option is specified on the command-line or that use-envmodules is set to true in your profile (see below). When that is done, Snakemake will automatically load the environment modules listed in the envmodules section of a rule:

1rule my_rule:
2    input: "my_input.bam"
3    output: "my_output.stats.txt"
4    shell: "samtools stats {input} > {output}"
5    envmodules:
6        "libdeflate/1.18",
7        "samtools-libdeflate/1.18",

Tip

Remember to specify version numbers for the module you are using; this helps ensures that your analyses are reproducible and that they won't suddenly break when new versions of modules are added.

Snakemake profile

Snakemake settings can be set automatically by using profiles. The following profile sets the options recommended above and the settings required to enable Slurm. Make sure to remove the options that do not correspond to your version of Snakemake:

 1# Maximum number of jobs to queue at once
 2jobs: 32
 3
 4# FIXME: Remove this line if you DO NOT use Snakemake 7 or older:
 5slurm: true # Enable slurm
 6
 7# FIXME: Remove these t wo lines if you DO NOT use Snakemake 8 or newer:
 8executor: slurm # Enable slurm
 9default-resources: true # Calculate resource requirements
10
11# (Optional) Enable the use of environmental modules
12use-envmodules: true
13# Wait up to 60 seconds for the network file system
14latency-wait: 60
15# Re-run incomplete jobs
16rerun-incomplete: True
17
18# Standard slurm resources; these match the `sbatch` defaults:
19default-resources:
20  # Use standard queue by default (silences warning)
21  - "slurm_partition=standardqueue"
22  # Same mem-per-CPU as Slurm defaults
23  - "mem_mb_per_cpu=15948"
24  # (Optional) Runtime limit in minutes to catch jobs that hang
25  #- "runtime=720"

To use this profile, save it as config.yaml in a folder, for example in your project folder. Then run Snakemake with the --profile option pointing to the folder in which you saved your profile:

$ snakemake --profile /path/to/profile/

To simplify using Snakemake, this profile is available at /projects/cbmr_shared/apps/config/, for different versions of Snakemake:

Snakemake

Path

Version 7

/projects/cbmr_shared/apps/config/snakemake/7

Version 8

/projects/cbmr_shared/apps/config/snakemake/8

Version 9

/projects/cbmr_shared/apps/config/snakemake/9

For example, to use the profile for Snakemake 9:

$ module load snakemake/9.9.0
$ snakemake --profile /projects/cbmr_shared/apps/config/snakemake/9

Options specified in this profile can be overridden on the command-line simply by specifying the option again:

$ snakemake --profile /projects/cbmr_shared/apps/config/snakemake/9 --jobs 16

Troubleshooting

sacct: error: Problem talking to the database: Connection refused

If you are running Snakemake with the --slurm option on a compute node, i.e. not the head node, then you will receive errors such as the following:

Job 0 has been submitted with SLURM jobid 512921 (log: .snakemake/slurm_logs/rule_foo/512921.log).
The job status query failed with command: sacct -X --parsable2 --noheader --format=JobIdRaw,State --name 2d898259-73e4-435d-aa77-44dc44d84c1b
Error message: sacct: error: slurm_persist_conn_open_without_init: failed to open persistent connection to host:localhost:6819: Connection refused
sacct: error: Sending PersistInit msg: Connection refused
sacct: error: Problem talking to the database: Connection refused

To solve this, simply start your Snakemake pipeline on the head node when using the --slurm option.