6. Automating data analysis

Important

Everything required (software, scripts) is available in the Singularity container. Go to the preparation chapter to see how to acquire this container.

In the previous chapters, you learned how to perform each step of the metagenomic data analysis pipeline manually. While this is a valuable learning experience, it’s not practical for analyzing large datasets or for ensuring reproducibility in the long term.

We can make use of a tool called Snakemake to automate the previous steps into a single pipeline. With Snakemake, you can define the steps of your analysis in a Snakefile and then let Snakemake handle the execution, dependency management, and error handling.

6.1 Preparing to run the workflow

Important

Raw read files must adhere to the naming scheme as described here.

To run the automated workflow, you’ll need to make sure that your project directory is set up correctly.

To make the project setup process even easier, we’ve created a simple command-line tool called prepare_project.py. This tool automates the creation of the project directory, the sample configuration file (sample.tsv), and the general settings configuration file (config.yaml), guiding you through each step with clear prompts and error checking.

The prepare_project.py tool is built into the singularity container image. Instead of using singularity shell, we can use singularity exec to directly execute commands. Try accessing prepare_project.py:

singularity exec imam_workflow.sif python /prepare_project.py --help

usage: prepare_project.py [-h] [-p PROJECT_DIR] -n STUDY_NAME -r RAW_FASTQ_DIR [-t THREADS]

Interactive tool for setting up a Snakemake project.

options:
  -h, --help            show this help message and exit
  -p PROJECT_DIR, --project_dir PROJECT_DIR
                        Project directory path (default: current directory)
  -n STUDY_NAME, --study_name STUDY_NAME
                        Name of the study
  -r RAW_FASTQ_DIR, --raw_fastq_dir RAW_FASTQ_DIR
                        Directory containing raw FASTQ files
  -t THREADS, --threads THREADS
                        Maximum number of threads for the Snakefile (default: 8)

Now prepare your project directory with prepare_project.py as follows:

singularity exec \
  --bind /mnt/viro0002:/mnt/viro0002 \
  --bind $HOME:$HOME \
  --bind $PWD:$PWD \
  imam_workflow.sif \
  python /prepare_project.py \
    -p {project.folder} \
    -n {name} \
    -r {reads} \
    -t {threads}

• {name} is the name of your study, no spaces allowed.
• {project.folder} is your project folder. This is where you run your workflow and store results.
• {reads} is the folder that contains your raw .fastq.gz files.

The --bind arguments are needed to explicitly tell Singularity to mount the necessary host directories into the container. The part before the colon is the path on the host machine that you want to make available. The path after the colon is the path inside the container where the host directory should be mounted.

As a default, Singularity often automatically binds your home directory ($HOME) and the current directory ($PWD). We also explicitly bind /mnt/viro0002 in this example. If your input files (reads, reference, databases) or output project directory reside outside these locations, you MUST add specific --bind /host/path:/container/path options for those locations, otherwise the container won’t be able to find them.

Note

When prepare_project.py prompts for the Reference and DB paths, you must enter the absolute host paths, and these paths must be accessible via one of the bind mounts.

Also, it’ll ask if you want to create a raw_data/ folder with softlinks to your raw fastq.gz files. This is not required for running the workflow, but it can be convenient to have softlinks to your raw data available in your project directory.

Once the setup is completed, move to your newly created project directory with cd, check where you are with pwd.

Next, use the ls command to list the files in the project directory and check if the following files are present: sample.tsv, config.yaml and Snakefile.

• The sample.tsv should have 3 columns: sample (sample name), fq1 and fq2 (paths to raw read files). Please note that samples sequenced by Illumina machines can be ran across different lanes. In such cases, the Illumina software will generate multiple fastq files for each sample that are lane specific (e.g. L001 = Lane 1, etc). So you may end up with a sample.tsv file that contains samples like 1_S1_L001 and 1_S1_L002, even though these are the same sample, just sequenced across different lanes. The snakemake workflow will recognize this behaviour and merge these files together accordingly.

• The config.yaml contains more general information like the indexed reference and database you supplied as well as the amount of default threads to use.

• The Snakefile is the “recipe” for the workflow, describing all the steps we have done by hand, and it is most commonly placed in the root directory of your project (you can open the Snakefile with a text editor and have a look).

6.2 Running the workflow

After setting everything up, we can redo the analysis for all samples in a single step. First we will test out a dry run to see if any errors appear. A dry run will not execute any of the commands but will instead display what would be done. This will help identify any errors in the Snakemake file.

Run inside of your project directory:

singularity exec \
  --bind /mnt/viro0002:/mnt/viro0002 \
  --bind $HOME:$HOME \
  --bind $PWD:$PWD \
  imam_workflow.sif \
  snakemake --snakefile Snakefile \
  --cores {threads} \
  --dryrun

If no errors appear, then remove the --dryrun argument and run it again to fully execute the workflow.