Draft:Snakemake

• Comment: Reads like an advertisement. Please demonstrate WP:NCORP in the article and ensure tone is WP:NPOV. microbiologyMarcus ^{(petri dish)} 15:50, 25 October 2023 (UTC)

• Comment: I almost approved it but there are a handful of bits like A cornerstone of science is the need to validate experimental results so that confidence can be gained on the result's correctness and Information gained from science must be shared so that findings can be built upon and utilized, therefore the experiment and results must be comprehensible that should be more neutral. Gusfriend (talk) 12:36, 12 June 2022 (UTC)

• Comment: As a note: if this were to be submitted as-is, I'd expect it to be declined as it reads more like an advertisement than an encyclopedia article. Bsoyka (talk · contribs) 04:19, 25 February 2022 (UTC)

---

Snakemake

Repository	github.com/snakemake/snakemake
Written in	Python, YAML
Platform	Windows, Linux, macOS
License	MIT License
Website	snakemake.github.io

Snakemake is a scientific workflow management framework that uses a domain-specific language (DSL) to specify allowed data transformations for automatic generation of processing pipelines. Subscribing to the declarative paradigm, the usage of Snakemake involves stating desired products instead of an explicit course of action. Individual steps, however, must still be defined as rules in a similar structure to what can be found in GNU Make. These rules are written in a workflow config file called a "Snakefile" using Snakemake's DSL, which allows both Python and YAML syntax since the DSL is implemented as an extension of Python. Snakmake executes the pipeline defined in Snakefiles by selecting and running each step as prerequisites become satisfied. Steps with no interdependence may be executed in parallel.

The author of Snakemake has stated that the goals of this software tool is to enhance reproducibility, transparency, and adaptability in bioinformatic analysis. The following characteristics of Snakemake support these goals: Native integration of external tools like Conda and Singularity borrow their functionalities, namely dependency resolution and containerization. Being platform agnostic allows it to run the same workflow on local or high performance compute (HPC) environments. Reports can be generated to describe and log the conditions and actions performed at each step.

Operation

Example generated workflow from available actions & goal

Rules

The set of actions that Snakemake can take towards generating a desired data product is specified in a file called the Snakefile. These actions are called "rules" and include a name, inputs, outputs, and how to start the represented action. Inputs and outputs are declared as explicit file paths, cloud storage URIs, or sets of locations through patterns and wildcards. Several methods exist to start an action, including a shell command, python script, or Snakemake wrapper. A wrapper is a predefined configuration that specifies how Snakemake can interface with existing software such as Samtools. In a Snakefile, YAML adhering to Snakemake's DSL and native Python is allowed, but upon execution, the rules are translated into Python via a state machine algorithm.

Example rule:

rule user_defined_name:
    input:
        "path/to/input.file"
    output:
        "path/to/output.file"
    shell:
        "shell_command {input} > {output}"

Workflows and Execution

When given a set of rules and target outputs, Snakemake will first either look for files that match the targets or search for defined rules that produce them. By then redefining the targets to the inputs of found rules and repeating recursively until all inputs are existing files, an implicit workflow is generated in the form of a directed acycic graph (DAG) of steps. The execution order of these steps is computed by optimizing a mixed integer linear program for various parameters such as runtime, total disk space required for temporary files, and user defined priority. Where possible, intermediate data product from previous runs can be used to skip ahead if prerequisite steps are found to be equivalent. Equivalence is determined by comparing the prior steps used to generate the data, recorded through a ledger system analogous to that of block chains.

Features

Example Snakefile and tool wrapper reuse

Reproducibility

Reproducibility refers to the level of determinism exhibited by a piece of software, that is, how similar are the outputs of different executions when processing the same input each time. Snakemake can use Conda or containerization via Docker or Singularity to reconstruct the compute environment with required software dependencies between executions on different machines. Reports can also be generated to summarize the various parameters and events at each processing step, including an image of the computed DAG.

Transparency

Transparency refers to the interpretability of a software's execution or how well the software reports its actions. In addition to the report generation mentioned above, Snakefiles contain more commonly understood terms rather than technical or Snakemake-specific language. This supports transparency by reducing the amount of domain knowledge required to understand what a pipeline does and what the results mean.

Reusability

Previously written Snakefiles can be referenced in new Snakefiles, thus allowing rules to be reused. Certain predefined rules, called tool wrappers, integrate external software for workflow generation and are publicly available.

Comparisons

Snakemake belongs to a family of implicit convention frameworks, members of which include Make and Nextflow. The common characteristic of this group of tools is the ability to generate specific workflows from a collection of discrete steps. Specifically, the GNU derivative of Make used for building source code is considered to be the source of inspiration for Snakemake and contributes to its model of operation and certain features like wildcards. Usage-wise, Snakemake uses text-based configuration instead of a GUI, like that of Galaxy, and attempts to be environment agnostic.

Usage Examples

Bioinformatics

• SnakeChunks is a set of Snakemake rules used for processing Next-Generation Sequencing data, including quality control, peak calling, and visualization steps. The authors demonstrated its use on transcriptome (RNA-seq) and genome wide location (ChIP-seq) data.
• Recount3 is a database of human and mouse RNA-seq data that was curated and annotated with a Snakemake based workflow.
• VIPER is a snakemake based pipeline for producing graphical reports of RNA-seq data.
• RiboDoc is a ribosomal profiling package that also involves complex steps of quality control and data-intensive statistical analysis managed by Snakemake and Docker.
• Dadasnake integrates amplicon sequencing analysis with DADA2 using Snakemake. The implied workflow is for phylogenetic analysis of specific genes.
• In order to compile timelapses of selective plane illumination microscopy images of cells, Snakemake was used to parallelize the task on high performanc compute clusters.

Other

• HEXME is a dataset of tetrahedral meshes generated from CAD models, using Snakemake. Rules were specified to generate three variants for each input.