Skip to content

What is Module?#

module or more exactly Environment Modules (http://modules.sourceforge.net/) is a working environment manager that allows to choose which package//module is loaded in the user environment. It's features allows to:

  • display the whole package//module available on the cluster.
  • display the package//module collection loaded by the user.
  • load or unload a package in the working environment.
  • create modules specific for an user or a team.
  • create modules for specific projects.

module provides you a list of module//packages available on the cluster which you can choose the one you want to use.
Let's say you can view the module//package as Plug In that you load, remove for your working environment regarding your needs.

At the date this page was created, more than 1000 modulefiles covering 553 bio-informatics packages and 112 development tools are available. this list will still grow

useful module commands#

module av#

module av will display the whole package//module available on the cluster.

example:

Code Block (text)

maestro:~ > module av 

----------------------------- /opt/gensoft/modules -----------------------------
abricate/1.0.1(default)               mfold_util/4.6(default)                  
AdmixTools/6.0(default)               MHC_I/3.1.1(default)                     
ALE/1.0(default)                      micca/1.7.2                              
AlienRemover/1.0(default)             minced/0.4.2(default)                    
AlienTrimmer/0.4.0                    minimap2/2.17(default)                   
AlienTrimmer/2.0(default)             mlst/2.19.0(default)                     
amap/2.2(default)                     MMseqs2/11-e1a1c                         
angsd/0.935(default)                  MMseqs2/12-113e3                         
annovar/24.10.2019(default)           MMseqs2/13-45111(default)                
antismash/5.1.1(default)              mob-suite/3.0.1(default)                 
ANTs/2.3.5(default)                   mothur/1.44.3(default)                   
anvio/7(default)                      MotionCor2/1.4.0(default)                
any2fasta/0.4.2(default)              mrbayes/3.2.7(default)                   
aragorn/1.2.38(default)               msa/2.1(default)                         
augustus/3.3.3(default)               msaprobs/1.0.5(default)                  
autodock_vina/1.1.2(default)          MToolBox/1.2.1(default)                  
BamM/1.7.3(default)                   MultiQC/1.8                              
BAMStats/1.25(default)                MultiQC/1.9(default)                     
Bandage/0.8.1(default)                MUMmer/3.23(default)                     
barrnap/0.9(default)                  MUMmer/4.0.0rc1                          
..... etc etc

Using the previous command you will get the list of ALL available modules.
This list displays various information /
package_name is the name of the corresponding module//package
version will teach you on the package version
the default field (default) will be covered later.

e.g.: blast+/2.10.1 is the NCBI's blast package version 2.2.21

NB : this command takes quite a long time to run (we are currently working on this to improve it's reactivity) and prints it's results via PAGER program (usualy less)

module av allows you to search for specific modulefiles using a word.
Warning this word correspond to the beginning of the module//package and is case insensitive.

For example module av bla will list all the modules whose name STARTS WITH bla.

Code Block (text)

 maestro:~ > module av bla 

----------------------------- /opt/gensoft/modules -----------------------------
blast+/2.2.31  blast+/2.10.1(default)  blastTaxoAnalysis/1.0(default)  
blast+/2.10.0  blast/2.2.26(default)   blat/36(default)                

--------------------------- /opt/gensoft/devmodules ----------------------------
BLAS/3.8.0

NB: /opt/gensoft/modules holds general scientific tools while /opt/gensoft/devmodules holds development tools, ie: compiler, interpreters, libraries and so on...

module load#

The command to use in order to load a module in your working environment is module load <package name> , or in order to load a specific version of the package module load <package_name>/<version> .

If you don't specify the version number, module will automatically load the package//module tagged (default)

example:

Code Block (text)

maestro:~ > module av blast+ 

----------------------------- /opt/gensoft/modules -----------------------------
blast+/2.2.31  blast+/2.10.1(default)  blast+/2.10.0
  • if you issue the command module load blast+ without any versioning, you will load the default module i.e. blast+/2.10.21
  • if you issue the command module load blast+/2.10.1 you will explicitly load blast/2.10.1
  • if you issue the command module load blast+/2.2.31 you will explicitly load blast/2.2.31

NB we recommend using versioning as the default may evolve over the time
NB even if it's possible to load various version of the same module//package in the same time, we STRONGLY RECOMMEND to not
NB module load accept more than one argument, e.g. module load blast/2.2.26 html4blast msa/2.1 will load both modules at the same time.

module list#

In order to know which packages//modules are currently loaded in your working environment issue the command module list.

example:

Code Block (text)

maestro:~ > module list 
Currently Loaded Modulefiles:
  1) blast/2.2.26       2) html4blast/1.7a       3) msa/2.1

we see that this user has loaded the following packages//modules blast,  html4blast andmsa with respective version numbers 2.2.26, 1.7a and 2.1

module show#

module show <package name> command allows you to see all modifications that will be doone when the module is loadded.
It is usefull to check if any environement variable is set by the module that you can use in command line ou thru your scripts

example
maestro:~ > module show blast ------------------------------------------------------------------- /opt/gensoft/modules/blast/2``.2.26:   module-whatis    Set environnement for blast (2.2.26) module-whatis    Categorie: EDAM-0002421 module-whatis    Categorie: EDAM-0002422 module-whatis    Categorie: EDAM-0000338 module-whatis    Categorie: EDAM-0000239 module-whatis    Categorie: EDAM-0000346 module-whatis    Categorie: EDAM-0000491 module-whatis    Categorie: EDAM-0000496 module-whatis    Categorie: EDAM-0003096 prepend-path     PATH /opt/gensoft/exe/blast/2``.2.26``/bin prepend-path     PATH /opt/gensoft/exe/blast/2``.2.26``/scripts prepend-path     MANPATH /opt/gensoft/exe/blast/2``.2.26``/man setenv       NCBI /opt/gensoft/exe/ncbitools/20122006/share setenv       BLASTDB /local/databases/index/blast2 -------------------------------------------------------------------

we can see that once blast/2.2.26 module is loaded:

  • PATH will be modified and will point to 2 additional locations:  /opt/gensoft/exe/blast/2.2.26/bin and /opt/gensoft/exe/blast/2.2.26/scripts
  • 2 environnement variables will be set  NCBI et BLASTDB that can be used in command line or thru you scripts

module unload#

To remove a module//package from your working environment the command to use is: module unload <package name>. It is not necessary to tell the version of the package to remove.

example

Code Block (text)

maestro:~ > module list 
Currently Loaded Modulefiles:
  1) blast/2.2.26       2) html4blast/1.7a    3) msa/2.1

maestro:~ > module unload html4blast

maestro:~ > module list 
Currently Loaded Modulefiles:
  1) blast/2.2.26       2) msa/2.1

module purge#

module purge is a super module unload, ie : this command remove all previously loaded modules.

module use#

The module use command allows you to add one or more modulefiles directory to the module searching PATH Same may be accomplished by setting MODULEPATH envrionment varaible.

By default module use prepends the given directories to MODULEPATH
You can change this behaviour, i.e. append the given directory to MODULEPATH using the -a or --append. option.

example:

Code Block (text)

maestro:~ > echo $MODULEPATH
/opt/gensoft/adm/Modules/4.4.0/modulefiles:/opt/gensoft/modules:/opt/gensoft/devmodules
maestro:~ > module use ~/privatemodules
maestro:~ > echo $MODULEPATH
/pasteur/appa/homes/edeveaud/privatemodules:/opt/gensoft/adm/Modules/4.4.0/modulefiles:/opt/gensoft/modules:/opt/gensoft/devmodules

module unuse.#

remove specified directories from the MODULEPATH environment variable.

module help#

module help <package name> command will display some information on the given module//package.
such as (when available)

  • brief description.
  • home page of the package.
  • availability of locally installed documentation
  • the list of commands//tools provided by the package.
  • dependencies on other module(s), modules requested to be loaded with the package.

example:

Code Block (text)

maestro:~ > module help ptools

----------- Module Specific Help for 'ptools/0.99d' ----------------

This modulefile defines the requisite environment
needed to use package: ptools version (0.99d)

Parallel versions of various tools

WARNING: ptools relies at least on one of the module(s) < blast or bwa or bowtie2 or taxoptimizer or cutadapt/1.9.1 or igblast or GenomeAnalysisTK >. Load them accordingly to your needs

package provides following commands:
        mGenomeAnalysisTK
        mblastall
        mbowtie2
        mbwa
        mcutadapt
        migblastn
        migblastp
        mtaxoptimizer

module display#

module display <package name> will print the changes to be performed when module will be loaded.

example:

Code Block (text)

maestro:~ > module display toppred
-------------------------------------------------------------------
/opt/gensoft/modules/toppred/1.10:

module-whatis    Set environnement for toppred (1.10) 
module-whatis    Categorie: EDAM-0000269 
module-whatis    Categorie: EDAM-0000270 
module-whatis    Categorie: EDAM-0002241 
prepend-path     PATH /opt/gensoft/exe/toppred/1.10/bin 
prepend-path     PATH /opt/gensoft/exe/toppred/1.10/scripts 
prepend-path     MANPATH /opt/gensoft/exe/toppred/1.10/man 
prepend-path     INFOPATH /opt/gensoft/exe/toppred/1.10/man 
setenv           TOPPREDDATA /opt/gensoft/exe/toppred/1.10/share/toppred 
-------------------------------------------------------------------

module whatis.#

module whatis <package name> prints a brief description of the package//module and the EDAM categories this package//module belongs to.

module apropos#

module apropos <string> will print the modules that the whatis information (as displayed by the module whatis) match the

module apropos allows you to search all modules//packages that match a given string in the description, EDAM categories and so on.

Automatic module load at logging.#

If you want to automatically load some useful modules, just put the corresponding module load commands in you init shell files, as ~/.bashrc, ~/.zshrc, ~/.cshrc, regarding you favorite shell.

here is an example from an user ~/.bashrc:

Code Block (text)

#---- alias

alias ls='ls -F'
alias ll='ls -Fl'
alias m='more'
alias l='less -aiMQ'

#---- modules load at login

module load R
module load boost
module load gsl/1.15

This user wil have default (as defined by the module system) R and library boost loaded plus gsl version 1.15 loaded each time it will log on the cluster

Write and use your own modulefiles#

Why would I want to do that ?#

Let assume you wrote a pipeline that need some specific version for various tools. assume that pipeline need to have environment variables and aliases to be set and so on. If you want to share the pipeline with collaborators, take the easy way for setting up the environment for this pipeline use a modulefile that controls and set up everything.

Another study case wil be that you have to deal with different projects that use different tools version, different data location. Guess what? specific modulefiles for each project will allows to fine tune and control the behaviour for each project.

How to write a modulefile#

Modulefiles are written in TCL. But you don't need to be a TCL hacker to be able to write your own modulefiles.

example:

Code Block (text)

#%Module1.0 

proc ModulesHelp { } {
   puts stderr "\texample modlefile" 
}

module-whatis "basic modulefile" 

# define ENV_VAR1
setenv ENV_VAR1 needed by the project

# add path to some project homebrew tools.
preprend-path PATH /quelque/part/par/la
preprend-path MANPATH /quelque/part/par/ici

# define alias my_cmd
set-alias my_cmd "cmd -p 25 -t N" 

# load specific project modules tools
module load foo/1.0 bond/007

Some syntax explanation.

  • ModulesHelp: the function that controls what module help moddulefile. will display
  • module-whatis: this will displayed when module whatis moddulefile. is called
  • setenv: setenv var value set the environment variable var to the the given value
  • prepend-path: prepend-path var value prepend environment variable var of type PATH with the given value.
  • append-path: append-path var value append environment variable var of type PATH with the given value.
  • set-alias: set-alias name value défine alias name with value

For a detailed description of the modulefile syntax, see: *man modulefile*

NB: all modulefiles available on the cluster are located to /local/gensoft2/modules you can check them as a basis for your own ones.

how to use your modulefiles#

Once they are written just create the following directory ~/privatemodules and place your modulfiles to it.
next after loading the module use.own: module load use.own you will have access to your modulefiles.
Warning issuing module purge command will remove the module use.own, and thus will remove your ability to access the modulefiles from ~/privatemodules as long as you have not loaded module use.own again.
Alternatively you can use the module use any_path to add the directory hosting your modulefiles to the module search path. (the second will prevent the module purge command to remove your ability to load your own modulefiles.)

Code Block (text)

maestro:~ > echo $MODULEPATH
/opt/gensoft/adm/Modules/4.4.0/modulefiles:/opt/gensoft/modules:/opt/gensoft/devmodules
maestro:~ > module use ~/privatemodules
maestro:~ > echo $MODULEPATH
/pasteur/socnic/homes/edeveaud/privatemodules:/opt/gensoft/adm/Modules/4.4.0/modulefiles:/opt/gensoft/modules:/opt/gensoft/devmodule

NB we recommend using a structured tree to store your modulefiles.

modules and scripting.#

Using module in shell scripts#

Regarding the shells, the logging method, jobs submissions the module function may be defined or not.
To avoid problems it is recommended that your scripts source the module init file for your favorite shell.

maestro cluster:#

on maestro first lines of script should be

Code Block (text)

#!/bin/bash

# get sure module function is defined
# if using another shell, adapt 
source /opt/gensoft/adm/etc/profile.d/modules.sh

#be sure environment is clean
module purge

# load the modules we need for this script
module load cufflinks/2.0.1   tophat/2.0.8 bowtie2/2.1.0

# do the job