Singularity¶

Singularity is as far as I can tell similar to Docker and the idea is sometimes you want to have software working on a machine (e.g. a computer cluster) but it is difficult to install it natively on the cluster for whatever reason (e.g. no admin rights, clashing compilers/OS/libraries etc.) The idea here is you build the software package on a machine where you have the administrator rights as some image/container, then send and load that image/container to some production machine so that the image/container can piggyback on the available resources there.

Of course it doesn’t have to be just used this way, e.g. you can make a singularity container (with a Linux OS) within a Mac and then use it on the same Mac, which bypasses some potentially annoying problems Macs have with compilers.

See the Singularity website for the manual.

Key singularity commands used¶

[sudo] singularity build [--sandbox]

build a sandbox, build an image from a sandbox, or build an image from a definition file

[sudo] singularity shell [--writable] [--writable-tmpfs] [--overlay IMAGE]

shell into the contained (image and/or sandbox), options respectively are

persistent write in sandbox/image itself
non-persistent write in sandbox/image itself
persistent write but in an overlay image (created separately)

[sudo] singularity exec [--writable] [--writable-tmpfs] [--overlay IMAGE]

run the executable build in container, with write privileges accordingly if required

Installing singularity¶

Things needed really are a computer with admin/root privileges, and some familiarity with shell commands. On my laptop I run Linux Ubuntu 20.04 LTS, and doing uname -r tells me that my kernel version is 5.4.0-40-generic, so I’m going to do the following with Singularity ver. 3 (ver. 2 is old), specifically with version 3.5.2.

Warning

The kernel version is important because it seems Singularity versions prior to and including 3.5.0 do not play with the 5.4 kernels because of some deflation problem (e.g. see here). When trying to do something like singularity build (see later) the error you get is something like:

kernel reported a bad superblock for squashfs image partition
possible causes are that your kernel doesn't support the compression algorithm or the image is corrupted

Just use a newer version to fix this.

Installing singularity can be done through apt or yum or equivalent, though you might be limited by what versions are available. To build you own follow instructions on here (you will need to also install the language Go). In my case I effectively copy and pasted the code here which gets me Singularity ver. 3.5.2.

Making the image¶

Once you have singularity the easiest (but by no means the most reproducible) way to do things is use the --sandbox option with the build command:

sudo singularity build --sandbox sandbox_name docker://ubuntu:20.04

which downloads in this case ubuntu 20.04 from docker to be used as the container operating system, which will be within a folder called sandbox_name in whatever directory you triggered the command from (e.g. ./sandbox_name/usr/bin would be where the container binaries sit). Other options available may be found here.

Then you can go into the container by

sudo singularity shell --writable sandbox_name

The --writable command allows you to modify the things in the container as if you are whatever OS you decided to have (in this case Ubuntu 20.04 because I did docker://ubuntu:20.04 in the build command). *It is highly recommended you do this as ``sudo`` or ``root``* (skip below warning if you don’t care why).

Warning

First reason to do it as sudo is that to update the libraries and packages in the container presumably you would want to use package managers like apt (e.g. apt update && apt install python2.7), which can’t be done without sudo because the follow up dpkg command needs root privileges.

Second is more subtle and should be fixable but I haven’t bothered to figure out how to fix it. If you don’t use sudo then when you shell in, you end up exactly where you triggered the command from (e.g. pwd will tell you something like /home/jclmak/singularity_images/sandbox_name, compared to doing it with sudo, where pwd should then be in /root). If you do the former and start installing things then $PATH and other things you install will most likely have the wrong PATH when you build it into an image later to send somewhere else. For example:

### without sudo ###
(ubuntu)      singularity shell --writable sandbox_name/
(singularity) pwd
              >> /home/jclmak/singularity_images/sandbox_name

# install Firedrake into what should be the container /home/firedrake
(singularity) cd home/ && python3 firedrake-install
              >> (whole load of stuff) DONE

(singularity) ls /home/jclmak/singularity_images/sandbox_name/home  # because we can see the host folders
              >> firedrake

# build image
(singularity) exit
(ubuntu)      sudo singularity build testing.sif sandbox_name/
              >> (builds sandbox into an image) DONE

# send image
(ubuntu)      rsync -arv testing.sif jclmak@hpc3.ust.hk:~/
              >> DONE
(ubuntu)      ssh jclmak@hpc3.ust.hk   # log onto external machine
(hpc3)        module load singularity  # load the singularity module

# look for where firedrake should have been installed in the container
(hpc3)        singularity exec testing.sif ls /home
              >> NOTHING
# because it actually got installed in /home/jclmak/singularity_images/sandbox_name/home which doesn't exist on the remote machine

### with sudo ###
(ubuntu)      sudo singularity shell --writable sandbox_name/
(singularity) pwd
              >> /root

# install Firedrake into what should be the container /home/firedrake
(singularity) cd ../home/ && python3 firedrake-install
              >> (whole load of stuff) DONE

(singularity) ls /home/jclmak/singularity_images/sandbox_name/home
              >> NOTHING (because folder does not exist in the container)
(singularity) ls /home/
              >> firedrake

# build image
(singularity) exit
(ubuntu)      sudo singularity build testing.sif sandbox_name/
              >> (builds sandbox into an image) DONE

# send image
(ubuntu)      rsync -arv testing.sif jclmak@hpc3.ust.hk:~/
              >> DONE
(ubuntu)      ssh jclmak@hpc3.ust.hk   # log onto external machine
(hpc3)        module load singularity  # load the singularity module

# look for where firedrake should have been installed in the container
(hpc3)        singularity exec testing.sif ls /home/
              >> firedrake

Doing it without sudo seems to let it work within the machine where the container was created (because the folders are there as part of the sandbox on the creation machine), but doesn’t work when you send the container elsewhere because it can no longer find the files in the sandbox.

Once you are in the shell (as sudo probably) you can do whatever you might normally do. For example doing apt update && apt install python3.8 installs python3 (a link of python3.8) into /usr/bin.

Using the image¶

Then you can trigger what you installed by (as a sandbox here)

(ubuntu)      sudo singularity shell --writable sandbox_name/
(singularity) apt update && apt install python3.8
              >> (install some packages) DONE
(singularity) exit
(ubuntu)      singularity exec sandbox_name/ /usr/bin/python3 --version
              >> 3.8.?

/usr/bin should be in $PATH anyway so you could do

(ubuntu)      singularity exec sandbox_name/ which python3
              >> /usr/bin/python3
(ubuntu)      singularity exec sandbox_name/ python3 --version
              >> 3.8.?

To use it elsewhere, build it as an image via

(ubuntu)      sudo singularity build sandbox_image.sif sandbox_name/
               >> DONE
(ubuntu)      singularity exec sandbox_image.sif python3 --version
               >> 3.8.?

The image can be sent to other machines, e.g.

(ubuntu)      rsync -arv sandbox_image.sif jclmak@hpc3.ust.hk:~/
(ubuntu)      ssh jclmak@hpc3.ust.hk
(hpc3)        module load singularity
(hpc3)        singularity exec sandbox_image.sif python3 --version
              >> python 3.8.?
(hpc3)        python3 --version # calling the HPC3 python3
              >> python 3.6.8

If doing something like the above where you are just running an executable, the outputs will be in the host machine and not in the container (because the container should now be an immutable image, and the image has not been specified as --writable). For the Firedrake example below the program wants to write some cache but directly into the image itself, so I have a slight work around there.

Working example 1: Julia¶

Julia is a new computing language that appears to combine the user benefits of Python and computationally optimised languages such as Fortran.

There is already an existing guide for installing Julia as a container. The thing to be aware of is that it may be worthwhile creating an overlay with more space (see next working example) or keeping the sandbox folder lying around to make Julia images with on the creation computer in case more packages need to be downloaded, since the singularity image shouldn’t be written to.

Working example 2: Firedrake¶

Firedrake is a finite element based computational framework with automatic code generation capabilities. High/abstract level code in Python is converted into machine code at the production stage for performance reasons. Parallelised code is automatically generated when more cores are provided at the run/compile stage, without a need to change any of the high/abstract level code.

The following will get Firedrake working in the container (given in --sandbox mode here for the time being):

(ubuntu)      sudo singularity build --sandbox firedrake docker://ubuntu:20.04
(ubuntu)      sudo singularity shell --writable firedrake
(singularity) apt update && apt install curl python2.7 python3.8

# I am going to install it in /home rather than /root because
# /root can't be seen when the container will be run in due course
(singularity) cd /home
(singularity) curl -O https://raw.githubusercontent.com/firedrakeproject/firedrake/master/scripts/firedrake-install
(singularity) python3 firedrake-install --disable-ssh
              >> YES TO INSTALL ALL THE SUGGESTED DEPENDENCIES
              >> TAKES BLOODY AGES (>= 1hr) TO INSTALL (PETSc)
(singularity) ls -lh /home
              >> drwxr-xr-x root root firedrake/

So here the owner and the group of firedrake/ is both root (because I installed it as sudo) and we see that we have 755 access (no write for non-owner).

This will be fine *if* not for the fact that firedrake needs to write to ./firedrake/.cache, so no one but owner of the folder (root) can actually run with the firedrake modules once it’s wrapped into a container because only root can create the .cache folder. The way I got round this to do the lazy thing:

(singularity) chmod -R 777 /home/firedrake && ls -lh /home
              >> drwxrwxrwx root root firedrake/

So while the firedrake folder can now in principle be written to, in reality when it is packaged in singularity image the stuff within the image itself cannot be touched without sudo access (because it was created with sudo and is supposed to be immutable) and the --writable flag.

This creates another dilemma because then we have nowhere to write .cache since the image is supposed to be untouchable. There are two ways around this, and I recommend the second one for firedrake:

when singularity exec is called, give it the --writable-tmpfs flag, which will provide a non-persistent write option (so it writes to some temporary space but everything is discarded when the singularity exec command is finished)
create an overlay so the writing is done to some allocated space, but this overlay can be re-used, which means the cache that is created is persistent across runs. To do this,

(singularity) ls -lh /home
              >> drwxrwxrwx root root firedrake/
(singularity) exit
(ubuntu)      sudo singularity build firedrake.sif firedrake/
              >> DONE

# create an 500MB overlay with dd and mkfs-ext3 (no need for sudo access)
(ubuntu)      dd if=/dev/zero of=firedrake_cache bs=1M count=500 && mkfs.ext3 firedrake_cache
(ubuntu)      singularity exec --overlay firedrake_cache firedrake.sif /home/firedrake/bin/python linear_SWE2d.py

where linear_SWE2d.py is a file I have that solves something so requiring a write to firedrake/.cache. If you shell into the image with the overlay option you should be able to see firedrake/.cache there (and it should not exist in the over shelled into without the overlay).

Installation can be done through a definition file too (see below), triggered by sudo singularity build firedrake.sif firedrake.def. It takes upwards of an hour on my laptop largely because PETSc takes ages to build. The resulting firedrake.sif file is about 2GBs (can probably be smaller, I haven’t figured out which of the files in src/PETSc are safe to delete…)

BootStrap: docker
From: ubuntu:20.04

%post
    export DEBIAN_FRONTEND=noninteractive
    apt -y update
    apt -y upgrade
    apt -y install git curl nano python2.7 python3.8 build-essential
    apt -y install dialog apt-utils autoconf automake bison flex cmake gfortran
    apt -y install libblas-dev liblapack-dev libtool
    apt -y install python3-dev python3-pip python3-tk python3-venv
    apt -y install zlib1g-dev libboost-dev
    cd /home/
    curl -O https://raw.githubusercontent.com/firedrakeproject/firedrake/master/scripts/firedrake-install
    python3 firedrake-install --disable-ssh

    echo "changing permissions to enable temporary writes if need be (particularly in .cache)"
    chmod -R 777 /home/firedrake

%environment
    export LC_ALL=C
    export PATH="/home/firedrake/bin:$PATH"

%runscript
    echo "in firedrake.sif"
    echo "attempting to call python --version"
    python --version

%labels
    Author Julian Mak