MIDOSS MOHID on graham¶
This section describes the steps to set up and run the MIDOSS version of the MOHID code on the ComputeCanada graham.computecanada.ca HPC cluster.
Modules Setup¶
When working on graham, the module load command must be used to load extra software components.
You can manually load the modules each time you log in,
or you can add the lines to your $HOME/.bashrc
file so that they are automatically loaded upon login.
The module load commands needed are:
module load StdEnv/2016.4
module load nco/4.6.6
module load netcdf-fortran/4.4.4
module load proj4-fortran/1.0
module load python/3.8.2
Warning
MIDOSS-MOHID does not build successfully in the StdEnv/2020 that became the default on graham on 1-Apr-2021. Please ensure that you have done module load StdEnv/2016.4 prior to building MIDOSS-MOHID.
Warning
The nco/4.6.6 module is incompatible with the module load netcdf-fortran-mpi/4.4.4 module that is required to run NEMO. So, if you are running both MIDOSS-MOHID and NEMO, you will need to manually load the appropriate modules as necessary.
Create a Workspace and Clone the Tools, Code and Configurations Repositories¶
graham provides several different types of file storage. We use project space for our working environments because it is large, high performance, and backed up. Scratch space is even larger, also high performance, but not backed up, so we use that as the space to execute MOHID runs in, but generally move the run results to project space. Files more than 60 days old are automatically purged from scratch space on graham.
graham automatically provides environment variables that are more convenient that remembering full paths to access your project and scratch spaces:
Your project space is at
$PROJECT/$USER/
Your scratch space is at
$SCRATCH/
Create MIDOSS/
directory trees in your project and scratch spaces:
$ mkdir -p $PROJECT/$USER/MIDOSS/results
$ mkdir -p $SCRATCH/MIDOSS/runs
Note
If the above command fails,
it may be because the symbolic link that PROJECT
points to was not created when your graham account was set up.
Try:
$ cd $HOME
$ ln -s $HOME/projects/def-allen project
Clone the following repositories:
Make-MIDOSS-Forcing that provides the make-hdf5 tool for generating forcing files for MIDOSS-MOHID-CODE code runs
MIDOSS-MOHID-CODE, the MIDOSS project version of MOHID that includes developed at Dalhousie University
MIDOSS-MOHID-config, the collection of MOHID run data (.dat) files and run description YAML files for the various oil spill scenarios
MIDOSS-MOHID-grid, the collection of grid-related files
moad_tools that provides the hdf5-to-netcdf4 tool for by MOHID-Cmd (or at the command-line, if necessary)
MOHID-Cmd, the MIDOSS-MOHID command processor for setting up and managing MIDOSS-MOHID-CODE code runs
NEMO-Cmd, the NEMO command processor that MOHID-Cmd uses code from
SalishSeaCast/grid, the collection of grid-related files for the SalishSeaCast NEMO model
$ cd $PROJECT/$USER/MIDOSS
$ git clone git@github.com:MIDOSS/Make-MIDOSS-Forcing.git
$ git clone git@github.com:MIDOSS/MIDOSS-MOHID-CODE.git
$ git clone git@github.com:MIDOSS/MIDOSS-MOHID-config.git
$ git clone git@github.com:MIDOSS/MIDOSS-MOHID-grid.git
$ git clone git@github.com:UBC-MOAD/moad_tools.git
$ git clone git@github.com:MIDOSS/MOHID-Cmd.git
$ git clone git@github.com:SalishSeaCast/NEMO-Cmd.git
$ git clone git@github.com:SalishSeaCast/grid.git SalishSeaCast-grid
Install Python Packages¶
Note
This method of installing the moad_tools, NEMO-Cmd, and MOHID-Cmd Python packages employs the “user scheme” for installation. It is appropriate and necessary on graham where we do not have our own Anaconda Python distribution installed. This method should not be used on EOAS work stations or other machines where you have Anaconda Python installed.
$ cd $PROJECT/$USER/MIDOSS
$ python3 -m pip install --user --editable Make-MIDOSS-Forcing
$ python3 -m pip install --user --editable moad_tools
$ python3 -m pip install --user --editable NEMO-Cmd
$ python3 -m pip install --user --editable MOHID-Cmd
You can confirm that the Make-MIDOSS-Forcing package and the make-hdf5 tool are correctly installed with the command:
$ make-hdf5 --help
from which you should see output like:
Usage: make-hdf5 [OPTIONS] YAML_FILENAME [%Y-%m-%d] [N_DAYS]
Create HDF5 forcing files for a MIDOSS-MOHID run.
YAML_FILENAME: File path/name of YAML file to control HDF5 forcing files
creation.
[%Y-%m-%d]: Date on which to start HDF5 forcing files creation.
N_DAYS: Number of days plus 1 of HDF5 forcing to create in each file.
Use 1 to create 2 days of forcing which is what is required for a 1 day
MOHID run.
Options:
--version Show the version and exit.
--help Show this message and exit.
You can confirm that the moad_tools package and the hdf5-to-netcdf4 tool are correctly installed with the command:
$ hdf5-to-netcdf4 --help
from which you should see output like:
Usage: hdf5-to-netcdf4 [OPTIONS] HDF5_FILE NETCDF4_FILE
Transform selected contents of a MOHID HDF5 results file HDF5_FILE into a
netCDF4 file stored as NETCDF4_FILE.
Options:
-v, --verbosity [debug|info|warning|error|critical]
Choose how much information you want to see
about the progress of the transformation;
warning, error, and critical should be
silent unless something bad goes wrong.
[default: warning]
--help Show this message and exit.
You can confirm that the NEMO-Cmd and MOHID-Cmd packaged and the mohid command processor are correctly installed with the command:
$ mohid --help
from which you should see output like:
usage: mohid [--version] [-v | -q] [--log-file LOG_FILE] [-h] [--debug]
MIDOSS-MOHID Command Processor
optional arguments:
--version show program's version number and exit
-v, --verbose Increase verbosity of output. Can be repeated.
-q, --quiet Suppress output except warnings and errors.
--log-file LOG_FILE Specify a file to log output. Disabled by default.
-h, --help Show help message and exit.
--debug Show tracebacks on errors.
Commands:
complete print bash completion command (cliff)
gather Gather results files from a MIDOSS-MOHID run.
help print detailed help for another command (cliff)
monte-carlo Prepare for and execute a collection of Monte Carlo runs of the MIDOSS-MOHID model.
prepare Set up the MIDOSS-MOHID run described in DESC_FILE and print the path of the temporary run directory.
run Prepare, execute, and gather results from a MIDOSS-MOHID model run.
Compile MIDOSS-MOHID¶
Compile and link the Mohid_Base_1, Mohid_Base_2, and MohidWater parts of the MOHID Framework.
Use an interactive job on graham for compilation because it is substantially faster (≥15%). Be sure to request at least 1024 MB of memory:
$ salloc --time=0:30:0 --cpus-per-task=1 --mem-per-cpu=1024m --account=def-allen
$ cd $PROJECT/$USER/MIDOSS/MIDOSS-MOHID-CODE/Solutions/linux
$ ./compile_mohid.sh -mb1 -mb2 -mw
The output looks something like:
#### Mohid Base 1 ####
compile mohidbase1 OK
#### Mohid Base 2 ####
compile mohidbase2 OK
#### Mohid Water ####
compile MohidWater OK
==========================================================================
build started: Tue Dec 18 13:10:09 PST 2018
build completed: Tue Dec 18 13:16:07 PST 2018
---> Executables ready <---
total 0
lrwxrwxrwx 1 dlatorne def-allen 36 Dec 18 13:16 MohidWater.exe -> ../src/MohidWater/bin/MohidWater.exe
==========================================================================
You can delete all of the compiled objects, libraries, and executables with:
$ ./compile_mohid.sh --clean
so that the next build will be “clean”; i.e. it won’t be able to include any products from previous builds.
Test MIDOSS-MOHID¶
The MIDOSS-MOHID-config/MarathassaConstTS/
directory contains a configuration that you can use to do a test run of your setup on graham.
It is the constant temperature and salinity version of the 2014 Marathassa spill in English Bay.
You should be able to run the test with:
$ cd $PROJECT/$USER/MIDOSS/MIDOSS-MOHID-config/MarathassaConstTS/
$ mohid run MarathassaConstTS.yaml $PROJECT/$USER/MIDOSS/results/MarathassaConstTS
The output looks something like:
mohid_cmd.run INFO: Created temporary run directory /scratch/dlatorne/MIDOSS/runs/MarathassaConstTS_2019-01-10T173855.512111-0800
mohid_cmd.run INFO: Wrote job run script to /scratch/dlatorne/MIDOSS/runs/MarathassaConstTS_2019-01-10T173855.512111-0800/MOHID.sh
mohid_cmd.run INFO: Submitted batch job 15523561
You can use the squeue command to monitor the status of your job:
$ squeue -u $USER
JOBID USER ACCOUNT NAME ST START_TIME TIME_LEFT NODES CPUS GRES MIN_MEM NODELIST (REASON)
15656820 dlatorne def-allen_cp MarathassaCons PD N/A 1:30:00 1 1 (null) 20000M (Priority)
An alias for squeue that provides more information and better formatting is:
alias sq='squeue -o "%.12i %.8u %.9a %.22j %.2t %.10r %.19S %.10M %.10L %.6D %.5C %N"'
sq -u $USER
JOBID USER ACCOUNT NAME ST REASON START_TIME TIME TIME_LEFT NODES CPUS NODELIST
15656820 dlatorne def-allen MarathassaConstTS PD Priority N/A 0:00 1:30:00 1 1
The oil particle trajectories calculated by MOHID will be in the Lagrangian_MarathassaConstTS.nc
file,
and the oil mass balance will be in the resOilOutput.sro
file.
Note
If the mohid run command prints an error message, you can get a Python traceback containing more information about the error by re-running the command with the --debug flag.
Using hdf5-to-netcdf4¶
Note
The mohid run command generates a MOHID.sh
shell script that includes using the hdf5-to-netcdf4 command-line tool to transform a MOHID Lagrangian.hdf5
output file into a netCDF4 file.
So,
you generally shouldn’t need to use hdf5-to-netcdf4 by itself,
but this section describes how to do so if necessary.
The hdf5-to-netcdf4 command-line tool can be used to transform a MOHID Lagrangian.hdf5
output file into a netCDF4 file.
Doing so is resource intensive in terms of memory and disk i/o,
so it has to be done in an interactive slurm session on graham.
Start an interactive slurm session with a command like:
$ salloc --time=00:20:0 --cpus-per-task=1 --mem-per-cpu=800m --account=def-allen
Choose the --time value to be close to what you expect to need in order to avoid having to wait too long for the session to be allocated to you.
For guidance,
transformation of a Lagrangian.hdf5
from a MOHID run for 7 days of model time on the SalishSeaCast domain takes anywhere from 6m30s to 17m30s,
depending on how graham is operating.
Once the interactive session starts, do the transformation by:
Copying the
.hdf5
to fast, local SSD storage on the nodeRunning hdf5-to-netcdf4 to store the
.nc
file on SSD storageCopying the
.nc
file back to project or scratch storage
$ cp Lagrangian.hdf5 $SLURM_TMPDIR/Lagrangian.hdf5
$ hdf5-to-netcdf4 Lagrangian.hdf5 Lagrangian.nc
$ cp $SLURM_TMPDIR/Lagrangian.nc Lagrangian.hdf5
You can prefix the hdf5 and nc file names with paths. You can get progress information from hdf5-to-netcdf4 by using the options --verbosity info or --verbosity debug. Please see hdf5-to-netcdf4 --help for details.