Documentation
Contents
|
Introduction
ALSVID-UQ is the Multilevel Monte Carlo Finite Volume solver for uncertainty
quantification in hyperbolic systems of conservation laws up to 3 space dimensions.
The code is written in an efficient modular fasion, hence extending the code is comparably easy.
If you have some suggestions or would like to report a bug, please contact me: jonas.sukys@sam.math.ethz.ch
Installation
Requirements
You need to have a C++ compiler (GCC works, but may not neccessarily generate the fastest executable).
You need to have installed a recent version of Python (2.5 or later). For using the bundled scripts for displaying the output you will also need the numpy and matplotlib packages.
For running ALSVID-UQ in parallel using multiple CPU cores or clustered computers you need to install an implementation of the MPI library, (eg OpenMPI).
Many of these packages will be installed or available pre-built to install on any recent Linux distribution
(eg Debian and Ubuntu have all the above in their package database).
MacOS X distributions of these packages are available from their web pages,
and are easy to install (perhaps apart from MayaVi).
We have not tried running ALSVID-UQ under Windows,
but versions of all these packages are available, so it should be possible.
Download and installation
Download the latest version of ALSVID-UQ from
http://www.sam.math.ethz.ch/alsvid-uq/downloads.html,
and uncompress the .tar.gz file.
For serial (1 core) simulations, no more installation is neccessary.
Test simulation
For a simple MLMC-FVM test, run (in terminal):
cd alsvid-uq/run python ../make.py model:orszagtang ML=4 L=2
This will build and run the Orzag-Tang vortex in 2d with randomly pertrubed initial magnetic fields.
The results can be visualised using the included scripts:
python -i ../plot.py
Then, at the Python promt, type:
r = Run() r.plot(eu.RHO)
Compiling and running
The 'make.py' script is used to build and run the code.
This script works out which source files are required for a particular run and only compiles those ones into the executable.
The make process is more complex than for usual software as all the settings for a run are specified at compile time for efficiency.
This means that a fresh compile is required for each new run, but on a computer that is not 10 years old, it will not take long.
The script will generate two files in the current directory:
1) executable 'alsvid-uq'
2) configuration file 'config.py' containing the alternatives and parameters used for this run
Script option '--name' can be used to set different name instead of 'config.py'.
Running the executable (which is done by the script unless specified otherwise) will generate at least three more files:
| File name | Description | When is generated |
|---|---|---|
| output.dat | a binary file containing the mean values of the conserved and extra variables for each saved timestep | always |
| times.py | contains the times at which output was generated | always |
| runtime.py | contains the runtime of the simulation (for MPI runs, several entries are available) | always |
| var_output.dat | is a binary file containing the variances of the conserved and extra variables for each saved timestep | only for stochastic simulations (L > 0 or ML > 1) |
| hist_data.py | contains histogram data | only for HISTOGRAMS=1 |
The default names of the above files are also modified by the script option '--name'.
ALTERNATIVES
The main choices of alternatives available for running ALSVID-UQ are 'equation', 'flux', 'model', 'solver', 'space', 'rng', and 'multi'.
These select which source code files will be compiled.
The available choice of 'flux', 'model' and 'solver' alternatives depend on which alternative was chosen for 'equation'.
There are also some equation-specific alternatives, for instance, 'topography' and 'material'.
To see a list of all available values for each alternative use the --ask script option.
Brief description of each alternative is provided in the table below:
| Alternative | Explanation | Examples | Default |
|---|---|---|---|
| equation | conservation law to be simulated (selects conserved variables). | mhd, euler, sw, burgers, advection, wave, bl | mhd |
| flux | specifies if the flux is: deterministic ('det') or stochastic. Depends on 'equation'. |
euler: specheats bl: permeability, perturbed, viscosity burgers: kl, kl-exp, perturbed |
det |
| model | specifies the model to run, i.e.: domain size, initial data, boundary conditions, time horizon. Depends on 'equation'. |
lots of models available; use the --ask script option | depends on 'equation' |
| solver | specifies the solver to use for the simulation. Depends on 'equation'. | euler: hll3, hll3L mhd: hll3, hll3L, hll3R, hll5 sw: ec, es-roe, es-rusanov wave: fdm |
hll |
| space | specifies the implementation of space and time. | First order: o1, o1fdm Second order: o2central, o2minmod, o2eno, o2enof, o2weno, o2wenof, o2ecminmod, o2tecno Thrid order: o3weno, o3wenof, o3norec |
o1 |
| rng | specifies the random number generator. | well512a, well19937a, mt19937, lcg | well512a |
| topography | [only for equation:sw] specifies the bottom topography. | flat, shore, kl, kl2d, bump, bump2d, hr, hr-iso, hr2d, hr2d-iso | flat |
| material | [only for equation:wave] material coefficient. | const, logn-kl, logn-kl2d, logn-fft, logu-kl, multilayer, riemann | const |
Alternatives are specified on the command line with an ':' sign, i.e.
python ../make.py equation:mhd model:orszagtang solver:hll3 space:o2weno
PARAMETERS
There are also parameters that can be set.
Some of these are sepcific to the alternative choices made (especially the 'model'),
while some are available on all runs.
To see a list of all available parameters use the --ask script option.
Brief description of each parameter is provided in the table below:
| Parameter | Default | Explanation | (FVM part) |
|---|---|---|
| NX, NY, NZ | 1 | Number of cells in the X, Y and Z directions |
| CFL | 0.9 | The CFL number used to select the time step size |
| SAVE0 | 0 | sake a snapshot of the solution and t=0? |
| NSAVES | 1 | the number snapshots of the solution to save |
| INITIAL_DATA_SCALING | 1.0 | scaling of the initial data |
| INITIAL_DATA_OFFSET | 0.0 | offset of the initial data (applied after scaling) |
| (MLMC part) | ||
| ML | 1 | number of MC samples (if L=0) or number of samples at the finest level (if L>0). If set to '-1', then the number of samples is read from the file ML.txt. |
| L | 0 | highest level number for multilevel MC with levels 0, 1, ..., L. Setting L=0 results in conventional MC with ML samples. |
| SMOOTH_SOLUTION | 0 | is the solution smooth (continuous)? If not, the effective FVM convergence rate is half of formal order of accuracy (and never greater than 1). |
| DOWNSAMPLE | 0 | should downsampling be used? |
| COMPUTE_VARIANCE | 1 | should variance be computed? |
| VARIANCE_FIX | 1 | cut off all negative values in variance |
| COMPUTE_KURTOSIS | 0 | should kurtosis (4th centered moment) be computed instead of variance? |
| HISTOGRAMS | 0 | should histogram data be computed? |
| NUMERICAL_STABILIZATION | 1 | enable numerical stabilization of multilevel part with exp()-log() |
| RUN_NUMBER | 0 | for randomization of runs when multiple of them are executed to compute some average quantity: error, runtime, etc. |
| SAMPLES_OFFSET | 0 | only samples after SAMPLES_OFFSET are used. (needed for fault tolerance analysis) |
| PRIME_SHIFT | 10 | shift in the array of prime seeds |
| (Output customization) | ||
| PRINT_t | 1 | print all simulation times and timesteps |
| WRITE_DATAFILES | 1 | write out datafiles (*.dat) |
| DATA_PATH | "" | path for datafiles (*.dat) |
| (Equation specific) | ||
| POWELL | 1 | [only for equation:mhd] Set to 0 to switch off the Powell-Gudenov source term. The hll2 solver always behaves as if POWELL=0. |
| ANTI_ALIASING | 1 | [only for equation:sw] anti-aliasing for 'hr' and 'hr2d' bottom topographies |
Parameters are specified on the command line with an '=' sign, i.e.
python ../make.py equation:mhd model:orszagtang NX=256 NY=256
SCRIPT OPTIONS
The behaviour of the make.py script can be modified with options:
| Script option | Explanation |
|---|---|
| -a, --ask | Tell the script to query the user for values of the alternatives. If this option appears twice then the script will query the user for values of the parameters too. |
| -c file, --config=file | Specify a configuration file to use to specify alternatives and parameters to use for the run. Alternatives and parameters specified on the command line will override these. |
| -g, --debug | Tell the compiler to generate output suitable for use with a debugger. |
| -n name, --name=name | Modify the names of the files generated to include this name. Useful if results of more than one run are to be stored in a single directory. |
| -r, --rerun | Use the existing config.py configuration file. See the -c option. |
| -x, --norun | Compile only. Do not run the executable generated. |
| -q, --quiet | Run quietly: do not write information to the standard output. |
| -v, --verbose | Display parameter values, compiler output and job submission query. |
| -l, --log | Log output to file. |
COMPILER
If you wish to use a compiler other than GCC, then you should create a file in the same directory as 'make.py' called 'local.py'.
Copy one of the example files in the 'configs' directory, and modify is for your requirements.
PARALLEL SIMULATIONS
For running large models on multi-core computers and clusters, ALSVID-UQ supports parallelized runs.
To use this feature you should specify a parallelization module for the 'multi' alternative.
The only one currently available is the MPI implementation 'multi:mpi'. For this a MPI library is required.
Several implementations of the MPI standard are available. ALSVID-UQ has been tested with OpenMPI.
The following parameters are used for parallel runs:
| Parameter | Default | Explanation |
|---|---|---|
| MULTIX | 1 | number of processes to use in the X direction |
| MULTIY | 1 | number of processes to use in the Y direction |
| MULTIZ | 1 | number of processes to use in the Z direction |
| MULTIM | 1 | number of "samplers" (processor groups computing series of samples) |
| LOAD_BALANCING_TYPE | 1 | two optimized load balancing types are available, "1" and "2" |
Configuration
Running in MPI mode requires more configuration that is not neccessary to run in single process mode.
Several things must be added to the 'local.py' configuration file.
First a compiler specification must be added which specifies where to find the MPI library, or uses an MPI frontend to the compiler (mpiCC).
Secondly, on supercomputers and clusters, a job queue system is often used, and a script file is require for these to specify the job.
The make script 'make.py' will generate a suitable file from a template provided.
Thirdly, the command to use to run the MPI version needs to be specified.
Running the executable directly will not work.
This command can either be 'mpiexec' or 'mpirun', or it can be a command to submit a job to the job queue.
Since these details strongly depend on the MPI implementation used and on other aspects of the system being run on
(e.g. the job queue management software being used), this information cannot be supplied with ALSVID-UQ but must be specified for each system.
Examples of various 'local.py' files are available in the 'configs' directory,
and one of these that most closely meets your needs should be copied and modfied.
Configuration files are already available (see 'configs' directory) for the following clusters:
| Name | Description | File name |
|---|---|---|
| Brutus (ETH Zurich) | GNU compiler and debugger | brutus-local.py |
| Rosa (CSCS Lugano) | GNU compiler and debugger, CrayPat profiler | rosa-gnu-craypat-local.py |
| Todi (CSCS Lugano) | GNU compiler and debugger, CrayPat profiler | rosa-gnu-craypat-local.py |
| Todi (CSCS Lugano) | CCE compiler and debugger, CrayPat profiler | rosa-cce-craypat-local.py |
Script options for job submission
| Script option | Default | Explanation |
|---|---|---|
| -w, --walltime | 1 | number of expected runtime in hours (wall-time) |
| -m, --memory | 1024 | amount of needed memory (RAM) per core in MB |
| -t, --ptile | 32 | number of cores per node (useful when more memory (RAM) is needed per core) |
| -p, --profile | - | enable profiling |
| -o, --xopts | - | specify extra options for job submission |
Custom (cluster dependent) default values for -w -m -t can be specified in the 'local.py' file.
Parallel output
Instead of generating just one 'output.dat' file, a parallel run generates one file per process.
These have the coordinates of the process separated by '.' appended to the output file name.
A python script (which requires numpy) is available that will combine these into a single binary file:
python ../combine.py
will combine the output files from a run without any --name script option supplied, and:
python ../combine.py myrun
will do the same if 'myrun' is given as the name with the --name script option.
These commands assume that the current directory contains the output files and is one level down from the main ALSVID-UQ directory,
e.g. in the run subdirectory.
Combine uses information from the configuration file written by 'make.py'.
PLOTTING
Once the simulation has finished successfully, the results of the simulation are stored in a binary file.
The binary file can be very large, depending on the resolution you have asked for and the number of time steps you wish to have saved.
The parameters of the run are also saved, and can be used by generic tools to understand the layout of the data in the binary file.
If you wish to use your own tools for processing the output, you should know the following:
- the binary file is divided into a section for each time step that is saved
- each section contains (equation dependent) conserved and extra variables (see file 'variables.py')
- each variable is a 3d array, with the X index changing slowest (i.e. using the C indexing standard and not the Fortran one)
- the variables are in order: first all conserved variables, then all extra variables (equation dependent, see file 'variables.py')
1d and 2d plotting
The 'plot.py' and 'errorplot.py' scripts use the numpy and matplotlib packages.
Scripts can be used to:
- making 1d and 2d plots from the results of ALSVID-UQ simulations
- generating error convergence plots
- generating parallel efficiency and (strong or weak) scaling plots
- exporting output data to Matlab files
- generating movies (to see time evolution) of the simulations
Script 'plot.py' can be run with:
python -i ../plot.py
This assumes that you are one level below the main ALSVID-UQ directory. Script will load the run configuration into an object named 'r' and leave you with a Python prompt.
If you have used the '--name' option then the name you have used should be specified as a parameter to routine 'Run()':
r = Run("myrun")
This is especially useful if you want to compare several runs, you can load them into different Python variables (instead of 'r').
To make a plot from the results you can use the 'plot' routine:
r.plot(mhd.RHO, 5)
This will plot the density (RHO) for the MHD equations (equation:mhd) at the 5th snapshot (provided parameter 'NSAVES' is larger than 5).
For the complete list of variables that can be loaded from output files and plotted, see file 'variables.py'.
If the run is a 1d run, it will be a 1d plot, if it is a 2d run, it will be a 2d plot.
If no snapshot number is specified, then the final snapshot (at the final time) is used.
In order to have initial data and the solution in the same plot, one can use
r.plot0(mhd.RHO)
To save a plot to a file, one can specify the filename as an option to the 'plot' and 'plot0' routines:
r.plot(mhd.RHO, save='RHO_final.png')
Several other output formats (such as PDF) can be used.
The file extension specifies the format.
See the matplotlib documentation for a full list of recognised formats.
One can also load the values for one variable into a Python array (the snapshot number is required in this case):
rho = r.var(mhd.RHO, 5)
3d plotting
Todo.EXAMPLES
Serial MC run
python ../make.py equation:euler model:sod-shock-tube ML=16
Serial MLMC run
python ../make.py equation:euler model:sod-shock-tube ML=16 L=2
Parallel MC run
python ../make.py equation:euler model:sod-shock-tube multi:mpi MULTIX=2 MULTIM=4 ML=16
Parallel MLMC run
python ../make.py equation:euler model:sod-shock-tube multi:mpi MULTIX=2 MULTIM=4 ML=16 L=2
MLMC run with 2nd order space approximation
python ../make.py equation:euler model:sod-shock-tube space:o2weno ML=16 L=2
MLMC run with enabled downsampling
python ../make.py equation:euler model:sod-shock-tube DOWNSAMPLE=1 ML=16 L=2
EXTENDING ALSVID-UQ
Todo. A few short remarks to keep in mind:- Only ConservedVars are exchanged during MPI communication calls. XtraVars are computed afterwards. Keep in mind when extending 'structures.h'.
- For equation:sw, only NEUMANN boundary conditions are implemented.
- For equation:wave, only REPEAT boundary conditions are implemented.