Contents
|
ALSVID-UQ is the Multilevel Monte Carlo Finite Volume solver for uncertainty
quantification in hyperbolic systems of conservation laws up to 3 space dimensions.
The code is written in an efficient modular fasion, hence extending the code is comparably easy.
If you have some suggestions or would like to report a bug, please contact me: jonas.sukys@sam.math.ethz.ch
You need to have a C++ compiler (GCC works, but may not neccessarily generate the fastest executable).
You need to have installed a recent version of Python (2.5 or later). For using the bundled scripts for displaying the output you will also need the numpy and matplotlib packages.
For running ALSVID-UQ in parallel using multiple CPU cores or clustered computers you need to install an implementation of the MPI library, (eg OpenMPI).
Many of these packages will be installed or available pre-built to install on any recent Linux distribution
(eg Debian and Ubuntu have all the above in their package database).
MacOS X distributions of these packages are available from their web pages,
and are easy to install (perhaps apart from MayaVi).
We have not tried running ALSVID-UQ under Windows,
but versions of all these packages are available, so it should be possible.
Download the latest version of ALSVID-UQ from
http://www.sam.math.ethz.ch/alsvid-uq/downloads.html,
and uncompress the .tar.gz file.
For serial (1 core) simulations, no more installation is neccessary.
For a simple MLMC-FVM test, run (in terminal):
cd alsvid-uq/run python ../make.py model:orszagtang ML=4 L=2
This will build and run the Orzag-Tang vortex in 2d with randomly pertrubed initial magnetic fields.
The results can be visualised using the included scripts:
python -i ../plot.py
Then, at the Python promt, type:
r = Run() r.plot(eu.RHO)
The 'make.py' script is used to build and run the code.
This script works out which source files are required for a particular run and only compiles those ones into the executable.
The make process is more complex than for usual software as all the settings for a run are specified at compile time for efficiency.
This means that a fresh compile is required for each new run, but on a computer that is not 10 years old, it will not take long.
The script will generate two files in the current directory:
1) executable 'alsvid-uq'
2) configuration file 'config.py' containing the alternatives and parameters used for this run
Script option '--name' can be used to set different name instead of 'config.py'.
Running the executable (which is done by the script unless specified otherwise) will generate at least three more files:
| File name | Description | When is generated |
|---|---|---|
| output.dat | a binary file containing the mean values of the conserved and extra variables for each saved timestep | always |
| times.py | contains the times at which output was generated | always |
| runtime.py | contains the runtime of the simulation (for MPI runs, several entries are available) | always |
| var_output.dat | is a binary file containing the variances of the conserved and extra variables for each saved timestep | only for stochastic simulations (L > 0 or ML > 1) |
| samples.py | contains the number of samples used for each mesh level | always |
| balancer.txt | contains estimated loads for each multi sampler (formatted table) | only for parallel simulations |
| balancer.dat | contains estimated loads for each multi sampler (python list) | only for parallel simulations |
| loads.dat | contains estimated loads for each multi sampler (python list) | only for parallel simulations and for ESTIMATE_LOADS=1, i.e. for balacer:adaptive |
| hist_data.py | contains histogram data | only for HISTOGRAMS=1 |
The default names of the above files are also modified by the script option '--name'.
If the paramater WRITE_LEVEL_DATAFILES=1, then output files for MC-FVM estimates from each mesh resolution level and type (0 or 1) are also generated.
Files 'balancer.txt' and 'balancer.dat' are generated before simulation starts, and can be used for analysis of failed runs.
The main choices of alternatives available for running ALSVID-UQ are (problem-related) 'equation', 'flux', 'model', 'solver', 'space', 'stats'
and (implementation-related) 'rng', 'multi', 'balancer'.
These select which source code files will be compiled.
The available choice of 'flux', 'model' and 'solver' alternatives depend on which alternative was chosen for 'equation'.
There are also some equation-specific alternatives, for instance, 'topography' and 'material'.
To see a list of all available values for each alternative use the --ask script option.
Brief description of each alternative is provided in the table below:
| Alternative | Explanation | Examples | Default |
|---|---|---|---|
| equation | conservation law to be simulated (selects conserved variables). | mhd, euler, sw, burgers, advection, wave, bl | mhd |
| flux | specifies if the flux is: deterministic ('det') or stochastic. Depends on 'equation'. |
euler: specheats bl: permeability, perturbed, viscosity burgers: kl, kl-exp, perturbed |
det |
| model | specifies the model to run, i.e.: domain size, initial data, boundary conditions, time horizon. Depends on 'equation'. |
lots of models available; use the --ask script option | depends on 'equation' |
| stats | specifies the statistical estimates to be computed. | mean, mean-var, mean-covar, mean-kurtosis, mean-var-functional, mean-var-unstable, histograms, threshold-sw (equation:sw only) | mean-var |
| solver | specifies the solver to use for the simulation. Depends on 'equation'. | euler: hll3, hll3L mhd: hll3, hll3L, hll3R, hll5 sw: ec, es-roe, es-rusanov wave: fdm |
hll |
| space | specifies the implementation of space and time. | First order: o1, o1fdm Second order: o2central, o2minmod, o2eno, o2enof, o2weno, o2wenof, o2ecminmod, o2tecno Thrid order: o3weno, o3wenof, o3norec |
o1 |
| rng | specifies the random number generator. | well512a, well19937a, mt19937, lcg | well512a |
| topography | [only for equation:sw] specifies the bottom topography. | flat, shore, bump, bump2d, hr, hr-iso, hr2d, hr2d-iso | flat |
| material | [only for equation:wave] material coefficient. | const, logn-kl, logn-kl2d, logn-kl2d-int, logn-kl3d-int, logn-fft(-mpi), logn-fft-per(-mpi), logn-fft-per-int(-mpi), logu-kl, multilayer, riemann | const |
Alternatives are specified on the command line with an ':' sign, i.e.
python ../make.py equation:mhd model:orszagtang solver:hll3 space:o2weno
There are also parameters that can be set.
Some of these are sepcific to the alternative choices made (especially the 'model'),
while some are available on all runs.
To see a list of all available parameters use the --ask script option.
Brief description of each parameter is provided in the table below:
| Parameter | Default | Explanation | (FVM part) |
|---|---|---|
| NX, NY, NZ | 1 | Number of cells in the X, Y and Z directions |
| CFL | 0.9 | The CFL number used to select the time step size |
| SAVE0 | 0 | sake a snapshot of the solution and t=0? |
| NSAVES | 1 | the number snapshots of the solution to save |
| INITIAL_DATA_SCALING | 1.0 | scaling of the initial data |
| INITIAL_DATA_OFFSET | 0.0 | offset of the initial data (applied after scaling) |
| (MLMC part) | ||
| ML | 1 | number of MC samples (if L=0) or number of samples at the finest level (if L>0). If set to '-1', then the number of samples is read from the file ML.txt. |
| L | 0 | highest level number for multilevel MC with levels 0, 1, ..., L. Setting L=0 results in conventional MC with ML samples. |
| USE_OPTIMAL_NUMBER_OF_SAMPLES | 0 | uses asymptotically optimized (instead of equilibrated) number of samples (removes the log-term in error convergence estimates). |
| READ_SAMPLES_FROM_FILE | 0 | read the number of samples for each level from file SAMPLES_FILE. |
| SAMPLE_NUMBERS_FILE | "Ml.txt" | file from which the number of samples for each level should be read. |
| SMOOTH_SOLUTION | 0 | is the solution smooth (continuous)? If not, the effective FVM convergence rate is half of formal order of accuracy (and never greater than 1). |
| COMPUTE_VARIANCE | 1 | should variance be computed? |
| RUN_NUMBER | 0 | for randomization of runs when multiple of them are executed to compute some average quantity: error, runtime, etc. |
| SAMPLES_OFFSET | 0 | only samples after SAMPLES_OFFSET are used. (needed for fault tolerance analysis) |
| USE_PRIME_SEEDS | 0 | map seed to the array of prime numbers before seeding the PRNG |
| PRIME_SHIFT | 0 | shift in the array of prime seeds |
| (Output customization) | ||
| PRINT_t | 1 | print all simulation times and timesteps |
| WRITE_DATAFILES | 1 | write out datafiles (*.dat) |
| WRITE_LEVEL_DATAFILES | 0 | write out datafiles (*.dat) for each MLMC level and type |
| DATA_PATH | "" | path for datafiles (*.dat) |
| (Equation specific) | ||
| POWELL | 1 | [only for equation:mhd] Set to 0 to switch off the Powell-Gudenov source term. The hll2 solver always behaves as if POWELL=0. |
| ANTI_ALIASING | 1 | [only for equation:sw] anti-aliasing for 'hr' and 'hr2d' bottom topographies |
| VISCOUS | 0 | [only for equation:burgers] enables viscosity |
| VISCOSITY | 1e-3 | [only for equation:burgers] sets viscosity value |
Parameters are specified on the command line with an '=' sign, i.e.
python ../make.py equation:mhd model:orszagtang NX=256 NY=256
The behaviour of the make.py script can be modified with options:
| Script option | Explanation |
|---|---|
| -a, --ask | Tell the script to query the user for values of the alternatives. If this option appears twice then the script will query the user for values of the parameters too. |
| -c file, --config=file | Specify a configuration file to use to specify alternatives and parameters to use for the run. Alternatives and parameters specified on the command line will override these. |
| -g, --debug | Tell the compiler to generate output suitable for use with a debugger. |
| -n name, --name=name | Modify the names of the files generated to include this name. Useful if results of more than one run are to be stored in a single directory. |
| -r, --rerun | Use the existing config.py configuration file. See the -c option. |
| -x, --norun | Compile only. Do not run the executable generated. |
| -s, --simulate | Simulate 'make' (no actual compilation or running). |
| -q, --quiet | Run quietly: do not write information to the standard output. |
| -v, --verbose | Display parameter values, compiler output and job submission query. |
| -l, --log | Log output to file. |
If you wish to use a compiler other than GCC, then you should create a file in the same directory as 'make.py' called 'local.py'.
Copy one of the example files in the 'configs' directory, and modify it (if needed) for your requirements.
For running large models on multi-core computers and clusters, ALSVID-UQ supports parallelized runs.
To use this feature you should specify a parallelization module for the 'multi' alternative.
The only one currently available is the MPI implementation 'multi:mpi'. For this a MPI library is required.
Several implementations of the MPI standard are available. ALSVID-UQ has been tested with OpenMPI.
For parallel simulations, several load balancing algorithms could be used, by specifying the 'balancer' alternative.
At the moment, 'balancer:static' and 'balancer:adaptive' are available, both are parallelized.
Adaptive load balancing has a slighly larger overhead, however, is more general,
and allows for different paralllelization configurations of the same sampler:
for small samples, domain decomposition parallelization is replaced by MC parallelization,
enforcing the specified MIN_CELLS_PER_CORE and this way significantly increasing the efficiency.
Adaptive load balancer is a generalization of greedy load balancing algorithm for workers with non-uniform speed of execution,
ans is based on combined knowledge about a-priori FVM work estimates per time step (like in static)
and computation of (possibly random) time step size (based on CFL).
The following parameters are used for parallel runs:
| Parameter | Default | Explanation |
|---|---|---|
| MULTIX | 1 | number of processes to use in the X direction |
| MULTIY | 1 | number of processes to use in the Y direction |
| MULTIZ | 1 | number of processes to use in the Z direction |
| MULTIM | 1 | number of "samplers" (processor groups computing series of samples) |
| MULTI_CORES | 0 | total number of processes to be used (if specified, then MULTIX, MULTIY, MULTIZ and MULTIM are ignored) |
| ENSURE_PARALLEL_REPRODUCIBILITY | 1 | ensure, that the final result does not depend on the parallelization configuration or load balancing (can reduce efficiency) |
Running in MPI mode requires more configuration that is not neccessary to run in single process mode.
Several things must be added to the 'local.py' configuration file.
First a compiler specification must be added which specifies where to find the MPI library, or uses an MPI frontend to the compiler (mpiCC).
Secondly, on supercomputers and clusters, a job queue system is often used, and a script file is require for these to specify the job.
The make script 'make.py' will generate a suitable file from a template provided.
Thirdly, the command to use to run the MPI version needs to be specified.
Running the executable directly will not work.
This command can either be 'mpiexec' or 'mpirun', or it can be a command to submit a job to the job queue.
Since these details strongly depend on the MPI implementation used and on other aspects of the system being run on
(e.g. the job queue management software being used), this information cannot be supplied with ALSVID-UQ but must be specified for each system.
Examples of various 'local.py' files are available in the 'configs' directory,
and one of these that most closely meets your needs should be copied and modfied.
Configuration files are already available (see 'configs' directory) for the following clusters:
| Name | Description | File name |
|---|---|---|
| Local (Linux) | GNU compiler and debugger | gnu-local.py |
| Brutus (ETH Zurich) | GNU compiler and debugger | brutus-local.py |
| Rosa (CSCS Lugano) | GNU compiler and debugger, CrayPat profiler | rosa-gnu-craypat-local.py |
| Rosa (CSCS Lugano) | Intel compiler and debugger, CrayPat profiler | rosa-icc-craypat-local.py |
| Rosa (CSCS Lugano) | PGI compiler and debugger, CrayPat profiler | rosa-pgi-craypat-local.py |
| Rosa (CSCS Lugano) | CCE compiler and debugger, CrayPat profiler | rosa-cce-craypat-local.py |
| Todi (CSCS Lugano) | GNU compiler and debugger, CrayPat profiler | todi-gnu-craypat-local.py |
| Todi (CSCS Lugano) | Intel compiler and debugger, CrayPat profiler | todi-icc-craypat-local.py |
| Todi (CSCS Lugano) | PGI compiler and debugger, CrayPat profiler | todi-pgi-craypat-local.py |
| Todi (CSCS Lugano) | CCE compiler and debugger, CrayPat profiler | todi-cce-craypat-local.py |
| Script option | Default | Explanation |
|---|---|---|
| -w, --walltime | 1 | number of expected runtime in hours (wall-time), or, alternatively, HH.MM mean HH hours and MM minutes (MM is bounded by 59) |
| -m, --memory | 1024 | amount of needed memory (RAM) per core in MB |
| -t, --ptile | 32 | number of cores per node (useful when more memory (RAM) is needed per core) |
| -p, --profile | - | enable profiling |
| -o, --xopts | - | specify extra options for job submission |
Custom (cluster dependent) default values for -w -m -t can be specified in the 'local.py' file.
Instead of generating just one 'output.dat' file, a parallel run generates one file per process.
These have the coordinates of the process separated by '.' appended to the output file name.
A python script (which requires numpy) is available that will combine these into a single binary file:
python ../combine.py
will combine the output files from a run without any --name script option supplied, and:
python ../combine.py myrun
will do the same if 'myrun' is given as the name with the --name script option.
These commands assume that the current directory contains the output files and is one level down from the main ALSVID-UQ directory,
e.g. in the run subdirectory.
Combine uses information from the configuration file written by 'make.py'.
Script 'plot.py' can be run with:
python -i ../plot.py
This assumes that you are one level below the main ALSVID-UQ directory, and in the directory where all simulation data is.
Script will load the run configuration into an object named 'r' and leave you with a Python prompt.
If you have used the '--name' option then the name you have used should be specified as a parameter to routine 'Run()':
r = Run("myrun")
This is especially useful if you want to compare several runs, you can load them into different Python variables (instead of 'r').
To make a plot from the results you can use the 'plot' routine:
r.plot(mhd.RHO, 5)
This will plot the density (RHO) for the MHD equations (equation:mhd) at the 5th snapshot (provided parameter 'NSAVES' is larger than 5).
For the complete list of variables that can be loaded from output files and plotted, see file 'variables.py'.
If the run is a 1d run, it will be a 1d plot, if it is a 2d run, it will be a 2d plot, if it is a 3d run, it will be a 3d plot.
If needed, plotting parameter 'cut_slice' could be used to obtain 1d plots from 2d or 3d plots, and 2d plots from 3d plots.
If no snapshot number is specified, then the final snapshot (at the final time) is used.
In order to have initial data and the solution in the same plot (works only for 1d and 2d plots), one can use
r.plot0(mhd.RHO)
To save a plot to a file, one can specify the filename as an option to the 'plot' and 'plot0' routines:
r.plot(mhd.RHO, save='RHO_final.png')
Several other output formats (such as PDF) can be used.
The file extension specifies the format.
See the matplotlib documentation for a full list of recognised formats.
One can also load the values for one variable into a Python array (the snapshot number is required in this case):
rho = r.var(mhd.RHO, 5)
Below, we provide a table for some non-intuitive parameters,
which can be used within some plotting routines.
For the default values of the parameters, refer to the 'plot.py' script.
| Parameter | Description | Example |
|---|---|---|
| extent | specifies the extent for the values | [min, max] |
| cut_slice | cuts a slice at a position 'pos' of a coordinate 'dir' | {'dir':'x', 'pos':'0.1'} or {'dir':'x', 'pos':'0.1', 'dir2':'z', 'pos2':'0.6'} |
| view | sets the specified view in 3d plots | {'azimuth':45, 'elevation':10, 'distance':1.5} |
| contour_levels | specifies the values of the countours for contour plots | [3, 5.6, 7, 9.1, 10] |
| clip | clips a place (or two) of a 3d plot, allowing to observe the inner structures | {'origin' : (0.5, 0.5, 0.5), 'normals' : ((0,0,1), (0,1,0) or None, None)} |
| highlight | highlights a contour with a specified value in 3d plots | (highlight_mean, highlight_var) |
The 'scripts/plot1d.py', 'scripts/plot2d.py' and 'scripts/plot2d_histograms.py'
scripts use the numpy and matplotlib packages.
There are three main backends for the 3d plotting: MayaVi, VisIt and Paraview. The corresponding scripts are 'scripts/plot3d_mayavi.py', 'scripts/plot3d_visit.py' and 'scripts/plot3d_paraview.py'. The chosen backend must be specified in 'local_visualization.py' file.
Several (visualization) configuration files are already available (see 'configs' directory):
| Name | Description | File name |
|---|---|---|
| Local (Linux) | MayaVi | local-mayavi_visualization.py |
| Local (Linux) | VisIt | local-visit_visualization.py |
| Eiger (CSCS Lugano) | VisIt | eiger-visit-local_visualization.py |
For the chosen 3d visualization backend, the desired (parallel, backend-specific) engine can be specified in the plot commands by
rho = r.var(mhd.RHO, engine=get_engine(4,2))where the required engine could be generated using the respective function 'get_engine()' in the 'local_visualization.py' file.
| Routine | Explanation |
|---|---|
| plot_imbalance() | plots results of load balancing, including both theoretical and measured values |
| plot_loads() | plots the estimated loads for each sample, only for ESTIMATE_LOADS=1 |
Once the simulation has finished successfully, the results of the simulation are stored in binary files, together with some ASCII metafiles.
The binary files can be very large, depending on the resolution you have asked for and the number of time steps you wish to have saved.
The parameters of the run are also saved, and can be used by generic tools to understand the layout of the data in the binary file.
If you wish to use your own tools for processing the output, you should know the following:
Serial MC run
python ../make.py equation:euler model:sod-shock-tube ML=16
Serial MLMC run
python ../make.py equation:euler model:sod-shock-tube ML=16 L=2
Parallel MC run
python ../make.py equation:euler model:sod-shock-tube multi:mpi MULTIX=2 MULTIM=4 ML=16
Parallel MLMC run
python ../make.py equation:euler model:sod-shock-tube multi:mpi MULTIX=2 MULTIM=4 ML=16 L=2
MLMC run with 2nd order space approximation
python ../make.py equation:euler model:sod-shock-tube space:o2weno ML=16 L=2
MLMC run with enabled downsampling
python ../make.py equation:euler model:sod-shock-tube DOWNSAMPLE=1 ML=16 L=2