simplicity.runners package
Submodules
simplicity.runners.multiprocessing module
@author: jbescudie
simplicity.runners.serial module
@author: jbescudie
simplicity.runners.slurm module
@author: jbescudie
Slurm will create new processes (on possibily other hosts). We need to let these processes load user’s “run_seeded_simulation” function.
Given the user’s provided function 1. we determine a python importable path 2. in processes started by Slurm, the path is imported
- Caveat:
this imposes the following restriction: the user must import the run_seeded_simulation function before calling simplicity.runners.slurm.run_seeded_simulations(…). A test verifies this and raises before submitting jobs to Slurm.
Kown bugs: 1. if the run_seeded_function is importable when submitting, but not in the processes
started by Slurm. They will fail just after the .started signal is emmitted.
Interrupting the program before completion will leave held jobs in Slurm’s queue, and leaving the Data directory in a incomplete state (like for other simplicity.runners implementations).
In this case, manually cleaning Slurm (hint: squeue) and the Data directory (removing or archiving the experiment folder) is highly recommanded before retrying.
- class simplicity.runners.slurm.SimulationsStatus(total, submitted, released, left, pending, started, running, completed, failed)[source]
Bases:
NamedTuple- completed: int
Alias for field number 7
- failed: int
Alias for field number 8
- left: int
Alias for field number 3
- pending: int
Alias for field number 4
- released: int
Alias for field number 2
- running: int
Alias for field number 6
- started: int
Alias for field number 5
- submitted: int
Alias for field number 1
- total: int
Alias for field number 0
- simplicity.runners.slurm.get_platform_executable_extension()[source]
file extension to use when calling Slurm command-line utilities (sbatch, squeue, scontrol).
simplicity.runners.unit_run module
- simplicity.runners.unit_run.run_seeded_simulation(seeded_simulation_parameters_path: str, experiment_name: str) None[source]
runs one seeded simulation. This function isolates how to run one simulation from the looping over all seeded simulation parameters of one experiment.
Support for running Slurm requires this function to be importable from its name as reference (simplicity.runners.unit_run.run_seeded_simulation). Also this function’s arguments are passed by reference (achieved to serializing/deserializing the reference to the simulation parameters that are resolved via the settings_manager.read_seeded_simulation_parameters function).
Module contents
@author: jbescudie
Abstract functions for running the seeded simulations of an experiment given its name. Each seeded simulation is run by a run_seeded_simulation function passed as argument.
Current implementations: Target Description
simplicity.runners.serial Single host Simple pure python for loop.
simplicity.runners.multiprocessing Single host Uses CPython built-in concurrent.futures.ProcessPoolExecutor.
simplicity.runners.slurm Cluster Submits and monitors seeded simulations as a Slurm job array (Slurm must be installed).
- simplicity.runners.run_seeded_simulation(seeded_simulation_parameters_path: str, experiment_name: str) None[source]
Abstract function for running a single seeded simulation of an experiment.
seeded_simulation_parameters_path: str path to the file containing the simulation’s parameters incl. seed experiment_name: str reference to the experiment for other simplicity component like simplicity.settings_manager and simplicity.output_manager
- simplicity.runners.run_seeded_simulations(self, experiment_name: str, run_seeded_simulation: Callable[[str, str], None])[source]
Abstract function for running the seeded simulations of an experiment given its name.
experiment_name: str reference to the experiment for other simplicity component like simplicity.settings_manager and simplicity.output_manager run_seeded_simulation: Callable function with signature (seeded_simulation_parameters_path: str, experiment_name: str) -> None