romtools.hpc.components#

Focused helpers that the dispatchers compose.

class romtools.hpc.components.BaseCaller(*, config=None, logger=None)[source]#

Bases: Component, ABC

Executes a Python callable named by a “module:qualname” target string.

Arguments and return values may contain numpy arrays, tuples, lists, and dictionaries thereof.

Parameters:

config (dict)
logger (Logger)

class romtools.hpc.components.BaseFileManager(*, config=None, logger=None)[source]#

Bases: Component, ABC

Reads and writes files on whichever machine a dispatcher runs work on.

Parameters:

config (dict)
logger (Logger)

abstractmethod remove_dir(path)[source]#

Remove a directory and its contents. A missing directory is not an error.

Parameters:: path (str)
Return type:: None

class romtools.hpc.components.BaseTransferManager(*, files, config=None, logger=None, campaign_directory=None)[source]#

Bases: CampaignComponent

Stages the files a campaign needs into its run directory, and brings its results back afterwards.

Nothing moves by default: a dispatcher whose work already runs against the files in place needs neither half. Subclasses override whichever half genuinely has work to do.

Parameters:

files (BaseFileManager) – The file manager whose resolve_path decides what paths mean
campaign_directory (str) – The campaign directory results are gathered into
config (dict) – The dispatcher’s configuration dictionary
logger (Logger) – An instance of the Logger class for logging

Raises:

ValueError – if the configured collect or upload patterns are invalid.

collect_results()[source]#

Gather a finished job’s output into the local campaign directory.

Return type:: None

upload(run_directory=None)[source]#

Put the configured upload patterns into the run directory.

Parameters:: run_directory (str)
Return type:: None

class romtools.hpc.components.Component(*, config=None, logger=None)[source]#

Bases: object

Base for the dispatcher’s helpers: each carries the dispatcher’s configuration and logger, and stands alone with defaults for both.

Parameters:

config (dict) – The dispatcher’s configuration dictionary
logger (Logger) – An instance of the Logger class for logging

class romtools.hpc.components.LocalCaller(*, files, config=None, logger=None)[source]#

Bases: BaseCaller

Imports and runs the target in the current process.

A configured python_setup or python_command names an interpreter other than this one - a Python provided by a module, or a wrapper such as srun - so the call is staged out to a subprocess instead of imported here.

Parameters:

files (LocalFileManager) – The local file manager used to stage inputs and read results back
config (dict)
logger (Logger)

class romtools.hpc.components.LocalFileManager(*, config=None, logger=None)[source]#

Bases: BaseFileManager

Operates directly on the local filesystem.

Parameters:

config (dict)
logger (Logger)

get(remote_path, local_path)[source]#

Local ‘get’ is just a copy from remote_path to local_path.

Parameters:

remote_path (str)
local_path (str)

Return type:

None

np_savez(path, **arrays)[source]#

Write multiple arrays to a .npz file. The .npz file is written directly to the specified path.

Parameters:: path (str)
Return type:: None

put(local_path, remote_path)[source]#

Local ‘put’ is just a copy from local_path to remote_path.

Parameters:

local_path (str)
remote_path (str)

Return type:

None

read_text(path)[source]#

Return the contents of a text file.

Parameters:: path (str)
Return type:: str

remove_dir(path)[source]#

Remove a directory and everything under it, if it exists.

Parameters:: path (str)
Return type:: None

resolve_path(path=None)[source]#

Local paths are used as given; relative ones follow the process cwd.

Parameters:: path (str)
Return type:: str

class romtools.hpc.components.LocalTransferManager(*, files, config=None, logger=None, campaign_directory=None)[source]#

Bases: BaseTransferManager

Copies the configured upload patterns from the current directory into the run directory.

The work runs on this machine against these same files, so there is nothing to pack up and nothing to bring back: only upload does anything.

Parameters:

files (BaseFileManager)
config (dict)
logger (Logger)
campaign_directory (str)

upload(run_directory=None)[source]#

Put the configured upload patterns into the run directory.

Parameters:: run_directory (str)
Return type:: None

class romtools.hpc.components.RemoteCaller(connection, *, files, config=None, logger=None)[source]#

Bases: StagedCaller

Runs the target on a remote host over an SSH connection.

Parameters:

connection (Connection) – An established Connection to the remote host
files (RemoteFileManager) – The remote file manager used to stage inputs and read results back
config (dict)
logger (Logger)

class romtools.hpc.components.RemoteFileManager(connection, *, config=None, logger=None)[source]#

Bases: BaseFileManager

Operates on the remote filesystem over an SSH connection.

Relative paths are resolved against the configured remote root.

Parameters:

connection (Connection) – An established Connection to the remote host
config (dict) – The dispatcher’s configuration dictionary
logger (Logger) – An instance of the Logger class for logging

np_savez(path, **arrays)[source]#

Write multiple arrays to a .npz file, staged locally then uploaded.

Parameters:: path (str)
Return type:: None

read_text(path)[source]#

Return the contents of a remote text file.

Parameters:: path (str)
Return type:: str

remove_dir(path)[source]#

Remove a remote directory and everything under it.

Parameters:: path (str)
Return type:: None

resolve_path(path=None)[source]#

Resolve a remote path: absolute paths pass through, relative ones are taken under the remote root, and no path at all means the root itself.

Parameters:: path (str)
Return type:: str

class romtools.hpc.components.RemoteTransferManager(connection, *, files, config=None, logger=None, campaign_directory=None)[source]#

Bases: BaseTransferManager

Moves the files a campaign needs onto the remote host, and its results back, in both directions through a single tar.gz archive.

Parameters:

connection (Connection) – An established Connection to the remote host
campaign_directory (str) – The campaign directory, mirrored locally and remotely
files (RemoteFileManager) – The remote file manager, whose resolve_path decides what paths mean
config (dict)
logger (Logger)

collect_results()[source]#

Collect results from remote HPC runs.

Return type:: None

upload(run_directory=None)[source]#

Put the configured upload patterns into the run directory.

Parameters:: run_directory (str)
Return type:: None

class romtools.hpc.components.SlurmJobManager(run_cmd, *, files, config=None, logger=None, campaign_directory=None)[source]#

Bases: CampaignComponent

Generates and submits SLURM scripts, then polls the job until it finishes.

Serves both dispatchers, since sbatch/squeue/sacct are the same wherever they are issued.

Parameters:

run_cmd (Callable[[str], Result]) – Runs a shell command where the scheduler lives and returns a Result. Connection.run for a remote host, run_local_bash for this one.
campaign_directory (str) – The directory jobs run in when no run_directory is given
files (BaseFileManager) – The file manager for the machine the job runs on
config (dict)
logger (Logger)

submit(cmd=None, run_directory=None)[source]#

Generate, stage, and submit a SLURM script, returning the job ID.

Parameters:

cmd (str)
run_directory (str)

Return type:

str

wait(job_id)[source]#

Block until the SLURM job leaves the queue, then return its exit code in Python style (negative for a signal), or None if sacct never reported.

Parameters:: job_id (str)
Return type:: str

class romtools.hpc.components.StagedCaller(run_cmd, *, files, config=None, logger=None)[source]#

Bases: BaseCaller

Runs the target under a separate interpreter, through a runner script staged in a scratch directory beside the run directory.

Arguments and results travel as a JSON + .npz pair, so that interpreter needs no access to romtools; run_cmd and files decide which machine it lives on.

Parameters:

run_cmd – Callable that issues a shell command on the execution host
files (BaseFileManager) – The file manager used to stage inputs and read results back
config (dict)
logger (Logger)

Modules

`archive`
`call_runner`	The wire contract behind dispatcher.call(): resolving a "module:qualname" target, packing arguments and results into a JSON + .npz pair, and building the standalone runner script that executes the target under a remote interpreter.
`caller`
`component`	Shared construction for the helpers that the dispatchers compose.
`file_manager`	File operations for the dispatchers, in local and remote flavors.
`slurm`
`slurm_job_manager`	SLURM job submission and monitoring for the dispatchers.
`transfer_manager`	Moving a campaign's input files to where it runs, and its results back.