romtools.hpc#

Tools for running ROM workflows on HPC resources.

Workflows take a dispatcher, which decides where model evaluations run: LocalDispatcher keeps them in place, RemoteDispatcher sends them to a cluster over SSH and brings the results back.

class romtools.hpc.BaseDispatcher(campaign_directory='hpctools', logger=None, config=None)[source]#

Bases: ABC

Shared configuration, logging, and public API for the dispatchers.

The public methods here are a flat facade over composed helpers, so that workflows and models depend on one stable interface rather than on how the work is split up internally. Subclasses install the local or remote implementation of each helper, and only override a method when the behavior genuinely differs.

Parameters:

campaign_directory (str)
logger (Logger)

call(target, *args, run_directory=None, **kwargs)[source]#

Run a Python callable named as “module:qualname” (e.g. “my_model:evaluate”) and return its result, forwarding the remaining arguments to it.

Modules staged in run_directory are importable; it defaults to the dispatcher’s own default directory.

Parameters:

target (str)
run_directory (str)

collect_results()[source]#

Bring a finished job’s output back to the local machine. Work that ran here leaves its results in place, so only remote dispatchers act.

Return type:: None

abstractmethod run(cmd, run_directory=None)[source]#

Run a command directly on the execution host.

A command that exits non-zero is reported, not raised: check result.ok or result.exit_code. Every dispatcher behaves this way, so a model can be moved between them unchanged.

Returns a Result object (with stdout, stderr, exit_code, ok)

Parameters:

cmd (str)
run_directory (str)

Return type:

Result

submit_job(cmd=None, run_directory=None)[source]#

Submit work to SLURM, wait for it to finish, and read back its output.

cmd runs from run_directory, which defaults to the campaign directory; omit cmd only when a configured SLURM script carries the command itself.

Returns a Result object (with stdout, stderr, exit_code, ok)

Parameters:

cmd (str)
run_directory (str)

Return type:

Result

upload(run_directory=None)[source]#

Put the configured upload patterns into run_directory, which defaults to the campaign directory, as submit_job() does.

Parameters:: run_directory (str)
Return type:: None

class romtools.hpc.Configuration(config_path=None)[source]#

Bases: object

Handles parsing a yaml file and any supplied command-line args.

Precedence:

CLI args (overwrite YAML)
YAML file values (if provided)
Class defaults

Parameters:: config_path (str | None) – Path to the YAML file to load. When omitted, the path is taken from -c or –hpc-config on this program’s command line.

classmethod defaults()[source]#

Schema defaults, reading neither a YAML file nor the command line.

Return type:: Configuration

exception romtools.hpc.ConfigurationError[source]#

Bases: Exception

Raised when the command line cannot be parsed against SCHEMA.

class romtools.hpc.Connection(host, user=None, port=None, persist_seconds=600)[source]#

Bases: object

A simple SSH connection class that uses subprocess to execute SSH and SCP commands.

This uses connection multiplexing (ControlMaster) to reuse an existing connection per SSH session (thus only requiring authentication & handshakes once).

close()[source]#: Close the SSH connection

get(remote, local=None)[source]#: Download a file from the remote host

local(cmd)[source]#: Execute a command locally

put(local, remote=None)[source]#: Upload a file to the remote host

run(command)[source]#: Execute a command on the remote host

class romtools.hpc.LocalDispatcher(campaign_directory='hpctools', logger=None, config=None)[source]#

Bases: BaseDispatcher

Runs ROM workflows on the machine this process runs on.

Paths address the local filesystem and commands run in a local bash shell rather than over SSH. That machine may itself be a cluster node, in which case submit_job() reaches the scheduler directly and results need no transferring; upload() copies the configured files into the run directory rather than sending them anywhere.

Parameters:

campaign_directory (str) – The directory jobs run in when given no run_directory
logger (Logger) – An instance of the Logger class for logging
config – A YAML path or a Configuration. Defaults to reading the command line.

run(cmd, run_directory=None)[source]#

Run a command on the local machine, from run_directory if given and the current working directory otherwise.

Returns a Result object (with stdout, stderr, exit_code, ok)

Parameters:

cmd (str)
run_directory (str)

Return type:

Result

class romtools.hpc.Logger(debug=False)[source]#

Bases: object

Prints dispatcher output, tagging remote messages with the hostname.

Messages logged through debug() appear only when debug logging is enabled.

Parameters:: debug (bool)

class romtools.hpc.RemoteDispatcher(campaign_directory='hpctools', logger=None, connection=None, config=None)[source]#

Bases: BaseDispatcher

Runs ROM workflows on a remote host over SSH, connecting as user@remote:port.

Coordinates composed helpers rather than doing the work itself: files go through RemoteFileManager, Python calls through RemoteCaller, batch jobs through SlurmJobManager, and archives through RemoteTransferManager. Paths are taken relative to the configured remote root.

Parameters:

logger (Logger) – An instance of the Logger class for logging
campaign_directory (str) – An optional string naming the directory this campaign runs in, relative to the remote root. It is mirrored locally and under the remote root.
config – A YAML path or a Configuration. Defaults to reading the command line.
connection (Connection | None)

run(cmd, run_directory=None)[source]#

Run a command directly on the remote host, without SLURM.

This lands wherever the SSH session does, which on a cluster is the login node, so it suits quick staging or preprocessing; anything long or parallel belongs in submit_job(). run_directory is relative to the remote root, and defaults to the root itself.

Returns a Result object (with stdout, stderr, exit_code, ok)

Parameters:

cmd (str)
run_directory (str)

Return type:

Result

class romtools.hpc.Result(stdout, stderr, exit_code)[source]#

Bases: object

Simple class to encapsulate the result of a command execution.

romtools.hpc.resolve_dispatcher(dispatcher=None)[source]#

Fall back to local execution when the caller supplies no dispatcher.

Parameters:: dispatcher (BaseDispatcher | None)
Return type:: BaseDispatcher

romtools.hpc.resolve_local_dispatcher(dispatcher=None)[source]#

Return a local dispatcher, reusing the supplied one when it is already local.

Keeps in-process work on this machine even when the rest of the workflow is dispatched to a remote host.

Parameters:: dispatcher (BaseDispatcher | None)
Return type:: LocalDispatcher

Modules

`components`	Focused helpers that the dispatchers compose.
`configuration`
`connection`
`dispatchers`
`logger`