romtools.hpc.dispatchers#

class romtools.hpc.dispatchers.BaseDispatcher(campaign_directory='hpctools', logger=None, config=None)[source]#

Bases: ABC

Shared configuration, logging, and public API for the dispatchers.

The public methods here are a flat facade over composed helpers, so that workflows and models depend on one stable interface rather than on how the work is split up internally. Subclasses install the local or remote implementation of each helper, and only override a method when the behavior genuinely differs.

Parameters:

campaign_directory (str)
logger (Logger)

call(target, *args, run_directory=None, **kwargs)[source]#

Run a Python callable named as “module:qualname” (e.g. “my_model:evaluate”) and return its result, forwarding the remaining arguments to it.

Modules staged in run_directory are importable; it defaults to the dispatcher’s own default directory.

Parameters:

target (str)
run_directory (str)

collect_results()[source]#

Bring a finished job’s output back to the local machine. Work that ran here leaves its results in place, so only remote dispatchers act.

Return type:: None

abstractmethod run(cmd, run_directory=None)[source]#

Run a command directly on the execution host.

A command that exits non-zero is reported, not raised: check result.ok or result.exit_code. Every dispatcher behaves this way, so a model can be moved between them unchanged.

Returns a Result object (with stdout, stderr, exit_code, ok)

Parameters:

cmd (str)
run_directory (str)

Return type:

Result

submit_job(cmd=None, run_directory=None)[source]#

Submit work to SLURM, wait for it to finish, and read back its output.

cmd runs from run_directory, which defaults to the campaign directory; omit cmd only when a configured SLURM script carries the command itself.

Returns a Result object (with stdout, stderr, exit_code, ok)

Parameters:

cmd (str)
run_directory (str)

Return type:

Result

upload(run_directory=None)[source]#

Put the configured upload patterns into run_directory, which defaults to the campaign directory, as submit_job() does.

Parameters:: run_directory (str)
Return type:: None

class romtools.hpc.dispatchers.LocalDispatcher(campaign_directory='hpctools', logger=None, config=None)[source]#

Bases: BaseDispatcher

Runs ROM workflows on the machine this process runs on.

Paths address the local filesystem and commands run in a local bash shell rather than over SSH. That machine may itself be a cluster node, in which case submit_job() reaches the scheduler directly and results need no transferring; upload() copies the configured files into the run directory rather than sending them anywhere.

Parameters:

campaign_directory (str) – The directory jobs run in when given no run_directory
logger (Logger) – An instance of the Logger class for logging
config – A YAML path or a Configuration. Defaults to reading the command line.

run(cmd, run_directory=None)[source]#

Run a command on the local machine, from run_directory if given and the current working directory otherwise.

Returns a Result object (with stdout, stderr, exit_code, ok)

Parameters:

cmd (str)
run_directory (str)

Return type:

Result

class romtools.hpc.dispatchers.RemoteDispatcher(campaign_directory='hpctools', logger=None, connection=None, config=None)[source]#

Bases: BaseDispatcher

Runs ROM workflows on a remote host over SSH, connecting as user@remote:port.

Coordinates composed helpers rather than doing the work itself: files go through RemoteFileManager, Python calls through RemoteCaller, batch jobs through SlurmJobManager, and archives through RemoteTransferManager. Paths are taken relative to the configured remote root.

Parameters:

logger (Logger) – An instance of the Logger class for logging
campaign_directory (str) – An optional string naming the directory this campaign runs in, relative to the remote root. It is mirrored locally and under the remote root.
config – A YAML path or a Configuration. Defaults to reading the command line.
connection (Connection | None)

run(cmd, run_directory=None)[source]#

Run a command directly on the remote host, without SLURM.

This lands wherever the SSH session does, which on a cluster is the login node, so it suits quick staging or preprocessing; anything long or parallel belongs in submit_job(). run_directory is relative to the remote root, and defaults to the root itself.

Returns a Result object (with stdout, stderr, exit_code, ok)

Parameters:

cmd (str)
run_directory (str)

Return type:

Result

romtools.hpc.dispatchers.resolve_dispatcher(dispatcher=None)[source]#

Fall back to local execution when the caller supplies no dispatcher.

Parameters:: dispatcher (BaseDispatcher | None)
Return type:: BaseDispatcher

romtools.hpc.dispatchers.resolve_local_dispatcher(dispatcher=None)[source]#

Return a local dispatcher, reusing the supplied one when it is already local.

Keeps in-process work on this machine even when the rest of the workflow is dispatched to a remote host.

Parameters:: dispatcher (BaseDispatcher | None)
Return type:: LocalDispatcher

Modules

`base_dispatcher`
`local_dispatcher`
`remote_dispatcher`
`resolve`	Helpers for resolving an optional dispatcher argument to a concrete dispatcher.