romtools.hpc.components.transfer_manager#

Moving a campaign’s input files to where it runs, and its results back.

Classes

`BaseTransferManager`(*, files[, config, ...])	Stages the files a campaign needs into its run directory, and brings its results back afterwards.
`LocalTransferManager`(*, files[, config, ...])	Copies the configured upload patterns from the current directory into the run directory.
`RemoteTransferManager`(connection, *, files)	Moves the files a campaign needs onto the remote host, and its results back, in both directions through a single tar.gz archive.

class romtools.hpc.components.transfer_manager.BaseTransferManager(*, files, config=None, logger=None, campaign_directory=None)[source]#

Bases: CampaignComponent

Stages the files a campaign needs into its run directory, and brings its results back afterwards.

Nothing moves by default: a dispatcher whose work already runs against the files in place needs neither half. Subclasses override whichever half genuinely has work to do.

Parameters:

files (BaseFileManager) – The file manager whose resolve_path decides what paths mean
campaign_directory (str) – The campaign directory results are gathered into
config (dict) – The dispatcher’s configuration dictionary
logger (Logger) – An instance of the Logger class for logging

Raises:

ValueError – if the configured collect or upload patterns are invalid.

collect_results()[source]#

Gather a finished job’s output into the local campaign directory.

Return type:: None

upload(run_directory=None)[source]#

Put the configured upload patterns into the run directory.

Parameters:: run_directory (str)
Return type:: None

class romtools.hpc.components.transfer_manager.LocalTransferManager(*, files, config=None, logger=None, campaign_directory=None)[source]#

Bases: BaseTransferManager

Copies the configured upload patterns from the current directory into the run directory.

The work runs on this machine against these same files, so there is nothing to pack up and nothing to bring back: only upload does anything.

Parameters:

files (BaseFileManager)
config (dict)
logger (Logger)
campaign_directory (str)

upload(run_directory=None)[source]#

Put the configured upload patterns into the run directory.

Parameters:: run_directory (str)
Return type:: None

class romtools.hpc.components.transfer_manager.RemoteTransferManager(connection, *, files, config=None, logger=None, campaign_directory=None)[source]#

Bases: BaseTransferManager

Moves the files a campaign needs onto the remote host, and its results back, in both directions through a single tar.gz archive.

Parameters:

connection (Connection) – An established Connection to the remote host
campaign_directory (str) – The campaign directory, mirrored locally and remotely
files (RemoteFileManager) – The remote file manager, whose resolve_path decides what paths mean
config (dict)
logger (Logger)

collect_results()[source]#

Collect results from remote HPC runs.

Return type:: None

upload(run_directory=None)[source]#

Put the configured upload patterns into the run directory.

Parameters:: run_directory (str)
Return type:: None