romtools.hpc.components.file_manager#

File operations for the dispatchers, in local and remote flavors.

Classes

`BaseFileManager`(*[, config, logger])	Reads and writes files on whichever machine a dispatcher runs work on.
`LocalFileManager`(*[, config, logger])	Operates directly on the local filesystem.
`RemoteFileManager`(connection, *[, config, ...])	Operates on the remote filesystem over an SSH connection.

class romtools.hpc.components.file_manager.BaseFileManager(*, config=None, logger=None)[source]#

Bases: Component, ABC

Reads and writes files on whichever machine a dispatcher runs work on.

Parameters:

config (dict)
logger (Logger)

abstractmethod remove_dir(path)[source]#

Remove a directory and its contents. A missing directory is not an error.

Parameters:: path (str)
Return type:: None

class romtools.hpc.components.file_manager.LocalFileManager(*, config=None, logger=None)[source]#

Bases: BaseFileManager

Operates directly on the local filesystem.

Parameters:

config (dict)
logger (Logger)

get(remote_path, local_path)[source]#

Local ‘get’ is just a copy from remote_path to local_path.

Parameters:

remote_path (str)
local_path (str)

Return type:

None

np_savez(path, **arrays)[source]#

Write multiple arrays to a .npz file. The .npz file is written directly to the specified path.

Parameters:: path (str)
Return type:: None

put(local_path, remote_path)[source]#

Local ‘put’ is just a copy from local_path to remote_path.

Parameters:

local_path (str)
remote_path (str)

Return type:

None

read_text(path)[source]#

Return the contents of a text file.

Parameters:: path (str)
Return type:: str

remove_dir(path)[source]#

Remove a directory and everything under it, if it exists.

Parameters:: path (str)
Return type:: None

resolve_path(path=None)[source]#

Local paths are used as given; relative ones follow the process cwd.

Parameters:: path (str)
Return type:: str

class romtools.hpc.components.file_manager.RemoteFileManager(connection, *, config=None, logger=None)[source]#

Bases: BaseFileManager

Operates on the remote filesystem over an SSH connection.

Relative paths are resolved against the configured remote root.

Parameters:

connection (Connection) – An established Connection to the remote host
config (dict) – The dispatcher’s configuration dictionary
logger (Logger) – An instance of the Logger class for logging

np_savez(path, **arrays)[source]#

Write multiple arrays to a .npz file, staged locally then uploaded.

Parameters:: path (str)
Return type:: None

read_text(path)[source]#

Return the contents of a remote text file.

Parameters:: path (str)
Return type:: str

remove_dir(path)[source]#

Remove a remote directory and everything under it.

Parameters:: path (str)
Return type:: None

resolve_path(path=None)[source]#

Resolve a remote path: absolute paths pass through, relative ones are taken under the remote root, and no path at all means the root itself.

Parameters:: path (str)
Return type:: str