pypers.batch

class pypers.batch.BatchLoader(task_cls: Type[Task], task_loader: TaskLoader, inject: dict = {}, override: dict = {})

Bases: object

Loads all tasks from a given directory (see Task specification).

Parameters:

inject – Dictionary of task specification settings which are to be added (before parsing)
override – Dictionary of task specification settings which are to be overwritten (after parsing).

load(path): Loads all task from the root directory path.

task(path)

property task_list

class pypers.batch.JSONLoader

Bases: TaskLoader

dump(filepath, data)

load(filepath)

suffix_hint = 'json'

class pypers.batch.StatusReport(scheduled_tasks, filepath=None)

Bases: object

get_task_status(task)

save()

update(task, status, save=True)

class pypers.batch.Task(loader, path, data, parent_task=None)

Bases: object

Batch processing task that bundles one or muiltiple input files to be processed using the same set of manaully set hyperparameters (see Task specification).

Parameters:

path – The path of the directory where the task specification resides.
data – Dictionary corresponding to the task specification (JSON data).
parent_task – The parent task or None if there is no parent task.

property available_outputs

cleanup(dry)

property config_digest: Hash code of the hyperparameters of this task.

create_pipeline(dry)

find_best_pickup_candidate(pipeline)

find_parent_task_with_result()

find_pickup_candidates(pipeline)

find_runnable_parent_task()

get_marginal_fields(pipeline)

Get the marginal fields from the given pipeline.

The marginal fields are all outputs produced by marginal stages. Marginal stages are those stages, for which the is_stage_marginal() method returns True.

Args:: pipeline (Pipeline): The pipeline object.
Returns:: set: A set of marginal fields.

property is_pending: True if the task needs to run, and False if the task is completed or not runnable.

is_stage_marginal(stage)

Check if a given stage is marginal.

Args:: stage: The stage to check.
Returns:: True if the stage is marginal, False otherwise.

outputs = []

pickup_previous_task(pipeline, dry=False, pickup=True, out=None)

reset()

resolve_path(path)

property root_path: The root path of the task (see Batch processing).

run(task_info=None, dry=False, verbosity=0, force=False, one_shot=False, report=None, pickup=True, return_full_data=True, out=None)

write_results(pipeline, data, timings, out=None)

class pypers.batch.TaskLoader

Bases: object

property default_suffix: str

dump(filepath: Union[Path, str], data: dict)

filepath(filepath: Union[Path, str]) → Union[Path, str]

load(filepath: Union[Path, str]) → dict

suffix_hint = ''

property suffixes: List[str]

class pypers.batch.YAMLLoader

Bases: TaskLoader

dump(filepath, data)

load(filepath)

suffix_hint = 'yml,yaml'

pypers.batch.resolve_pathpattern(pathpattern: Union[Path, str], fileid: Any) → str

Resolves a path pattern into a string by substituting a file ID into the pattern.

If the path pattern is None, the function returns None. Otherwise, the function converts the path pattern to a string and substitutes the file ID into the pattern using the ‘%’ operator.

Parameters:

pathpattern (PathType) – The path pattern to resolve. This should be a string containing one ‘%s’ placeholder where the file ID should be substituted.
fileid (Any) – The file ID to substitute into the path pattern.

Returns:

The resolved path pattern, or None if the input path pattern was None.

Return type:

str

pypers.batch.run_cli(task_cls, task_loader=<pypers.batch.JSONLoader object>, parser=None)