Project class

Phenopype projects are composed of a directory tree in which each folder contains the copy or a link to a single raw image file. Each raw should have only one folder, so that result or intermediate output from different iterations of processing and analysis are stored side by side. The project root folder should be separate from the raw data, e.g. as a folder inside of your project folder:

import phenopype as pp

myproj = pp.Project(root_dir="my_project")
myproj.add_files(image_dir="my-data")
myproj.add_config(tag = "v1", template_path="templates\template1.yaml")
...

## after closing Python, the project can be accessed by simply using Project class again:
myproj = pp.Project(root_dir="\my_project\phenopype")

Above code creates a folder structure as follows:

my_project\
        data\                                                   # created automatically
                file1\                                          # add files to project using "pp.Project.add_files"
                        raw.jpg                                 # created by "pp.Project.add_files"
                        attributes.yaml                 # created by "pp.Project.add_files"
                        pype_config_v1.yaml             # added with "pp.Project.add_config"
                        results_v1.csv
                file2\
                ...
        ...
my-data
scripts
templates
template1.yaml
manuscript
figures
...

Important

Currently the only useful information contained in the project object (myproj) is a list of all directories inside the project’s directory tree. Always save your progressing using the appropriate functions in (Export).

class phenopype.main.Project(root_dir, load=True, overwrite=False, ask=True)

Bases: object

Initialize a phenopype project with a root directory path. Phenopype will create the project folder at the provided location.

Parameters:

  • rootdir (str) – path to root directory of the project where folder gets created

  • overwrite (bool, optional) – overwrite option, if a given root directory already exist (WARNING: also removes all folders inside)

Returns:

project – phenopype project

Return type:

project

add_files(image_dir, filetypes=['jpg', 'JPG', 'jpeg', 'JPEG', 'tif', 'png', 'bmp'], include=[], include_all=True, exclude=[], mode='copy', n_max=None, randomize=False, nested=False, image_format=None, recursive=False, overwrite=False, resize_factor=1, resize_max_dim=None, unique='path', **kwargs)

Add files to your project from a directory, can look recursively. Specify in- or exclude arguments, filetypes, duplicate-action and copy or link raw files to save memory on the harddrive. For each found image, a folder will be created in the “data” folder within the projects root directory. If found images are in subfolders and “recursive==True”, the respective phenopype directories will be created with flattened path as prefix.

E.g., with “raw_files” as folder with the original image files and “phenopype_proj” as rootfolder:

  • raw_files/file.jpg ==> phenopype_proj/data/file.jpg

  • raw_files/subdir1/file.jpg ==> phenopype_proj/data/1__subdir1__file.jpg

  • raw_files/subdir1/subdir2/file.jpg ==> phenopype_proj/data/2__subdir1__subdir2__file.jpg

Parameters:

  • image_dir (str) – path to directory with images

  • filetypes (list or str, optional) – single or multiple string patterns to target files with certain endings. “_vars.default_filetypes” are configured in _vars.py: [‘jpg’, ‘JPG’, ‘jpeg’, ‘JPEG’, ‘tif’, ‘png’, ‘bmp’]

  • include (list or str, optional) – single or multiple string patterns to target certain files to include

  • (optional) (include_all) – either all (True) or any (False) of the provided keywords have to match

  • exclude (list or str, optional) – single or multiple string patterns to target certain files to exclude - can overrule “include”

  • recursive ((optional): bool,) – “False” searches only current directory for valid files; “True” walks through all subdirectories

  • unique ({"file_path", "filename"}, str, optional:) – how to deal with image duplicates - “file_path” is useful if identically named files exist in different subfolders (folder structure will be collapsed and goes into the filename), whereas filename will ignore all similar named files after their first occurrence.

  • mode ({"copy", "mod", "link"} str, optional) – how should the raw files be passed on to the phenopype directory tree: “copy” will make a copy of the original file, “mod” will store a .tif version of the orginal image that can be resized, and “link” will only store the link to the original file location to attributes, but not copy the actual file (useful for big files, but the orginal location needs always to be available)

  • overwrite ({"file", "dir", False} str/bool (optional)) – “file” will overwrite the image file and modify the attributes accordingly, “dir” will overwrite the entire image directory (including all meta-data and results!), False will not overwrite anything

  • ext ({".tif", ".bmp", ".jpg", ".png"}, str, optional) – file extension for “mod” mode

  • resize_factor (float, optional)

  • kwargs – developer options

add_config(tag, template_path, subset=[], interactive=False, image_number=1, overwrite=False, **kwargs)

Add pype configuration presets to all image folders in the project, either by using the templates included in the presets folder, or by adding your own templates by providing a path to a yaml file. Can be tested and modified using the interactive flag before distributing the config files.

Parameters:

  • tag (str) – tag of config-file. this gets appended to all files and serves as and identifier of a specific analysis pipeline

  • template_path (str, optional) – path to a template or config-file in yaml-format

  • subset (list, optional) – provide a list of images or folders in the project where the config should be added. useful if you do not want to modify the config for all images

  • interactive (bool, optional) – start a pype and modify template before saving it to phenopype directories

  • interactive_image (str, optional) – to modify pype config in interactive mode, select image from list of images (directory names) already included in the project. special flag “first” is default and takes first image in “data” folder.

  • overwrite (bool, optional) – overwrite option, if a given pype config-file already exist

  • kwargs – developer options

add_model(model_path, model_id, model_config_path=None, model_type='segmentation', overwrite=False, copy=True, **kwargs)

Add a deep learning model.

Parameters:

  • overwrite (bool, optional) – overwrite option, if a given pype config-file already exist

  • template (bool, optional) – should a template for reference detection be created. with an existing template, phenopype can try to find a reference card in a given image, measure its dimensions, and adjust pixel-to-mm-ratio and colour space

add_reference_template(image_path, reference_id, template=True, overwrite=False, **kwargs)

Add pype configuration presets to all project directories.

Parameters:

  • image_path (str) – Path to the template image, either a file link, project directory, or int (idx in project directories).

  • reference_id (str) – Unique identifier for the reference.

  • activate (bool, optional) – Whether to activate this reference across the project.

  • template (bool, optional) – Whether to create a template for reference detection.

  • overwrite (bool, optional) – Whether to overwrite existing files if they exist.

add_reference(template)
check_files(feedback=True, image_links=False, new_dir=None, ret_missing=False)

Check all project files for completeness by comparing the images in the “data” folder to the file list the project attributes. Will attempt to fix discrepancies, but ask for feedback first.

Parameters:

  • feedback (bool, optional) – Asks whether project attributes should be updated. The default is True.

  • image_links (bool, optional) – checks whether image can be loaded from path, otherwise tries to load from original filepath (will ask first). The default is False.

Return type:

None.

collect_results(tag, files, folder='', aggregate_csv=True, overwrite=False, **kwargs)

Collect canvas from each folder in the project tree. Search by name/safe_suffix (e.g. “v1”).

Parameters:

  • tag (str) – pype tag / save_suffix

  • files (str) – list results (canvas, annotation-file, or csv-files) to be aggregated to a single directory at the project root

  • folder (str, optional {default: <annotation-name>_<tag> }) – folder in the root directory where the aggregated results are stored

  • aggregate_csv (bool, optional) – concatenate csv files of the same type and store in results folder instead of each file separately

  • overwrite (bool, optional) – should the results be overwritten

copy_tag(tag_src, tag_dst, copy_annotations=True, copy_config=True, copy_exports=False, overwrite=False, **kwargs)

Make a copy of data generated under a specific tag and save it under a new tag - e.g.:

annotations_v1.json ==> annotations_v2.json pype_config_v1.yaml ==> pype_config_v2.yaml

Parameters:

  • tag_src (str) – name of tag to be copied (source tag)

  • tag_dst (str) – name of new tag (destination tag)

  • copy_annotations (bool, optional) – copy annotations file. The default is True.

  • copy_config (bool, optional) – copy config file. The default is True.

  • copy_exports (bool, optional) – copy export files ending with tag_src + “.csv”. The default is False.

  • overwrite (bool, optional) – overwrites if tag exists. The default is False.

  • kwargs – developer options

Return type:

None.

edit_config(tag, target, replacement, subset=[], **kwargs)

Add or edit functions in all configuration files of a project. Finds and replaces single or multiline string-patterns. Ideally this is done via python docstrings that represent the parts of the yaml file to be replaced.

Parameters:

  • tag (str) – tag (suffix) of config-file (e.g. “v1” in “pype_config_v1.yaml”)

  • target (str) – string pattern to be replaced. should be in triple-quotes to be exact

  • replacement (str) – string pattern for replacement. should be in triple-quotes to be exact

  • subset (list, optional) – provide a list of images or folders in the project where the config should be modified. useful if you do not want to modify the config for all images

export_zip(tag=None, results=False, models=False, images=False, exports=False, save_dir=None, overwrite=True, **kwargs)
Parameters:

  • tag (str, optional) – Export only files associated with a specified tag. The default is None.

  • images (bool, optional) – Don’t include images from the data folder. The default is True.

Return type:

None.

export_training_data(tag, method, params={}, folder=None, annotation_id=None, overwrite=False, img_size=224, parameters=None, **kwargs)
Parameters:

  • tag (str) – The Pype tag from which training data should be extracted.

  • method ({"ml-morph"} str) –

    For which machine learning framwork should training data be created. Currently instructions for the following architectures are supported:

    • ”ml-morph” - Machine-learning tools for landmark-based morphometrics https://github.com/agporto/ml-morph

    • ”keras-cnn-semantic” - Images and masks to be used for training an image segmentation model in Keras

  • folder (str) – Name of the folder under “root/training_data” where the formatted training data will be stored under.

  • annotation_id (str, optional) – select a specific annotation id. The default is None.

  • overwrite (bool, optional) – Should an existing training data set be overwritten. The default is False.

Return type:

None.