DynamicDL

class dynamicdl.dynamicdl.DynamicData(root: str, form: dict, bbox_scale_option: str = 'auto', seg_scale_option: str = 'auto', get_md5_hashes: bool = False, purge_duplicates: bool | None = None)

Bases: object

Main dataset class. Accepts root directory path and dictionary form of the structure. DynamicDL expands a generic dataset form and interprets it through a series of recursive hierarchical inheritances, to flatten the dataset into a list of entries fit for image processing.

Parameters:

• root (str) – The root directory to access the dataset.

• form (dict) – The form of the dataset. See documentation for further details on valid forms.

• bbox_scale_option (str) – Choose from either auto, zeroone, or full scale options to define, or leave empty for automatic. zeroone assumes detection coordinates to be interpreted on a 0-1 scale as ratios dependent on image size. full leaves coordinates as is. auto for auto-detection. Default: auto

• seg_scale_option (str) – Choose from either auto, zeroone, or full scale options to define, or leave empty for automatic. zeroone assumes segmentation coordinates to be interpreted on a 0-1 scale as ratios dependent on image size. full leaves coordinates as is. auto for auto-detection. Default: auto

• get_md5_hashes (bool) – When set to True, create a new column which finds md5 hashes for each image available, and makes sure there are no duplicates. Default: False

• purge_duplicates (Optional[bool]) – When set to True, remove all duplicate image entries. Duplicate images are defined by having the same md5 hash, so this has no effect when get_md5_hashes is False. When set to False, do not purge duplicates. Default: None

clear_image_sets(sets: list[str | int] | None = None) → None

Clear image sets from the dict if they contain no elements.

Parameters:

sets (list[str | int], Optional) – If defined, only scan the provided list, otherwise scan all sets. Default: None.

delete_image_set(image_set: str | int) → None

Delete image set from all entries. If an entry has only that image set, replace with the default dataset.

Parameters:

image_set (str | int) – The image set to delete. Accepts both name and ID.

get_dataloader(mode: str = 'inference', batch_size: int = 16, shuffle: bool = True, num_workers: int = 0, remove_invalid: bool = True, store_dim: bool = False, preset_transform: bool = True, calculate_stats: bool = True, mean: Tuple[float, ...] = (0.485, 0.456, 0.406), std: Tuple[float, ...] = (0.229, 0.224, 0.225), normalize: bool = True, image_set: int | str | None = None, transform: Callable | None = None, target_transform: Callable | None = None, transforms: tuple[Callable] | None = None, resize: tuple[int, int] | None = None, normalize_to: str | None = None) → DataLoader

Retrieve the PyTorch dataloader (torch.utils.data.DataLoader) for this dataset.

Parameters:

• mode (str) – The mode of training to select. See available modes with available_modes.

• batch_size (int) – The batch size of the image. Default: 16.

• shuffle (bool) – Whether to shuffle the data before loading. Default: True.

• num_workers (int) – Number of workers for the dataloader. Default: 0.

• remove_invalid (bool) – If set to True, deletes any NaN/corrupt items in the image set pertaining to the relevant mode. In the False case, either NaN values are substituted with empty values or an error is thrown, depending on the mode selected.

• store_dim (bool) – If set to True, the labels in the dataset will return a dict with two keys. label contains the standard PyTorch labels and dim contains the image’s former dimensions.

• preset_transform (bool) – Whether to use default preset transforms. Consists of normalization with either calculated mean of the dataset about to be used or standard ImageNet statistics depending on calculate_stats. Default: True

• calculate_stats (bool) – Whether to calculate mean and std for this dataset to be used in normalization transforms. If False, uses ImageNet default weights. Only has effect when preset_transform is set to True. Default: True

• mean (Tuple[float, ...]) – Default mean statistics for the dataset. Has no effect when calculate_stats = True. Default: ImageNet values.

• std (Tuple[float, ...]) – Default mean statistics for the dataset. Has no effect when calculate_stats = True. Default: ImageNet values.

• normalize (bool) – When set to True, normalize the dataset according to some mean/std values, either from calculated stats or ImageNet default. This statement is overriden when calculate_stats is set to T`rue. Default: True.

• image_set (Optional[str]) – The image set to pull from. Default: all images.

• transform (Optional[Callable]) – The transform operation to apply to the images.

• target_transform (Optional[Callable]) – The transform operation to apply to the labels.

• transforms (Optional[Tuple[Optional[Callable], ...]]) – Tuple in the format (transform, target_transform). Obtain default transforms from DynamicData.get_transforms(), or supply your own.

• resize (Optional[Tuple[int, ...]]) – If provided, resize all images to exact (width, height) configuration.

• normalize_to (Optional[str]) – If provided, normalize bounding box/segmentation coordinates to a specific configuration. Options: ‘zeroone’, ‘full’

get_dataset(mode: str = 'inference', remove_invalid: bool = True, store_dim: bool = False, preset_transform: bool = True, calculate_stats: bool = True, mean: Tuple[float, ...] = (0.485, 0.456, 0.406), std: Tuple[float, ...] = (0.229, 0.224, 0.225), normalize: bool = True, image_set: int | str | None = None, transform: Callable | None = None, target_transform: Callable | None = None, transforms: tuple[Callable] | None = None, resize: tuple[int, int] | None = None, normalize_to: str | None = None) → DynamicDS

Retrieve the PyTorch dataset (torch.utils.data.Dataset) of a specific mode and image set.

Parameters:

• mode (str) – The mode of training to select. See available modes with available_modes.

• remove_invalid (bool) – If set to True, deletes any NaN/corrupt items in the image set pertaining to the relevant mode. In the False case, either NaN values are substituted with empty values or an error is thrown, depending on the mode selected.

• store_dim (bool) – If set to True, the labels in the dataset will return a dict with two keys. label contains the standard PyTorch labels and dim contains the image’s former dimensions.

• preset_transform (bool) – Whether to use default preset transforms. Consists of normalization with either calculated mean of the dataset about to be used or standard ImageNet statistics depending on calculate_stats. Default: True

• calculate_stats (bool) – Whether to calculate mean and std for this dataset to be used in normalization transforms. If False, uses ImageNet default weights. Only has effect when preset_transform is set to True. Default: True

• mean (Tuple[float, ...]) – Default mean statistics for the dataset. Has no effect when calculate_stats = True. Default: ImageNet values.

• std (Tuple[float, ...]) – Default mean statistics for the dataset. Has no effect when calculate_stats = True. Default: ImageNet values.

• normalize (bool) – When set to True, normalize the dataset according to some mean/std values, either from calculated stats or ImageNet default. This statement is overriden when calculate_stats is set to True. Default: True.

• image_set (Optional[str]) – The image set to pull from. Default: all images.

• transform (Optional[Callable]) – The transform operation to apply to the images.

• target_transform (Optional[Callable]) – The transform operation to apply to the labels.

• transforms (Optional[Tuple[Optional[Callable], ...]]) – Tuple in the format (transform, target_transform). Obtain default transforms from DynamicData.get_transforms(), or supply your own.

• resize (Optional[Tuple[int, ...]]) – If provided, resize all images to exact (width, height) configuration.

• normalize_to (Optional[str]) – If provided, normalize bounding box/segmentation coordinates to a specific configuration. Options: ‘zeroone’, ‘full’

get_image_set(image_set: str | int) → DataFrame

Retrieve the sub-DataFrame which contains all images in a specific image set.

Parameters:

image_set (str | int) – The image set. Accepts both string and int.

get_transforms(mode: str = 'inference', calculate_stats: bool = True, mean: Tuple[float, ...] = (0.485, 0.456, 0.406), std: Tuple[float, ...] = (0.229, 0.224, 0.225), normalize: bool = True, remove_invalid: bool = True, resize: Tuple[int, ...] | None = None) → Tuple[Callable | None, ...]

Retrieve the default standard image/label transforms for specified mode.

Parameters:

• mode (str) – Choose a mode out of available modes for the transforms. Each mode has slightly altered standard transforms.

• calculate_stats (bool) – When set to True, calculate the statistics for the entire dataset, overriding default mean/std kwarg (defaults from ImageNet). Use this feature when dataset mean/std is unknown and differs significantly from ImageNet. Default: True.

• mean (Tuple[float, ...]) – Default mean statistics for the dataset. Has no effect when calculate_stats = True. Default: ImageNet values.

• std (Tuple[float, ...]) – Default mean statistics for the dataset. Has no effect when calculate_stats = True. Default: ImageNet values.

• normalize (bool) – When set to True, normalize the dataset according to some mean/std values, either from calculated stats or ImageNet default. This statement is overriden when calculate_stats is set to True. Default: True.

• remove_invalid (bool) – Remove invalid entries when calculating the statistics, assuming calculate_stats is set to True. If calculate_stats is False, this value has no effect. Default: True.

• resize (Optional[Tuple[int, ...]]) – Resize to specific tuple dimensions before calculating statistics, only when calculate_stats is set to True, just like remove_invalid. Default: None.

Returns:

A tuple of two callable transforms, the first being the standard image transform and the latter being the standard target transform.

Return type:

Tuple[Optional[Callable], …]

classmethod load(filename: str = '') → Self

Load a DynamicData object from file. Warning: do not load any json files that you did not create. This method uses jsonpickle, an insecure loading system with potential for arbitrary Python code execution.

Parameters:

filename (str) – The filename to load the data from.

parse(override: bool = False, verbose: bool = False) → None

Must be called to instantiate the data in the dataset instance. Performs the recursive populate_data algorithm and creates the dataframe, and then cleans up the data.

Parameters:

• override (bool) – Whether to overwrite existing data if it has already been parsed and cleaned. Default: False

• verbose (bool) – Whether to show more details about the merging process. May have an impact on runtime. Default: False

sample_image(dpi: float = 1200, mode: str | list[str] | None = None, idx: int | None = None) → None

Sample an image from the dataset.

Parameters:

• dpi (float) – The image display size, if not in segmentation mode.

• mode (Optional[str | list[str]]) – Pick from any of the available modes, or supply a list of modes. Default: all modes.

• idx (Optional[int]) – Use a specific idx from the dataset. Default: a random image.

save(filename: str = '', overwrite: bool = False, safe: bool = True) → None

Save the dataset into DynamicData json format.

Parameters:

• filename (str) – The filename to save the dataset.

• overwrite (bool) – Whether to overwrite the file if it already exists. Default: False.

• safe (bool) – If True, do not encode form with jsonpickle. Then dataset cannot be re-parsed, but is no longer subject to arbitrary code injection upon load.

split_image_set(image_set: str | int, *new_sets: tuple[str, float], inplace: bool = False, seed: int | None = None) → None

Split the existing image set into new image sets. If inplace is True, the existing image set will receive the percentage that is missing from the rest of the sets, or deleted if the other sets add up to 1.

Parameters:

• image_set (str | int) – The old image set name to split. Accepts both name and ID.

• new_sets (Tuple[str, float]) – Each entry of new_sets has a name for the set accompanied with a float to represent the percentage to split data into.

• inplace (bool) – Whether to perform the operation inplace on the existing image set. If False, then the new sets are required to add up to exactly 100% of the compositions. If True, any remaining percentages less than 100% will be filled back into the old image set. Default: False.

• seed (Optional[int]) – The seed to use for the operation, in case consistent dataset manipulation in memory is required. Default: None

class dynamicdl.dynamicds.DynamicDS(df: DataFrame, root: str, mode: str, id_mapping: dict[int, int] | None, image_type: str = 'RGB', normalization: str = 'full', store_dim: bool = False, resize: tuple[int, int] | None = None, normalize_to: str | None = None, transform: Callable | None = None, target_transform: Callable | None = None)

Bases: VisionDataset

Dataset implementation for the DynamicData environment.

Parameters:

• df (DataFrame) – The dataframe from DynamicData.

• root (str) – The root of the dataset folder.

• mode (str) – The mode of the data to retrieve, i.e. classification, segmentation, etc.

• id_mapping (dict[int, int]) – The id mapping from the dataframe to retrieve class names. This is used primarily as a safety feature in order to make sure that used IDs are provided in order starting from 0 without holes so that training works properly.

• image_type (str) – The type of the image to export, to convert PIL images to. Default: ‘RGB’. Also accepts ‘L’ and ‘CMYK’.

• normalization (str) – The type of normalization that the dataset currently is formatted in, for box and polygon items. Accepts ‘full’ or ‘zeroone’.

• normalize_to (str) – The type of normalization that the dataset is to be resized to, for box and polygon items. Accepts ‘full’ or ‘zeroone’.

• transform (Optional[Callable]) – The transform operation to apply to the images.

• target_transform (Optional[Callable]) – The transform operation on the labels.