API for managed folders

This class lets you interact with managed folders in Python recipes and notebooks. See Managed folders for more information.

class dataiku.core.managed_folder.Folder(lookup, project_key=None, ignore_flow=False)

This is a handle to interact with a managed folder.

Note: this class is also available as dataiku.Folder

get_info(sensitive_info=False)

Get information about the location and settings of this managed folder :rtype: dict

get_partition_info(partition)

Get information about the partitions of this managed folder :rtype: dict

get_path()

Gets the filesystem path of this managed folder. This method can only be called for managed folders that are stored on the local filesystem of the DSS server.

For non-filesystem managed folders (HDFS, S3, …), you need to use the various read/download and write/upload APIs.

is_partitioning_directory_based()

Whether the partitioning of the folder is based on sub-directories

list_paths_in_partition(partition='')

Gets the filesystem paths of the folder for the given partition (or for the entire folder)

list_partitions()

Gets the partitions in the folder

Return type:list
get_partition_folder(partition)

Gets the filesystem path of the directory corresponding to the partition (if the partitioning is directory-based)

get_id()
get_name()
file_path(filename)

Gets the filesystem path for a given file within the folder. This method can only be called for managed folders that are stored on the local filesystem of the DSS server.

For non-filesystem managed folders (HDFS, S3, …), you need to use the various read/download and write/upload APIs.

Parameters:filename (str) – Name of the file within the folder
read_json(filename)

Reads a JSON file within the folder and returns its parsed content

Parameters:filename (str) – Path of the file within the folder
Return type:list or dict: Depending on the content of the file
write_json(filename, obj)

Writes a JSON-serializable (mostly dict or list) object as JSON to a file within the folder

Parameters:
  • filename (str) – Path of the target file within the folder
  • obj (str) – JSON-serializable object to write (generally dict or list)
clear()

Removes all files from the folder

clear_partition(partition)

Removes all files from a specific partition of the folder.

clear_path(path)

Removes a file or directory from the folder

get_path_details(path='/')

Get details about a specific path (file or directory) in the folder

Return type:dict
get_download_stream(path)

Gets a file-like object that allows you to read a single file from this folder. If the file already exists, it will be replaced.

with folder.get_download_stream("myfile") as stream:
    data = stream.readline()
    print("First line of myfile is: \%s" \% data)
Return type:file-like
upload_stream(path, f)

Uploads the content of a file-like object to a specific path in the managed folder. If the file already exists, it will be replaced.

# This copies a local file to the managed folder
with open("local_file_to_upload") as f:
    folder.upload_stream("name_of_file_in_folder", f)
Parameters:
  • path (str) – Target path of the file to write in the managed folder
  • f – file-like object open for reading
upload_file(path, file_path)

Uploads a local file to a specific path in the managed folder. If the file already exists, it will be replaced.

Parameters:
  • path (str) – Target path of the file to write in the managed folder
  • file_path – Absolute path to a local file
upload_data(path, data)

Uploads binary data to a specific path in the managed folder. If the file already exists, it will be replaced.

Parameters:
  • path (str) – Target path of the file to write in the managed folder
  • data – str or unicode data to upload
get_writer(path)

Get a writer object to write incrementally to a specific path in the managed folder. If the file already exists, it will be replaced.

Parameters:path (str) – Target path of the file to write in the managed folder
get_last_metric_values(partition='')

Get the set of last values of the metrics on this folder, as a dataiku.ComputedMetrics object

get_metric_history(metric_lookup, partition='')

Get the set of all values a given metric took on this folder :param metric_lookup: metric name or unique identifier :param partition: optionally, the partition for which the values are to be fetched

save_external_metric_values(values_dict)

Save metrics on this folder. The metrics are saved with the type “external”

Parameters:values_dict – the values to save, as a dict. The keys of the dict are used as metric names
save_external_check_values(values_dict)

Save checks on this folder. The checks are saved with the type “external”

Parameters:values_dict – the values to save, as a dict. The keys of the dict are used as check names