API for custom FS providers

class dataiku.fsprovider.FSProvider(root, config, plugin_config)

The base interface for a Custom FS provider

Parameters:
  • root – the root path for this provider
  • config – the dict of the configuration of the object
  • plugin_config – contains the plugin settings
close()

Perform any necessary cleanup

stat(path)

Get the info about the object at the given path

Parameters:path – where the object to inspect is located
Returns:a dict with the fields:
  • ’path’ : the location of the object (relative to the root this instance was created with)
  • ’isDirectory’ : True if the object is a folder
  • ’size’ : size of the object in bytes, 0 for folders
  • ’lastModified’ : modification time in ms since epoch, -1 if not defined

If there is no object at the given location, return None

set_last_modified(path, last_modified)

Change the modification time of an object.

Parameters:
  • path – where the object to modify is located.
  • last_modidied – timestamp as ms since epoch
Returns:

True if the change was done, False if not or if the operation is not supported

browse(path)

Enumerate files non-recursively from a path

Parameters:path – what to enumerate
Returns:a dict with the fields:
  • ’fullPath’ : the path from the root this instance was created with
  • ’exists’ : True if there is something at path
  • ’directory’ : True if the path denotes a folder
  • ’size’ : the size of the file at path; 0 if it’s a folder

If the object at path is a folder, then there should be a ‘children’ attribute which is a list of dicts with the same fields (without a ‘children’ field for subfolders)

enumerate(prefix, first_non_empty)

Enumerate files recursively from a path.

Parameters:
  • prefix – where to start the enumeration
  • first_non_empty – if first_non_empty, stop at the first non-empty file.
Returns:

a list of dicts corresponding to the enumerated files (not folders). Each dict is expected to contain these fields:

  • ’path’ : the path relative to the root this instance was created with,
  • ’size’ : size in bytes
  • ’lastModified’ : modification time in ms since epoch (-1 if not defined)

If there is nothing at the prefix, not even empty folders, return None

delete_recursive(path)
Delete recursively
Parameters:path – path to the folder or file to remove
move(from_path, to_path)

Move (rename) an object

Parameters:
  • from_path – where the data to move is located (relative to the root this instance was created with)
  • to_path – the target path for the data
read(path, stream, limit)

Read data

Parameters:
  • path – where to read the data (relative to the root this instance was created with)
  • stream – a file-like to write the data into
  • limit – if not -1, max number of bytes needed. Any more bytes will be discarded by the backend
write(path, stream)

Write data

Parameters:
  • path – where to write the data (relative to the root this instance was created with)
  • stream – a file-like to read the data from