spark

Type Members

1. trait DataikuCustomRecipeSparkContext extends DataikuSparkContext

   

   Additional methods provided on the DataikuSparkContext for Plugin Spark-Scala recipe

2. trait DataikuSparkContext extends AnyRef

   

   API entry point to query DSS objects.

   API entry point to query DSS objects. Obtain one with DataikuSparkContext.getContext(sparkContext)

3. case class FSBrowsePath(truncated: Boolean, name: String, fullPath: String, pathElts: List[String], exists: Boolean, directory: Boolean, size: Long, children: List[FSBrowsePath]) extends Product with Serializable

   

4. case class FlowInput(fullName: String, smartName: String, kind: FlowComputableType, partitions: List[String], label: String, role: String) extends Product with Serializable

   

   Descriptor of an input of this recipe.

   Descriptor of an input of this recipe.

   fullName

   Full name of this input projectKey.itemId

   smartName

   “Smart” name of this input (relative to the project, prefixed only if in another project)

   kind

   Type of object

   partitions

   Input partitions if this item is a dataset. Null if whole dataset / all partitions (or not a dataset).

   label

   Item label (displayed name, e.g. for Managed Folders).

   role

   Input role of this item.

5. case class FlowOutput(fullName: String, smartName: String, kind: FlowComputableType, partition: String, label: String, role: String, writeMode: WriteMode) extends Product with Serializable

   

   Descriptor of an output of this recipe.

   Descriptor of an output of this recipe.

   fullName

   Full name of this output projectKey.itemId

   smartName

   “Smart” name of this output (relative to the project, prefixed only if in another project)

   kind

   Type of object

   partition

   Output partition if this item is a dataset. Null if whole dataset / all partitions (or not a dataset).

   label

   Item label (displayed name, e.g. for Managed Folders).

   role

   Output role of this item.

   writeMode

   Write mode (Overwrite or Append) for this output.

6. case class ManagedFolderInfo(projectKey: String, id: String, name: String, directoryBasedPartitioning: Boolean, fsType: String, path: String, accessInfo: Map[String, String]) extends Product with Serializable

   

   for the details of a path in a folder

7. trait ModelParameters extends AnyRef

   

8. trait MultiTransformer extends Serializable