Streaming data

Warning

Experimental: Support for Streaming is Experimental and not fully-supported:

Note

Streaming features are not enabled by default in DSS. You need to manually activate them by going to Administration > Settings > Misc. > Other, check the “Enable streaming” box and restart DSS.

• Concepts
  • Streaming endpoints
  • Continuous recipe
  • Continuous activity
• Kafka
  • Connection setup
  • Message format
    • Single-value
    • JSON
    • Avro
  • Timestamp handling
• AWS SQS
  • Connection setup
  • Message format
    • Single-value
    • JSON
• HTTP Server-Sent Events
• Continuous sync
  • Partitioning
• Continuous Python
  • Reading from streaming endpoints
    • Reading from Kafka endpoints
    • Reading from SQS endpoints
    • Reading from HTTP SSE endpoints
  • Writing to streaming endpoints
    • Writing to Kafka endpoints
  • Writing to datasets
  • Offset management
• Streaming Spark Scala