Advanced topics

Using log4j to send audit trail to Kafka

This is a non-supported example of how to configure the Log4J reporting target to send to Kafka

Apache Kafka is a distributed message queue, which can be used to get audit logs out of DSS

To enable audit queries to Kafka:

  • Add all jars from the Kafka distribution to the lib/java folder of your DSS datadir
  • Create or edit the resources/logging/dku-log4j.properties file (See Logging in DSS for more information)
  • Add the following lines
log4j.appender.AUDIT_KAFKA=org.apache.kafka.log4jappender.KafkaLog4jAppender
log4j.appender.AUDIT_KAFKA.BrokerList=kafka1:9092,kafk2:9092,kafka:9093
log4j.appender.AUDIT_KAFKA.Topic=dku-dss-audit
log4j.appender.AUDIT_KAFKA.layout=com.dataiku.dip.logging.JSONAuditLayout

log4j.logger.dku.audit= INFO, AUDIT_KAFKA
  • Replace dku-dss-audit by the name of your Kafka topic
  • Replace BrokerList by the bootstrap list of your Kafka brokers

Note

Your Kafka libraries must correspond to the version of your Kafka server