Logging and auditing¶
The API node outputs three kinds of logs:
- Regular runtime logs (in the
- Audit logs for the administration API
- Logs of queries
Logging of queries is especially important if you plan on implementing a feedback loop. Knowing what has been predicted for what records is important. You’ll also need to have a way to retrieve “what finally happened” for each record that the API node predicted (did this customer convert? churn? was it a fraud? did the sensor fail? ...)
- Administration API audit logs are written to the same
- Queries are logged to a log-rotated folder:
How to configure audit and query logging¶
Audit and query logging is done through the standard Java Log4J logging mechanism.
You can set the destination of these loggers by modifying the Log4J appenders in the
The loggers used for audit logging are:
- Logs all queries to prediction endpoints, in a JSON format. The log message includes the input features, the prediction results, and timing information
- Logs authentication failures, both on Admin and User APIs
- Logs all modifications done through the admin API. The log message includes details about the API key used to perform the call
- Logs basic information for all API calls, both Admin and User APIs. It is generally not recommended to enable this logger
Logging queries to Kafka¶
Apache Kafka is a distributed message queue, which can be used to get query logs out of the API node.
To enable logging queries to Kafka:
- Add all jars from the Kafka distribution to the
- Replace the “Queries logging” part of
bin/log4j.propertiesby the following snippet:
log4j.appender.QUERIES_KAFKA=org.apache.kafka.log4jappender.KafkaLog4jAppender log4j.appender.QUERIES_KAFKA.BrokerList=kafka1:9092,kafk2:9092,kafka:9093 log4j.appender.QUERIES_KAFKA.Topic=dku-apinode-audit log4j.appender.QUERIES_KAFKA.layout=com.dataiku.dip.logging.JSONAuditLayout log4j.additivity.dku.apinode.audit.queries=false log4j.logger.dku.apinode.audit.queries= INFO, QUERIES_KAFKA
You can also send administration and authentication audit logs to Kafka by setting appropriate configuration for the other audit loggers.