Dump Kafka Atlas Topic to Review Messages

When Atlas hook messages start misbehaving, the fastest way to see what is actually flowing through the pipeline is to dump the ATLAS_HOOK topic to a local file. The steps below cover the Kerberos and SASL configuration you need, the consumer command to capture messages, and a quick lag check to confirm the consumer group is keeping up.

Prerequisites

• You have an authorized Kerberos principal for the Kafka cluster.
• The kafka-console-consumer and kafka-consumer-groups CLI tools are available on your PATH.
• You know the fully qualified hostname of the Kafka broker (the examples below use hostname -f).

Create the Kafka SASL Config

Write a minimal properties file that tells the consumer to use SASL over plaintext:

echo 'security.protocol=SASL_PLAINTEXT' > /var/tmp/kafka.config

Create the JAAS Login Config

Kerberos authentication for the Kafka client requires a JAAS file. Create /var/tmp/kafka_jaas.conf with the following content:

KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
        useTicketCache=true
        renewTicket=true
        serviceName="kafka";
};

Authenticate and Dump the Topic

Obtain a Kerberos ticket, point the JVM at your JAAS file, and run the console consumer from the beginning of the topic:

kinit <an authorized user>
export KAFKA_OPTS='-Djava.security.auth.login.config=/var/tmp/kafka_jaas.conf'
kafka-console-consumer \
  --bootstrap-server $(hostname -f):9092 \
  --topic ATLAS_HOOK \
  --consumer.config /var/tmp/kafka.config \
  --from-beginning \
  > /tmp/atlas_hook.log

The output lands in /tmp/atlas_hook.log. You can tail -f that file while the consumer is running, or stop the consumer with Ctrl+C and review the full capture.

Check Consumer Group Lag

To confirm the atlas consumer group is not falling behind, run:

kafka-consumer-groups \
  --bootstrap-server $(hostname -f):9092 \
  --describe \
  --group atlas \
  --command-config /var/tmp/kafka.config

Look at the LAG column. A steadily growing lag means the consumer is not keeping up with the producer, which can point to a downstream processing issue in Atlas or a resource constraint on the consumer side.

• Bigdata

• Kerberos
• SysAdmin
• Troubleshooting