Kerberos CLI Debug Tracing & Hadoop

Luke Hebert

- 2 minutes read - 295 words

When a Kerberos ticket exchange fails, the default error messages rarely tell you which step broke. Enabling verbose trace output on the command line gives you the full exchange log, which makes it far easier to pinpoint whether the problem is a DNS lookup, a clock skew, a missing keytab entry, or a service principal mismatch.

Enabling the Trace

Set the following environment variables in your shell before running the Hadoop or Kerberos command you want to debug:

export KRB5_TRACE=/dev/stderr
export HADOOP_ROOT_LOGGER=TRACE,console
export HADOOP_JAAS_DEBUG=true
export HADOOP_OPTS="-Dsun.security.krb5.debug=true -Dsun.security.jgss.debug=true"

What each variable does:

  • KRB5_TRACE=/dev/stderr - Sends the full Kerberos protocol exchange (AS-REQ, TGS-REQ, etc.) to stderr so you can read it in real time.
  • HADOOP_ROOT_LOGGER=TRACE,console - Raises the Hadoop log level to TRACE and writes it to the console.
  • HADOOP_JAAS_DEBUG=true - Enables JAAS (Java Authentication and Authorization Service) debug output, which shows which login module is being used and whether it succeeds.
  • HADOOP_OPTS - Passes JVM system properties that turn on the underlying sun.security.krb5 and sun.security.jgss debug flags.

Tips for Reading the Output

  1. Look for the first ERROR or Exception line; everything before it is usually the successful part of the exchange.
  2. If you see Clock skew too great, check that the client and KDC clocks are within 5 minutes of each other.
  3. If the trace shows a KRB5KDC_ERR_PRINCIPAL_NOT_FOUND, the service principal you are requesting does not exist in the KDC.
  4. DNS-related failures often appear as Cannot locate KDC for realm or a getaddrinfo error.

Turning It Off

Once you have the information you need, unset the variables or open a fresh shell so you do not flood production logs with trace-level output:

unset KRB5_TRACE HADOOP_ROOT_LOGGER HADOOP_JAAS_DEBUG HADOOP_OPTS

Keeping these settings scoped to a single debugging session keeps your logs readable and your performance unaffected.