Kerberos Debug Mode for the JVM, Spark, and SPNEGO

Luke Hebert

- 2 minutes read - 369 words

Kerberos authentication failures are notoriously hard to diagnose because the JVM logs its Kerberos exchange to stdout with no timestamps, while your application logs go to stderr. The steps below show you how to turn on Kerberos debug output for a plain JVM, a Spark driver, and Spark executors, and how to merge that output with your Spark logs so you can actually follow the sequence of events.

Debug a Plain JVM

Add the following system properties to your Java command line:

-Dsun.security.krb5.debug=true -Dsun.security.jgss.debug=true -Dsun.security.spnego.debug=true

If you are running Hadoop, you can also set the environment variables:

export HADOOP_ROOT_LOGGER=TRACE,console
export HADOOP_JAAS_DEBUG=true
export HADOOP_OPTS="-Dsun.security.krb5.debug=true"

Debug Spark

The generic Java property for enabling Kerberos debug logging is sun.security.krb5.debug=true. For Spark, you need to pass this property to the Driver and/or the Executor JVMs separately using Spark’s extra Java options.

Spark Driver:

spark.driver.extraJavaOptions=-Dsun.security.krb5.debug=true

Spark Executors:

spark.executor.extraJavaOptions=-Dsun.security.krb5.debug=true

These properties can be set on the spark-submit command line. For example:

spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master yarn \
  --deploy-mode cluster \
  --conf "spark.driver.extraJavaOptions=-Dsun.security.krb5.debug=true" \
  --conf "spark.executor.extraJavaOptions=-Dsun.security.krb5.debug=true" \
  /opt/cloudera/parcels/CDH/lib/spark/lib/spark-examples.jar 10

Redirect Spark Logs to stdout

Java logs Kerberos debug information to the JVM’s stdout, which is available as the stdout of the YARN container for Spark components that execute on the cluster. The default Spark log4j configuration sends Spark logging output to stderr. The JVM does not give you flexibility for where it logs Kerberos output; it always goes to stdout.

Consequently, it is often helpful to interleave the Kerberos logs with Spark logs so you have more context about how the two relate in time. Kerberos log messages have no timestamps, but Spark log messages do. To do this, redirect Spark logs to stdout using a Spark gateway Safety Valve:

  1. Add Spark gateway log4j configuration to redirect Spark logs to stdout.

    In Cloudera Manager, navigate to Spark > Configuration > Scope: Gateway > Category: Advanced > “Gateway Logging Advanced Configuration Snippet (Safety Valve)” and add:

    log4j.appender.console.target=System.out

  2. Save changes and deploy client configuration for Spark.

    In Cloudera Manager, navigate to Spark > Actions > Deploy Client Configuration.

Note

Client-mode Spark applications will display the driver’s Kerberos debug information on the console.

References