Debug Kerberos Gen Credentials in CM

Kerberos credential generation in Cloudera Manager can fail silently, leaving you with a service that simply will not authenticate. The fastest way to see what is actually happening under the hood is to enable the KRB5_TRACE environment variable in the relevant gen_credentials script.

What the gen_credentials scripts do

Cloudera Manager uses a set of shell scripts to generate and refresh Kerberos credentials for managed services. Depending on your identity provider, one of three scripts handles the work:

• /opt/cloudera/cm/bin/gen_credentials.sh - generic Kerberos (KDC-based)
• /opt/cloudera/cm/bin/gen_credentials_ipa.sh - FreeIPA
• /opt/cloudera/cm/bin/gen_credentials_ad.sh - Active Directory

These scripts are invoked by the Cloudera Manager agent when a service needs a new keytab or credential cache. If the underlying kinit or kadmin call fails, the error is often swallowed and you only see a generic “credential generation failed” message in the CM UI.

Enabling the trace

Open the appropriate script for your environment (the paths above apply to Cloudera Manager 6.x and later) and add the following line near the top, before any kinit or kadmin calls:

export KRB5_TRACE=/tmp/generate_output.log

This tells the MIT Kerberos libraries to write a detailed, step-by-step trace of every network exchange and protocol message to the log file.

Reading the output

After triggering a credential refresh (for example, by restarting the affected service in Cloudera Manager), inspect the log:

cat /tmp/generate_output.log

Look for lines that indicate which stage failed:

• AS-REQ / TGS-REQ - the initial authentication or ticket-granting exchange. Failures here usually point to a wrong principal name, clock skew, or a missing keytab entry.
• KRB5 error codes - codes like KRB5_KT_NOTFOUND or KRB5_PRINCIPAL_NOT_FOUND map directly to common misconfigurations.
• DNS / SRV lookups - if the trace shows a failed DNS query, verify that your KDC or AD domain controller is resolvable from the CM agent host.

Common pitfalls

• Clock skew - Kerberos is strict about time. If the CM agent host is more than 5 minutes off from the KDC, authentication will fail. Check with date and ntpstat.
• Wrong principal - The principal in the script must match exactly what is registered in the KDC or AD. A single character difference (case, realm suffix) is enough to break the exchange.
• Keytab permissions - Ensure the cloudera-scm user (or the user running the agent) can read the keytab file referenced by the script.

Cleaning up

Once you have the trace and have resolved the issue, remove the export KRB5_TRACE line from the script and delete the log file:

rm -f /tmp/generate_output.log

Restart the affected service in Cloudera Manager to confirm the credential generates cleanly without the trace overhead.

• Security
• Bigdata

• Kerberos
• Active-Directory
• Sso