"Not That Key!" No, Your Other Key...

Luke Hebert

August 21, 2017 - 4 minutes read - 674 words

Kerberos error 44 (“Specified version of key is not available”) is a common authentication failure that occurs when a keytab file is out of sync with the KDC. The fix is straightforward once you confirm the mismatch, but it can be tricky to track down in environments with multiple domain controllers or load-balanced KDCs.

What Causes Error 44

Java (and the underlying Kerberos library) checks whether the version number (KVNO) in the keytab file matches the KVNO stored in the Kerberos database on the KDC server. If the two numbers differ, authentication fails with error 44.

Common Triggers for KVNO Drift

The KVNO increments every time the key material for a principal changes. Common events that cause this include:

A password or key was rotated on the service account
An administrator ran kadmin commands such as change_password or randkey
The account was recreated, for example after a domain migration or service account reset
A management agent (Cloudera, Ambari, or similar) re-provisioned the credential

If you recently performed any of these actions and authentication started failing, a KVNO mismatch is the most likely cause.

Diagnosing the Mismatch

Compare the KVNO in the KDC database against the KVNO in the keytab file. These two values must match.

Check the KVNO in the KDC Database

kvno hdfs/server-1.vpc.example.com@AD.SEC.EXAMPLE.COM

Expected output:

hdfs/server-1.vpc.example.com@AD.SEC.EXAMPLE.COM: kvno = 1

Check the KVNO in the Keytab File

kvno -k /var/run/cloudera-scm-agent/process/1320-hdfs-NAMENODE/hdfs.keytab hdfs/server-1.vpc.example.com@AD.SEC.EXAMPLE.COM

Expected output:

hdfs/server-1.vpc.example.com@AD.SEC.EXAMPLE.COM: kvno = 1, keytab entry valid

Inspect Keytab Entries with klist

The klist -kt command is widely available and shows the full entry, including encryption types and flags:

klist -kt /var/run/cloudera-scm-agent/process/1320-hdfs-NAMENODE/hdfs.keytab

Cross-Check with kadmin

When you have KDC admin access, getprinc shows the KVNO along with the key types, which helps confirm you are comparing the right principal and that the encryption types in the keytab are still valid:

kadmin -p admin/admin@AD.SEC.EXAMPLE.COM -q \
  "getprinc hdfs/server-1.vpc.example.com"

Important

If the KVNO values do not match, your keytab is out of date. You will need to regenerate it so that its KVNO matches what the KDC returns.

Regenerating the Keytab

Before regenerating, back up the existing keytab so you can roll back if something goes wrong:

cp /var/run/cloudera-scm-agent/process/1320-hdfs-NAMENODE/hdfs.keytab \
   /var/run/cloudera-scm-agent/process/1320-hdfs-NAMENODE/hdfs.keytab.bak.$(date +%Y%m%d)

MIT Kerberos

kadmin -p admin/admin@AD.SEC.EXAMPLE.COM -q \
  "ktadd -k /var/run/cloudera-scm-agent/process/1320-hdfs-NAMENODE/hdfs.keytab -k 0 hdfs/server-1.vpc.example.com"

The -k 0 flag ensures the new keytab contains only the latest key version.

Active Directory (ktpass)

ktpass -princ hdfs/server-1.vpc.example.com@AD.SEC.EXAMPLE.COM \
  -pass <password> -mapuser DOMAIN\hdfs \
  -out C:\hdfs.keytab -crypto AES256 -ptype KRB5_NT_PRINCIPAL

Note

After regenerating, verify the file permissions. A keytab that is world-readable will be rejected by some Java Kerberos implementations, and a keytab with the wrong owner can cause the agent to read a stale copy.

ls -l /var/run/cloudera-scm-agent/process/1320-hdfs-NAMENODE/hdfs.keytab
# Should be owned by the service user, mode 0400 or 0600

Verifying the Fix

Confirm the new keytab works before restarting your service:

kinit -kt /var/run/cloudera-scm-agent/process/1320-hdfs-NAMENODE/hdfs.keytab \
  hdfs/server-1.vpc.example.com@AD.SEC.EXAMPLE.COM
klist

A successful kinit with no error output means the KVNO now matches and the keytab is valid.

Load-Balanced or Replicated KDCs

If your KDCs are load-balanced and replicate to each other, independently check each KDC. Ensure that every KDC holds the same KVNO for the credentials you are testing.

As a rule of thumb, configure a specific KDC override rather than relying solely on the load balancer for credential operations. This principle applies to both Active Directory and MIT KDC deployments.

Warning

If the KVNO or any part of the credential information is not fully replicated or is out of sync across your KDCs, you will encounter random, intermittent authentication failures.

If the KVNO values match but authentication still fails, check for these common look-alikes:

Error Code	Meaning
44	KVNO mismatch (the issue covered here)
47	Clock skew exceeds 5 minutes
6	KDC cannot find the requested principal
23	Pre-authentication failed (wrong password)

Also verify that /etc/krb5.conf points to the correct KDC and realm. A stale kdc = or default_realm = entry can route the request to a different, older KDC.