Compare KVNO in DB and in Keytab (These Should Match)

Luke Hebert

- 2 minutes read - 260 words

When a Kerberized service suddenly stops authenticating, one of the first things to verify is that the key version number (KVNO) in the KDC database matches the KVNO baked into the local keytab file. If they disagree, the service will reject tickets even though the principal name looks correct. The two commands below give you a fast side-by-side comparison.

Step 1: Check the KVNO in the KDC Database

First, obtain a ticket. If a host keytab is available on the machine, use it; otherwise fall back to a working credential such as an admin user.

kinit -k

Then query the KDC for the current KVNO of the principal:

kvno hdfs/server-1.vpc.example.com@AD.SEC.EXAMPLE.COM

Expected output:

hdfs/server-1.vpc.example.com@AD.SEC.EXAMPLE.COM: kvno = 1

Step 2: Check the KVNO in the Local Keytab

Now inspect the keytab file that the service actually uses. In a Cloudera-managed deployment the path is typically under the agent process directory:

kvno -k /var/run/cloudera-scm-agent/process/1320-hdfs-NAMENODE/hdfs.keytab hdfs/server-1.vpc.example.com@AD.SEC.EXAMPLE.COM

Expected output:

hdfs/server-1.vpc.example.com@AD.SEC.EXAMPLE.COM: kvno = 1, keytab entry valid

What to Do If They Do Not Match

Important

A KVNO mismatch means the keytab is stale. Regenerate the keytab from the KDC and redeploy it to the service before restarting the process.

Common reasons for a mismatch include:

  • A password or key rotation was performed in Active Directory but the keytab was not re-exported.
  • The keytab was copied from another host and never refreshed.
  • A service principal was deleted and recreated, bumping the KVNO.

After regenerating the keytab, re-run both commands above to confirm the numbers line up, then restart the affected service.