Determine If a Datanode Is Dead (Formula)

When a Datanode stops responding, the NameNode does not immediately declare it dead. Instead, it waits through a calculated timeout window before removing the node from the cluster. Knowing that window helps you distinguish a transient network blip from a genuinely failed node, and it tells you how long to wait before escalating.

The Formula

The dead-Datanode threshold is calculated as:

2 * dfs.namenode.heartbeat.recheck-interval + 10 * 1000 * dfs.heartbeat.interval

Both values are expressed in milliseconds.

Default Values

Plugging in the defaults:

2 * 300000 + 10 * 1000 * 3
= 600000 + 30000
= 630000 ms
= 630 seconds (10.5 minutes)

So with stock settings, a Datanode is considered dead after roughly 10.5 minutes of missed heartbeats.

How the Mechanism Works

1. Each Datanode sends a heartbeat to the NameNode every dfs.heartbeat.interval milliseconds.
2. The NameNode tracks the last heartbeat it received from every Datanode.
3. Every dfs.namenode.heartbeat.recheck-interval milliseconds, the NameNode runs a recheck pass. During that pass it compares the current time against each Datanode’s last heartbeat.
4. If the elapsed time exceeds the formula above, the NameNode marks the Datanode as dead and begins re-replicating its blocks to other live nodes.

Practical Notes

• If you have tuned dfs.heartbeat.interval or dfs.namenode.heartbeat.recheck-interval in hdfs-site.xml, recalculate the threshold with your actual values. A shorter heartbeat interval reduces detection time but increases NameNode load.
• A Datanode that is slow to respond (GC pauses, disk I/O saturation) may miss a few heartbeats without being truly dead. The 10x multiplier on the heartbeat interval in the formula provides that grace period.
• When troubleshooting, check the NameNode logs for DatanodeDeathEvent entries and the Datanode’s own logs for heartbeat send errors. Network latency or firewall rules between the Datanode and NameNode are common culprits.

• Bigdata

• SysAdmin
• Troubleshooting