AES-NI Encryption in Flight for HDFS

Luke Hebert

June 27, 2017 - 2 minutes read - 393 words

If your HDFS cluster moves sensitive data across a shared network, you probably want that traffic encrypted in transit. The good news is that HDFS supports AES-NI hardware-accelerated encryption, which means you get strong protection without a noticeable performance hit on modern CPUs.

What AES-NI Encryption in Flight Does

When data travels between HDFS clients and DataNodes (or between DataNodes during replication), it can be encrypted using the cipher suite you specify. AES-NI is a set of CPU instructions introduced by Intel and AMD that offload AES operations to dedicated hardware, so the encryption overhead is minimal compared to a pure software implementation.

Configuration

You need to set two properties in hdfs-site.xml on both the server side (NameNode and DataNodes) and the client side (any application or tool that reads/writes HDFS data).

Server Side (hdfs-site.xml)

<property>
  <name>dfs.encrypt.data.transfer.cipher.suites</name>
  <value>AES/CTR/NoPadding</value>
</property>
<property>
  <name>dfs.encrypt.data.transfer.cipher.key.bitlength</name>
  <value>256</value>
</property>

Client Side (hdfs-site.xml)

<property>
  <name>dfs.encrypt.data.transfer.cipher.suites</name>
  <value>AES/CTR/NoPadding</value>
</property>
<property>
  <name>dfs.encrypt.data.transfer.cipher.key.bitlength</name>
  <value>256</value>
</property>

Property Breakdown

Property	Purpose
`dfs.encrypt.data.transfer.cipher.suites`	Defines the cipher mode used for data transfer. `AES/CTR/NoPadding` uses AES in Counter mode, which is fast and does not require block padding.
`dfs.encrypt.data.transfer.cipher.key.bitlength`	Sets the AES key size in bits. A value of `256` gives you AES-256, the strongest variant.

Things to Keep in Mind

Important

Both sides of the connection must agree on the same cipher suite and key length. If the client and server are configured differently, the data transfer will fail with a cipher mismatch error.

Hardware support: AES-NI is available on virtually all x86-64 CPUs manufactured after 2008. You can verify support by checking for the aes flag in /proc/cpuinfo.
Performance: Because the work is handled in hardware, the throughput penalty is typically under 5% compared to unencrypted transfers, even at 256-bit key lengths.
Scope: This setting encrypts data in flight (between nodes). It does not replace at-rest encryption, which is controlled by the HDFS encryption zones feature.

Verifying the Configuration

After restarting the affected HDFS services, you can confirm the cipher is active by checking the DataNode logs for lines referencing the data transfer cipher, or by running a simple hdfs dfs -ls / from a client and watching for successful encrypted sessions.

That is all you need to get AES-NI accelerated encryption running on your HDFS cluster. Two properties, two sides of the wire, and your data is protected in transit with minimal performance cost.