Are Unix Users Required for Hadoop

Hadoop requires OS-level user accounts on every node in the cluster, and this holds true whether you are running in secure mode with Kerberos or in simple mode. Understanding why this requirement exists helps you plan your user provisioning strategy before you hit runtime failures.

Do I Need OS Accounts in a Secure Cluster?

Short answer: yes. Even with Sentry enabled, task execution still happens as the end user when Kerberos is active. Hive and Impala may use system users for certain operations, but Spark, Pig, MapReduce, YARN, and similar engines run tasks as the regular submitting user. That means those users must exist on every node in the cluster, whether through OS-to-LDAP integration or another provisioning method.

What About Non-Secure Clusters?

The Hadoop Operations Guide lays out the distinction clearly:

Hadoop operates in either the default, so-called simple mode or secure mode, which provides strong authentication support via Kerberos. For many, the simple security mode is sufficient and offers reasonable protection from mistakes in a trusted environment. As its name implies, it is simple to configure and manage, relying primarily on the host for authentication. If, however, you are running Hadoop in an untrusted, multitenant environment or where accidental data exposure would be catastrophic, secure mode is the appropriate option. In secure mode, Hadoop uses the well-known Kerberos protocol to authenticate users and daemons themselves during all operations. Additionally, MapReduce tasks in secure mode are executed as the same OS user as the job was submitted, whereas in simple mode, they are executed as the user running the tasktracker.

The Standing Answer

The consistent guidance to date is that OS accounts are required for job runtime isolation, regardless of the security configuration, whether impersonation is disabled, or whether Sentry is in use. At the OS layer, users representing the job submitter must exist. In practice, security gets enabled at some point, and if the OS accounts are not already in place, jobs will fail.

Can This Requirement Be Removed?

For sites that do not want to provision end-user accounts onto cluster nodes, there is no supported solution that bypasses this requirement. As of the time of writing, no roadmap item addresses removing the dependency on OS-level users. From prior engineering discussions, the consensus is that this requirement is inherent to the platform and cannot be removed.

• Bigdata

• Kerberos
• SysAdmin