https://lukehebert.com/category/bigdata/Recent content in Bigdata on Luke HebertHugoen-usMon, 14 Apr 2025 00:00:00 +0000https://lukehebert.com/post/graph-database-white-papers/Mon, 14 Apr 2025 00:00:00 +0000https://lukehebert.com/post/graph-database-white-papers/An overview of graph databases, their structure, and advantages over relational models. Covers how nodes, relationships, and the Cypher query language work together to model complex data.https://lukehebert.com/post/galera-cluster-performance/Mon, 24 Jan 2022 00:00:00 +0000https://lukehebert.com/post/galera-cluster-performance/Tuning Galera’s gcache size and page size can reduce disk I/O during replication and keep write-set lookups in memory. A quick configuration change that pays off on busy clusters.https://lukehebert.com/post/mysql-fragmentation-levels/Mon, 24 Jan 2022 00:00:00 +0000https://lukehebert.com/post/mysql-fragmentation-levels/Learn how to measure InnoDB table fragmentation using information_schema queries and interpret the resulting ratio to decide when a table needs optimization.https://lukehebert.com/post/user-performance-data-on-mysql/Mon, 24 Jan 2022 00:00:00 +0000https://lukehebert.com/post/user-performance-data-on-mysql/A quick bash one-liner that pulls per-user CPU time, connection counts, and active session details from MySQL’s INFORMATION_SCHEMA and processlist tables.https://lukehebert.com/post/kafka-dump/Wed, 25 Aug 2021 00:00:00 +0000https://lukehebert.com/post/kafka-dump/A practical walkthrough for dumping Kafka topic data, including a generic consumer approach and a full Atlas-specific workflow covering JAAS configuration, Kerberos authentication, and log extraction.https://lukehebert.com/post/dump-kafka-atlas-topic-to-review-messages/Fri, 20 Aug 2021 00:00:00 +0000https://lukehebert.com/post/dump-kafka-atlas-topic-to-review-messages/A quick walkthrough for dumping the ATLAS_HOOK Kafka topic to inspect Atlas hook messages locally, including Kerberos authentication setup and consumer lag checks.https://lukehebert.com/post/hive-bundle-logs/Thu, 19 Aug 2021 00:00:00 +0000https://lukehebert.com/post/hive-bundle-logs/Hive bundle logs in a cloud data warehouse are stored under a predictable S3 path. Knowing where to look saves time when troubleshooting query performance or diagnosing failed jobs.https://lukehebert.com/post/delete-existing-users-from-ranger/Tue, 06 Jul 2021 00:00:00 +0000https://lukehebert.com/post/delete-existing-users-from-ranger/A quick walkthrough of using the Ranger REST API to bulk-delete existing users, including how to capture an auth cookie and loop through a user list with curl.https://lukehebert.com/post/debug-kerberos-gen-credentials-in-cm/Wed, 10 Feb 2021 00:00:00 +0000https://lukehebert.com/post/debug-kerberos-gen-credentials-in-cm/Learn how to enable Kerberos trace logging in Cloudera Manager’s credential generation scripts to diagnose authentication failures quickly.https://lukehebert.com/post/hdfs-named-node-memory-sizing-when-using-hdfs-encryption/Thu, 11 Jan 2018 00:00:00 +0000https://lukehebert.com/post/hdfs-named-node-memory-sizing-when-using-hdfs-encryption/Enabling HDFS encryption at rest adds per-file metadata overhead to the NameNode heap. This guide breaks down the memory impact and gives a practical rule of thumb for right-sizing your NameNode heap.https://lukehebert.com/post/hdfs-proxy-users/Wed, 06 Sep 2017 00:00:00 +0000https://lukehebert.com/post/hdfs-proxy-users/HDFS proxy users let you configure impersonation so specific users can submit jobs or access HDFS on behalf of others. This guide covers the core-site.xml properties, host and group restrictions, and Kerberos considerations for secure clusters.https://lukehebert.com/post/aes-ni-encryption-in-flight-for-hdfs/Tue, 27 Jun 2017 00:00:00 +0000https://lukehebert.com/post/aes-ni-encryption-in-flight-for-hdfs/Configure AES-NI accelerated encryption in transit for HDFS by setting the cipher suite and key length in hdfs-site.xml on both the NameNode/DataNode and client sides.https://lukehebert.com/post/are-unix-users-required-for-hadoop/Tue, 27 Jun 2017 00:00:00 +0000https://lukehebert.com/post/are-unix-users-required-for-hadoop/Hadoop requires OS-level user accounts on every cluster node regardless of security configuration. This post explains why, covering both secure and simple mode, and addresses whether the requirement can be removed.https://lukehebert.com/post/cm-license-api-update/Tue, 27 Jun 2017 00:00:00 +0000https://lukehebert.com/post/cm-license-api-update/A quick reference for updating a Cloudera Manager license through the REST API using a single curl command, including the required headers and authentication.https://lukehebert.com/post/cm-timezone-does-not-match/Tue, 27 Jun 2017 00:00:00 +0000https://lukehebert.com/post/cm-timezone-does-not-match/Cloudera Manager raises a timezone mismatch alert when the OS clock files disagree with the timezone CM expects. This post walks through the three key files to check and how to align them.https://lukehebert.com/post/delete-hue-users/Tue, 27 Jun 2017 00:00:00 +0000https://lukehebert.com/post/delete-hue-users/Hue does not provide a built-in UI option to remove user accounts. This workaround uses the Hue Django shell to delete one or more users directly from the database, with a clear caveat that the process is unsupported.https://lukehebert.com/post/determine-is-a-datanode-is-dead-formula/Tue, 27 Jun 2017 00:00:00 +0000https://lukehebert.com/post/determine-is-a-datanode-is-dead-formula/A quick reference for calculating the exact timeout window before Hadoop marks a Datanode as dead, based on the heartbeat and recheck-interval settings.https://lukehebert.com/post/jdk-break-downs-for-web-ui-and-spengo-u92_u91_u40_u45/Tue, 27 Jun 2017 00:00:00 +0000https://lukehebert.com/post/jdk-break-downs-for-web-ui-and-spengo-u92_u91_u40_u45/Specific JDK 1.8.0 builds (u40, u45, u91, u92) are known to break SPNEGO authentication and Hadoop web interfaces. This post documents the affected versions, the symptoms, and the recommended downgrade path.https://lukehebert.com/post/jvm-ssl-debug/Tue, 27 Jun 2017 00:00:00 +0000https://lukehebert.com/post/jvm-ssl-debug/Enable verbose SSL/TLS handshake logging in a JVM running Hadoop or Cloudera services to diagnose certificate and protocol issues.https://lukehebert.com/post/change-nn-trash-check-point-interval-can-correct-tgt-on-nn-non-ha-with-kms/Tue, 27 Jun 2017 00:00:00 +0000https://lukehebert.com/post/change-nn-trash-check-point-interval-can-correct-tgt-on-nn-non-ha-with-kms/Adjusting the NameNode trash checkpoint interval can resolve TGT renewal issues on non-HA NameNodes using KMS. This change is safe and does not affect trash retention.https://lukehebert.com/post/namenode-loglevel/Tue, 27 Jun 2017 00:00:00 +0000https://lukehebert.com/post/namenode-loglevel/How to set the HDFS Namenode StateChange log level to DEBUG via the web UI or the command line, and when that helps with troubleshooting.