https://lukehebert.com/tag/troubleshooting/Recent content in Troubleshooting on Luke HebertHugoen-usSat, 23 Aug 2025 00:00:00 +0000https://lukehebert.com/post/enable-kubelet-serving-certificates-in-kubernetes-if-you-did-not-do-so-during-cluster-deployment-metric-server/Sat, 23 Aug 2025 00:00:00 +0000https://lukehebert.com/post/enable-kubelet-serving-certificates-in-kubernetes-if-you-did-not-do-so-during-cluster-deployment-metric-server/Enables kubelet TLS bootstrapping so the controller-manager issues proper serving certificates, resolving the x509 certificate validation error in metrics-server. Covers deployment edits, controller-manager configuration, and automated certificate renewal.https://lukehebert.com/post/basic-iptables-insert/Thu, 21 Aug 2025 00:00:00 +0000https://lukehebert.com/post/basic-iptables-insert/Learn how to insert iptables rules at a specific position using the -I flag, with syntax breakdown and a practical SSH example.https://lukehebert.com/post/firewalld-cheatsheet/Thu, 21 Aug 2025 00:00:00 +0000https://lukehebert.com/post/firewalld-cheatsheet/A practical cheat sheet for managing firewalld on RHEL, Fedora, and CentOS systems. Covers zones, services, ports, rich rules, NAT, and direct iptables-style rules.https://lukehebert.com/post/ip-link-show-type/Tue, 19 Aug 2025 00:00:00 +0000https://lukehebert.com/post/ip-link-show-type/A guide to the ip link show type and ip address show type commands in Linux for filtering network interfaces by type. Covers physical, virtual, and tunneling interface types with practical examples.https://lukehebert.com/post/understanding-how-pods-talk-in-kubernetes-networks-dns/Tue, 19 Aug 2025 00:00:00 +0000https://lukehebert.com/post/understanding-how-pods-talk-in-kubernetes-networks-dns/A deep dive into how Kubernetes DNS resolves service names to IP addresses, covering A records, SRV records, search domains, and the full lookup path from pod to CoreDNS. Understanding this lifecycle is essential for debugging connectivity issues and designing resilient microservice architectures.https://lukehebert.com/post/finding-which-java-threads-are-consuming-your-cpu/Fri, 15 Aug 2025 00:00:00 +0000https://lukehebert.com/post/finding-which-java-threads-are-consuming-your-cpu/When a Java process is eating your CPU, you need to know exactly which thread is responsible. This guide walks through JVisualVM, jstack, top, and jvmtop to pinpoint the culprit.https://lukehebert.com/post/how-to-inspect-a-thread-dump-like-a-pro/Fri, 15 Aug 2025 00:00:00 +0000https://lukehebert.com/post/how-to-inspect-a-thread-dump-like-a-pro/Learn how to capture, read, and interpret Java thread dumps using JDK tools like jstack and jcmd. Covers thread states, CPU analysis, deadlock detection, and practical tools for spotting performance issues.https://lukehebert.com/post/not-enough-storage-available-while-using-jstack-win/Thu, 14 Aug 2025 00:00:00 +0000https://lukehebert.com/post/not-enough-storage-available-while-using-jstack-win/A quick fix for the ’not enough storage available’ error when running jstack on Windows, caused by the JVM running under the Local System Account. Two workarounds using Psexec or JMX remote monitoring.https://lukehebert.com/post/using-jcmd-jstack-to-get-thread-information/Tue, 12 Aug 2025 00:00:00 +0000https://lukehebert.com/post/using-jcmd-jstack-to-get-thread-information/A practical guide to using the jcmd and jstack utilities for collecting thread dumps, heap data, and CPU information from a running JVM process. Includes ready-to-use scripts and one-liners for troubleshooting slow or hung Java applications.https://lukehebert.com/post/kubernetes-resource-management/Mon, 11 Aug 2025 00:00:00 +0000https://lukehebert.com/post/kubernetes-resource-management/A practical guide to configuring CPU, memory, and storage requests and limits in Kubernetes, covering container-level, Pod-level, and namespace-level resource controls. Includes troubleshooting steps for common scheduling and OOM issues.https://lukehebert.com/post/pgbackrest-and-helpful-commands/Wed, 02 Jul 2025 00:00:00 +0000https://lukehebert.com/post/pgbackrest-and-helpful-commands/A practical reference for pgBackRest backup, restore, and cleanup commands for PostgreSQL. Covers manual cleanup, incremental backups, retention policies, and stanza management.https://lukehebert.com/post/understanding-machine-cpu-usage/Wed, 18 Jun 2025 00:00:00 +0000https://lukehebert.com/post/understanding-machine-cpu-usage/A practical walkthrough of reading node_cpu_seconds_total from the Prometheus Node exporter, calculating per-second rates, and deriving overall CPU utilization percentages across multiple machines.https://lukehebert.com/post/an-introductory-how-to-of-docker-exec/Thu, 29 May 2025 00:00:00 +0000https://lukehebert.com/post/an-introductory-how-to-of-docker-exec/A practical guide to the docker exec command, covering background execution, environment variables, interactive sessions, and working directory control. Includes real-world examples to help you get the most out of running commands inside containers.https://lukehebert.com/post/java-tls-client-debugging/Mon, 19 May 2025 00:00:00 +0000https://lukehebert.com/post/java-tls-client-debugging/A guide to inspecting SSL/TLS handshakes in Java applications when built-in debugging is unavailable. Includes a lightweight TLS client tool for testing certificate and handshake issues.https://lukehebert.com/post/udev-diagnostic-test/Thu, 27 Feb 2025 00:00:00 +0000https://lukehebert.com/post/udev-diagnostic-test/A walkthrough of using udevadm to diagnose and inspect udev rule application on a /dev/null device node. Covers the test command syntax and interprets the output for troubleshooting permission issues.https://lukehebert.com/post/monitoring-and-tuning-the-linux-networking-stack-receiving-data/Fri, 11 Oct 2024 00:00:00 +0000https://lukehebert.com/post/monitoring-and-tuning-the-linux-networking-stack-receiving-data/A deep dive into how the Linux kernel receives network packets, covering the full path from NIC hardware through NAPI, softirqs, GRO, RPS, and protocol layers. Includes practical monitoring commands and tuning knobs for each stage.https://lukehebert.com/post/linux-network-ring-buffers-trying-to-cover-what-i-dont-know-about/Fri, 11 Oct 2024 00:00:00 +0000https://lukehebert.com/post/linux-network-ring-buffers-trying-to-cover-what-i-dont-know-about/A practical walkthrough of how Linux allocates and manages RX/TX ring buffers for network interfaces, covering DMA, softirq processing, and tuning knobs like netdev_budget.https://lukehebert.com/post/myths-about-dev-urandom/Fri, 04 Oct 2024 00:00:00 +0000https://lukehebert.com/post/myths-about-dev-urandom/A practical breakdown of the most common misconceptions about /dev/urandom and /dev/random on Linux, explaining why /dev/urandom is the correct choice for cryptographic randomness and why blocking on low entropy is a usability problem, not a security one.https://lukehebert.com/post/wireshark-filter-for-ssl-traffic/Fri, 30 Aug 2024 00:00:00 +0000https://lukehebert.com/post/wireshark-filter-for-ssl-traffic/A collection of useful Wireshark display filters for analyzing SSL/TLS handshake traffic, including a reference table of handshake message types and notes on cipher suite considerations.https://lukehebert.com/post/docker-kubernetes-network-diagnostic-tools/Mon, 26 Aug 2024 00:00:00 +0000https://lukehebert.com/post/docker-kubernetes-network-diagnostic-tools/A practical guide to using the netshoot container for Docker and Kubernetes network troubleshooting. Covers network namespaces, tcpdump examples, and real-world diagnostic workflows.https://lukehebert.com/post/how-to-clean-up-pg-wal-if-the-disk-is-full/Mon, 04 Dec 2023 00:00:00 +0000https://lukehebert.com/post/how-to-clean-up-pg-wal-if-the-disk-is-full/A practical guide to resolving PostgreSQL pg_wal disk-full issues, covering both reactive recovery steps and proactive measures to prevent recurrence. Learn how to free space, move the WAL directory, and configure archiving correctly.https://lukehebert.com/post/analyzing-large-java-heap-dumps-using-memory-analyzer-cli-in-linux/Fri, 17 Nov 2023 00:00:00 +0000https://lukehebert.com/post/analyzing-large-java-heap-dumps-using-memory-analyzer-cli-in-linux/When MAT UI fails to handle very large heap dumps, the Memory Analyzer CLI provides a reliable alternative on Linux. This post walks through the setup and commands needed to parse multi-gigabyte dumps and generate shareable HTML reports.https://lukehebert.com/post/working-with-multiple-content-security-policy-headers/Thu, 28 Sep 2023 00:00:00 +0000https://lukehebert.com/post/working-with-multiple-content-security-policy-headers/Multiple Content-Security-Policy headers do not combine; the most restrictive one always wins. Learn how to correctly configure CSP in nginx by consolidating directives into a single header.https://lukehebert.com/post/the-beer-drinkers-guide-to-saml-duo-security/Mon, 11 Sep 2023 00:00:00 +0000https://lukehebert.com/post/the-beer-drinkers-guide-to-saml-duo-security/A plain-English walkthrough of how SAML authentication works, using the analogy of buying a beer at a concert. Covers IdP and SP configuration, IdP-initiated vs SP-initiated flows, and practical troubleshooting steps.https://lukehebert.com/post/how-to-start-a-sysv-init-script-after-network-starts-in-red-hat-enterprise-linux-7/Tue, 27 Jun 2023 00:00:00 +0000https://lukehebert.com/post/how-to-start-a-sysv-init-script-after-network-starts-in-red-hat-enterprise-linux-7/Configure a legacy SysV init script to wait for network availability in RHEL 7 by adding the Required-Start LSB directive. This guide shows how systemd honors those headers so third-party initscripts start in the correct order.https://lukehebert.com/post/override-systemd-unit-files/Tue, 27 Jun 2023 00:00:00 +0000https://lukehebert.com/post/override-systemd-unit-files/Learn how to safely override systemd unit and timer files using systemctl edit, avoiding the pitfalls of editing package-managed files directly. Covers creating drop-in overrides, full replacements, and reverting changes.https://lukehebert.com/post/how-to-change-docker-root-data-directory/Fri, 02 Jun 2023 00:00:00 +0000https://lukehebert.com/post/how-to-change-docker-root-data-directory/A step-by-step guide to relocating the Docker data directory to a new path. Covers daemon configuration, data migration, and cleanup for both older and newer Docker versions.https://lukehebert.com/post/docker-determining-container-responsible-for-largest-overlay-directories/Thu, 18 May 2023 00:00:00 +0000https://lukehebert.com/post/docker-determining-container-responsible-for-largest-overlay-directories/A practical guide to identifying which Docker containers occupy the largest overlay2 directories. Uses docker inspect, jq, and xargs to map cryptic layer hashes back to human-readable container names.https://lukehebert.com/post/a-walkthrough-of-a-tls-1.3-handshake/Wed, 10 May 2023 00:00:00 +0000https://lukehebert.com/post/a-walkthrough-of-a-tls-1.3-handshake/A byte-by-byte walkthrough of a TLS 1.3 handshake using OpenSSL’s s_client, comparing each message to its TLS 1.2 counterpart. Highlights how encryption, key exchange, and extensions differ in the latest revision.https://lukehebert.com/post/how-to-setup-and-secure-unifi-vlan/Fri, 05 May 2023 00:00:00 +0000https://lukehebert.com/post/how-to-setup-and-secure-unifi-vlan/A step-by-step guide to creating and securing VLANs in the UniFi Network Console, including firewall rules to block inter-VLAN traffic and restrict console access.https://lukehebert.com/post/intro-to-networking-introduction-to-virtual-lans-vlans-and-tagging/Fri, 05 May 2023 00:00:00 +0000https://lukehebert.com/post/intro-to-networking-introduction-to-virtual-lans-vlans-and-tagging/A beginner-friendly overview of Virtual LANs, covering their benefits, typical usage scenarios, and the fundamentals of 802.1Q VLAN tagging.https://lukehebert.com/post/unifi-network-creating-virtual-networks-vlans/Fri, 05 May 2023 00:00:00 +0000https://lukehebert.com/post/unifi-network-creating-virtual-networks-vlans/Learn how to create and configure virtual networks (VLANs) in UniFi Network to segment devices and improve network security. This guide covers wireless and wired client binding, trunk ports, and third-party gateway integration.https://lukehebert.com/post/dns/Tue, 02 May 2023 00:00:00 +0000https://lukehebert.com/post/dns/A practical guide to understanding DNS resolution tiers and troubleshooting name resolution issues in Linux environments. Covers /etc/nsswitch.conf, /etc/resolv.conf, and local caching services.https://lukehebert.com/post/optimize-postgresql-server-performance-through-configuration/Thu, 27 Apr 2023 00:00:00 +0000https://lukehebert.com/post/optimize-postgresql-server-performance-through-configuration/This guide walks through four key PostgreSQL configuration settings that significantly impact server performance. Learn how to tune shared_buffers, wal_buffers, effective_cache_size, and work_mem for your specific workload.https://lukehebert.com/post/using-sqlline-to-test-jdbc-connectors/Thu, 15 Dec 2022 00:00:00 +0000https://lukehebert.com/post/using-sqlline-to-test-jdbc-connectors/A practical guide to using SQLline to replicate SQL client behavior when testing JDBC connectors. Covers downloading the tool, setting up the classpath, and connecting to an Azure MySQL instance.https://lukehebert.com/post/id4262-the-saml-nameidentifier-user-is-of-format-urnoasisname/Tue, 29 Mar 2022 00:00:00 +0000https://lukehebert.com/post/id4262-the-saml-nameidentifier-user-is-of-format-urnoasisname/The ID4262 error occurs when a SAML NameIdentifier uses the entity format but contains an invalid URI. Two resolution steps fix the issue within your identity provider configuration.https://lukehebert.com/post/kerberos-message-stream-modified-41/Fri, 18 Feb 2022 00:00:00 +0000https://lukehebert.com/post/kerberos-message-stream-modified-41/JDK 1.8.0u242 and later introduced Kerberos referral handling per RFC 6806, which can break Active Directory name canonicalization and produce a KRB_AP_ERR_MODIFIED error. This post walks through the two configuration fixes that resolve the issue.https://lukehebert.com/post/a-parctical-guide-to-trace-route-tracecrt/Mon, 24 Jan 2022 00:00:00 +0000https://lukehebert.com/post/a-parctical-guide-to-trace-route-tracecrt/Traceroute is one of the most misunderstood tools in a network engineer’s toolkit. This guide walks through how traceroute actually works, how to read the output correctly, and how to avoid the common pitfalls that lead to wrong conclusions.https://lukehebert.com/post/awk-scan-for-wp-login/Mon, 24 Jan 2022 00:00:00 +0000https://lukehebert.com/post/awk-scan-for-wp-login/A quick awk pipeline to pull wp-login.php access attempts from web server logs and flag IPs that exceed a threshold. Useful for spotting brute-force or scanning activity against WordPress sites.https://lukehebert.com/post/check-for-ip-block/Mon, 24 Jan 2022 00:00:00 +0000https://lukehebert.com/post/check-for-ip-block/A quick way to verify whether a specific IP address is blocked by iptables rules on a Linux host. Covers the exact command and how to read the output.https://lukehebert.com/post/check-web-crawler-impact/Mon, 24 Jan 2022 00:00:00 +0000https://lukehebert.com/post/check-web-crawler-impact/A quick bash function to identify which search engine crawlers are hitting your Apache logs and how often, so you can spot unusual traffic patterns at a glance.https://lukehebert.com/post/connecting-ip-on-port-80-count-netstat-ss/Mon, 24 Jan 2022 00:00:00 +0000https://lukehebert.com/post/connecting-ip-on-port-80-count-netstat-ss/Two quick one-liners that count how many unique remote IPs are connected to port 80, using either netstat or ss. Handy for spotting unexpected traffic or verifying that only expected clients are hitting your web server.https://lukehebert.com/post/create-symbols-and-resolve-stack-mysql/Mon, 24 Jan 2022 00:00:00 +0000https://lukehebert.com/post/create-symbols-and-resolve-stack-mysql/Learn how to extract dynamic symbols from the mysqld binary and use them to resolve a crash stack dump into readable function names.https://lukehebert.com/post/dkim-verification-both-must-be-set-to-on/Mon, 24 Jan 2022 00:00:00 +0000https://lukehebert.com/post/dkim-verification-both-must-be-set-to-on/Exim offers two DKIM-related toggles that work together to validate and enforce sender signatures. Setting both to On ensures your mail server actually checks incoming DKIM records and rejects messages that fail validation.https://lukehebert.com/post/verbose-tracking-of-mail-being-sent/Mon, 24 Jan 2022 00:00:00 +0000https://lukehebert.com/post/verbose-tracking-of-mail-being-sent/A quick reference for enabling verbose output when sending mail from the command line, so you can see the full SMTP transaction and diagnose delivery issues.https://lukehebert.com/post/gbb-thread-dump/Mon, 24 Jan 2022 00:00:00 +0000https://lukehebert.com/post/gbb-thread-dump/A quick three-command recipe for grabbing a full thread backtrace from a running httpd process with gdb, useful when Apache appears hung or unresponsive.https://lukehebert.com/post/get-gateway-hostnamed-from-mail-log/Mon, 24 Jan 2022 00:00:00 +0000https://lukehebert.com/post/get-gateway-hostnamed-from-mail-log/A quick bash one-liner to pull offending IP addresses from exim relay-denied entries and resolve them to hostnames, making it easier to identify which gateways are misconfigured.https://lukehebert.com/post/innodb-corruption-and-recovery-guide/Mon, 24 Jan 2022 00:00:00 +0000https://lukehebert.com/post/innodb-corruption-and-recovery-guide/A practical, step-by-step guide for diagnosing and recovering from common InnoDB corruption issues, covering first response, log analysis, checksum verification, and data restoration techniques.https://lukehebert.com/post/inode-counts/Mon, 24 Jan 2022 00:00:00 +0000https://lukehebert.com/post/inode-counts/A quick bash one-liner to identify which directories are consuming the most inodes on a Linux system, helping you pinpoint the source of ‘No space left on device’ errors when disk space still appears available.https://lukehebert.com/post/lsof-check-listening-ports/Mon, 24 Jan 2022 00:00:00 +0000https://lukehebert.com/post/lsof-check-listening-ports/A quick way to list every port a process is listening on, using lsof and a simple grep filter. Handy for troubleshooting connectivity issues or auditing what services are exposed on a host.https://lukehebert.com/post/mod-remote-ip/Mon, 24 Jan 2022 00:00:00 +0000https://lukehebert.com/post/mod-remote-ip/A troubleshooting walkthrough for getting mod_remoteip working correctly under Apache 2.4 and cPanel, including the right LogFormat directives and RemoteIP configuration so that access logs, mod_security, and other modules all see the real client IP.https://lukehebert.com/post/mod-rewrite-block-bad-bot/Mon, 24 Jan 2022 00:00:00 +0000https://lukehebert.com/post/mod-rewrite-block-bad-bot/A quick Apache mod_rewrite snippet that returns a 403 Forbidden to any request whose User-Agent matches a bot pattern. Drop it into your .htaccess and you are done.https://lukehebert.com/post/mod-qos-tailblock/Mon, 24 Jan 2022 00:00:00 +0000https://lukehebert.com/post/mod-qos-tailblock/A lightweight bash script that monitors Apache error logs for repeated access-denied entries and automatically blocks offending IPs via iptables. Useful for quick-and-dirty rate-based blocking without a full WAF.https://lukehebert.com/post/my-trace-route-mtr-packet-loss-testing/Mon, 24 Jan 2022 00:00:00 +0000https://lukehebert.com/post/my-trace-route-mtr-packet-loss-testing/A quick walkthrough of using mtr in report mode to capture packet loss data across network hops. Learn how to run a 30-cycle test and read the output to isolate where loss is occurring.https://lukehebert.com/post/netstat-ss-check-listening-ports/Mon, 24 Jan 2022 00:00:00 +0000https://lukehebert.com/post/netstat-ss-check-listening-ports/A quick reference for checking listening ports on Linux using netstat and ss, including the flags that matter and what each output column means.https://lukehebert.com/post/ngins-flood-protection-with-limit-req/Mon, 24 Jan 2022 00:00:00 +0000https://lukehebert.com/post/ngins-flood-protection-with-limit-req/A practical walkthrough of Nginx’s limit_req module, showing how to cap request rates and protect PHP endpoints from HTTP-based denial-of-service attacks.https://lukehebert.com/post/nginx-non-standard-status-silent-connection-close/Mon, 24 Jan 2022 00:00:00 +0000https://lukehebert.com/post/nginx-non-standard-status-silent-connection-close/Nginx status code 444 silently drops a connection without sending any HTTP response to the client. Learn when and how to use it for security and traffic management.https://lukehebert.com/post/numa-mysql-swap-insanity/Mon, 24 Jan 2022 00:00:00 +0000https://lukehebert.com/post/numa-mysql-swap-insanity/A deep dive into why MySQL on NUMA systems suffers from unpredictable swapping, and how a single numactl interleave fix eliminates the problem entirely.https://lukehebert.com/post/percona-xtradb-recovery/Mon, 24 Jan 2022 00:00:00 +0000https://lukehebert.com/post/percona-xtradb-recovery/Percona XtraDB Cluster can recover a crashed node and rejoin it to the cluster using an incremental state transfer. This post walks through how GTID-based position recovery works under the hood and what you will see in the error log during the process.https://lukehebert.com/post/raid-error-code-key-qualifers/Mon, 24 Jan 2022 00:00:00 +0000https://lukehebert.com/post/raid-error-code-key-qualifers/A quick reference for understanding the key qualifiers that appear in RAID controller error codes, so you can pinpoint the exact failure condition instead of guessing from a generic error message.https://lukehebert.com/post/rewrite-block-page-check-for-patterns-check-google/Mon, 24 Jan 2022 00:00:00 +0000https://lukehebert.com/post/rewrite-block-page-check-for-patterns-check-google/Use Apache mod_rewrite to return a 400 error for a specific URI and optionally block hotlinking from other domains. A quick .htaccess snippet that keeps unwanted traffic off a single page.https://lukehebert.com/post/aws-vpc-structure/Tue, 18 Jan 2022 00:00:00 +0000https://lukehebert.com/post/aws-vpc-structure/A visual walkthrough of the core networking components that make up an Amazon VPC, from subnets and gateways to security groups and route tables. Understanding how these pieces interact is essential for designing secure, well-segmented cloud networks.https://lukehebert.com/post/kafka-dump/Wed, 25 Aug 2021 00:00:00 +0000https://lukehebert.com/post/kafka-dump/A practical walkthrough for dumping Kafka topic data, including a generic consumer approach and a full Atlas-specific workflow covering JAAS configuration, Kerberos authentication, and log extraction.https://lukehebert.com/post/dump-kafka-atlas-topic-to-review-messages/Fri, 20 Aug 2021 00:00:00 +0000https://lukehebert.com/post/dump-kafka-atlas-topic-to-review-messages/A quick walkthrough for dumping the ATLAS_HOOK Kafka topic to inspect Atlas hook messages locally, including Kerberos authentication setup and consumer lag checks.https://lukehebert.com/post/aws-volume-juggling-for-impaired-nodes-and-node-health-check/Wed, 28 Jul 2021 00:00:00 +0000https://lukehebert.com/post/aws-volume-juggling-for-impaired-nodes-and-node-health-check/A practical guide to diagnosing impaired EC2 instances by detaching and reattaching EBS root volumes to a rescue instance. Covers the full workflow from stopping the impaired node to restoring it to normal service.https://lukehebert.com/post/delete-existing-users-from-ranger/Tue, 06 Jul 2021 00:00:00 +0000https://lukehebert.com/post/delete-existing-users-from-ranger/A quick walkthrough of using the Ranger REST API to bulk-delete existing users, including how to capture an auth cookie and loop through a user list with curl.https://lukehebert.com/post/spark-token-debug/Thu, 04 Feb 2021 00:00:00 +0000https://lukehebert.com/post/spark-token-debug/Enable verbose Kerberos and token logging on Spark driver and executor nodes to capture the diagnostic data you need when authentication is failing. Covers log4j configuration files and the exact spark-submit flags to wire them in.https://lukehebert.com/post/common-error-codes-for-active-directory-authentication-acceptsecuritycontext/Tue, 24 Nov 2020 00:00:00 +0000https://lukehebert.com/post/common-error-codes-for-active-directory-authentication-acceptsecuritycontext/A quick reference for the LDAP error codes returned during Active Directory bind operations, including what each code means and how to interpret the AcceptSecurityContext error message.https://lukehebert.com/post/cannot-find-key-of-the-appropriate-type-to-decrypt-ap-rep/Tue, 03 Nov 2020 00:00:00 +0000https://lukehebert.com/post/cannot-find-key-of-the-appropriate-type-to-decrypt-ap-rep/A troubleshooting guide for the Kerberos error ‘Cannot find key of the appropriate type to decrypt AP REP,’ covering encryption type mismatches between keytabs and Active Directory, plus practical steps to align AES and RC4 settings.https://lukehebert.com/post/what-is-the-significance-of-the-version-field-in-a-tls-1-1-clienthello-message/Tue, 04 Dec 2018 00:00:00 +0000https://lukehebert.com/post/what-is-the-significance-of-the-version-field-in-a-tls-1-1-clienthello-message/The version field in a TLS ClientHello message tells the server the highest protocol version the client supports, but it appears in three distinct places with different roles. Understanding where and how each version field is used helps troubleshoot handshake failures and interoperability issues with legacy implementations.https://lukehebert.com/post/ldap-server-side-and-client-side-result-error-codes/Tue, 17 Oct 2017 00:00:00 +0000https://lukehebert.com/post/ldap-server-side-and-client-side-result-error-codes/A quick reference for LDAP result and error codes, covering both server-side responses and client-side indicators, including Active Directory-specific sub-codes under error 49.https://lukehebert.com/post/ptr-records/Tue, 05 Sep 2017 00:00:00 +0000https://lukehebert.com/post/ptr-records/A valid PTR record must resolve to a name that resolves back to the original address. Multiple PTR records for a single address break name resolution and can cause Kerberos ticket verification to fail.https://lukehebert.com/post/failure-unspecified-at-gss-api-level-mechanism-level-checksum-failed/Thu, 31 Aug 2017 00:00:00 +0000https://lukehebert.com/post/failure-unspecified-at-gss-api-level-mechanism-level-checksum-failed/A quick reference for the six most common causes of the Kerberos GSS-API checksum failure error, from wrong passwords to time skew and SPNEGO canonicalization issues.https://lukehebert.com/post/specified-version-of-key-is-not-available-44/Mon, 21 Aug 2017 00:00:00 +0000https://lukehebert.com/post/specified-version-of-key-is-not-available-44/Kerberos error 44 occurs when the keytab KVNO does not match the KDC database. This guide walks through diagnosing the mismatch and regenerating the keytab to restore authentication.https://lukehebert.com/post/java-sslhandshakeexception-no-cipher-suites-in-common/Mon, 21 Aug 2017 00:00:00 +0000https://lukehebert.com/post/java-sslhandshakeexception-no-cipher-suites-in-common/A common Java SSL handshake failure stems from passing a null KeyManager array when initializing the SSLContext, which silently disables all certificate-based cipher suites. This post walks through the root cause and the four most frequent triggers behind the error.https://lukehebert.com/post/handling-multiple-cns-on-a-single-certificate/Fri, 18 Aug 2017 00:00:00 +0000https://lukehebert.com/post/handling-multiple-cns-on-a-single-certificate/Multiple CNs in a certificate Subject field are deprecated and cause compatibility failures. The SAN extension is the correct way to cover multiple names on a single certificate.https://lukehebert.com/post/server-is-unwilling-to-perform-53/Wed, 16 Aug 2017 00:00:00 +0000https://lukehebert.com/post/server-is-unwilling-to-perform-53/LDAP error 53 means the destination server is refusing a password change request. This post walks through the most common Active Directory requirements and policy settings that trigger the error.https://lukehebert.com/post/server-not-found-kerberos/Wed, 16 Aug 2017 00:00:00 +0000https://lukehebert.com/post/server-not-found-kerberos/A quick troubleshooting guide for the ‘Server not found in Kerberos database’ error, covering the four most common root causes and how to verify each one.https://lukehebert.com/post/saml-custom-attributes-in-hue/Fri, 21 Jul 2017 00:00:00 +0000https://lukehebert.com/post/saml-custom-attributes-in-hue/Hue returns a Permission Denied error when the IDP sends custom SAML attributes that pysaml2 cannot map. This guide walks through creating an attribute mapping file and configuring Hue to resolve the issue.https://lukehebert.com/post/400-cannot-find-key-of-appropriate-type-to-decrypt-ap-rep/Tue, 18 Jul 2017 00:00:00 +0000https://lukehebert.com/post/400-cannot-find-key-of-appropriate-type-to-decrypt-ap-rep/A practical troubleshooting guide for the Kerberos AP REP decryption error, covering the most common causes from keytab mismatches to encryption type misconfigurations.https://lukehebert.com/post/trace-kerberos-in-the-cli/Thu, 29 Jun 2017 00:00:00 +0000https://lukehebert.com/post/trace-kerberos-in-the-cli/Kerberos authentication failures can be frustrating to debug. Setting the KRB5_TRACE environment variable gives you detailed library-level logging that pinpoints exactly where the exchange breaks down.https://lukehebert.com/post/check-dns-hostname-resolution-forward-and-backward-python-java/Tue, 27 Jun 2017 00:00:00 +0000https://lukehebert.com/post/check-dns-hostname-resolution-forward-and-backward-python-java/Inconsistent forward and reverse DNS lookups can cause a cascade of cluster and security issues. This guide walks through quick Python and Java one-liners to verify that your hostname, IP, and reverse record all line up.https://lukehebert.com/post/checking-host-resolution-using-getent/Tue, 27 Jun 2017 00:00:00 +0000https://lukehebert.com/post/checking-host-resolution-using-getent/A quick reference for using getent to verify how your Linux system resolves hostnames, short names, and IP addresses through the full NSS chain.https://lukehebert.com/post/cleanup-alternatives/Tue, 27 Jun 2017 00:00:00 +0000https://lukehebert.com/post/cleanup-alternatives/A quick procedure for resolving Cloudera Manager agent failures caused by corrupted system alternatives entries, including the exact commands to stop the agent, purge stale alternatives, and restart cleanly.https://lukehebert.com/post/cm-timezone-does-not-match/Tue, 27 Jun 2017 00:00:00 +0000https://lukehebert.com/post/cm-timezone-does-not-match/Cloudera Manager raises a timezone mismatch alert when the OS clock files disagree with the timezone CM expects. This post walks through the three key files to check and how to align them.https://lukehebert.com/post/compare-knvo-in-db-and-in-keytab-these-should-match/Tue, 27 Jun 2017 00:00:00 +0000https://lukehebert.com/post/compare-knvo-in-db-and-in-keytab-these-should-match/A quick two-step check to confirm the Kerberos key version number in your KDC database matches the value stored in a local keytab file. Mismatched KVNO values are a common cause of authentication failures in Hadoop and other Kerberized services.https://lukehebert.com/post/delete-hue-users/Tue, 27 Jun 2017 00:00:00 +0000https://lukehebert.com/post/delete-hue-users/Hue does not provide a built-in UI option to remove user accounts. This workaround uses the Hue Django shell to delete one or more users directly from the database, with a clear caveat that the process is unsupported.https://lukehebert.com/post/determine-is-a-datanode-is-dead-formula/Tue, 27 Jun 2017 00:00:00 +0000https://lukehebert.com/post/determine-is-a-datanode-is-dead-formula/A quick reference for calculating the exact timeout window before Hadoop marks a Datanode as dead, based on the heartbeat and recheck-interval settings.https://lukehebert.com/post/hostname-and-hostname-f-not-getting-fqdn/Tue, 27 Jun 2017 00:00:00 +0000https://lukehebert.com/post/hostname-and-hostname-f-not-getting-fqdn/When hostname -f returns a short name instead of a fully qualified domain name, the issue usually traces back to the HOSTNAME variable in /etc/sysconfig/network or a misconfigured /etc/hosts entry. Understanding the difference between the gethostname and gethostbyname system calls makes the fix straightforward.https://lukehebert.com/post/jdk-break-downs-for-web-ui-and-spengo-u92_u91_u40_u45/Tue, 27 Jun 2017 00:00:00 +0000https://lukehebert.com/post/jdk-break-downs-for-web-ui-and-spengo-u92_u91_u40_u45/Specific JDK 1.8.0 builds (u40, u45, u91, u92) are known to break SPNEGO authentication and Hadoop web interfaces. This post documents the affected versions, the symptoms, and the recommended downgrade path.https://lukehebert.com/post/kerberos-cli-debug-trace-hadoop/Tue, 27 Jun 2017 00:00:00 +0000https://lukehebert.com/post/kerberos-cli-debug-trace-hadoop/A quick reference for enabling verbose Kerberos and Hadoop debug output on the command line, so you can see exactly where authentication is failing.https://lukehebert.com/post/kerberos-debug-mode-for-the-jvmsparkspnego/Tue, 27 Jun 2017 00:00:00 +0000https://lukehebert.com/post/kerberos-debug-mode-for-the-jvmsparkspnego/Enable Kerberos debug logging in a JVM, Spark driver, and Spark executors to troubleshoot SPNEGO and authentication issues. Covers the exact Java system properties, Spark configuration keys, and how to redirect Spark logs to stdout so Kerberos output interleaves with application logs.https://lukehebert.com/post/keyctl-link-key-has-been-revoked/Tue, 27 Jun 2017 00:00:00 +0000https://lukehebert.com/post/keyctl-link-key-has-been-revoked/A quick guide to diagnosing and resolving the ‘Key has been revoked’ error in the Linux kernel keyring using keyctl commands.https://lukehebert.com/post/namenode-loglevel/Tue, 27 Jun 2017 00:00:00 +0000https://lukehebert.com/post/namenode-loglevel/How to set the HDFS Namenode StateChange log level to DEBUG via the web UI or the command line, and when that helps with troubleshooting.https://lukehebert.com/post/ntp-errors/Tue, 27 Jun 2017 00:00:00 +0000https://lukehebert.com/post/ntp-errors/A host health test flags NTP clock drift when the system clock falls out of sync with its NTP servers. This guide walks through diagnosing and resolving common NTP synchronization failures.https://lukehebert.com/post/ssl-certificate-debugging-openssl-s-client/Tue, 27 Jun 2017 00:00:00 +0000https://lukehebert.com/post/ssl-certificate-debugging-openssl-s-client/A practical cheat sheet of openssl s_client commands for inspecting, verifying, and troubleshooting SSL certificates in production environments.https://lukehebert.com/post/kerberos-krb-ap-err-not-us-or-bad-tgs-server-name/Tue, 27 Jun 2017 00:00:00 +0000https://lukehebert.com/post/kerberos-krb-ap-err-not-us-or-bad-tgs-server-name/A practical look at why the MIT krb5 KDC rejects a TGS-REQ with KRB_AP_ERR_NOT_US, including the exact validation logic and the most common root causes behind the error.https://lukehebert.com/post/wheres-all-my-ram/Fri, 21 Mar 2014 00:00:00 +0000https://lukehebert.com/post/wheres-all-my-ram/Ever wonder why the output of free -m seems to make it look like your system is out of memory?