Posts

Showing posts from April, 2025

Complete Guide to Backup and Restore Puppet Master Serve

Puppet Master Server Backup Guide 1. Prepare for Backup Ensure you have working backups of the Primary , Replica , and Compilers . Before initiating the backup, stop the pe_databases module timers to prevent pg_repack from interfering: systemctl stop pe_databases-*.timer 2. Create Backup Run the backup command on the Primary server: sudo puppet-backup create --dir=<BACKUP_DIRECTORY> --name=<BACKUP_NAME> 3. Backup Secret Keys Secure the secret keys used by Orchestration and LDAP services: Orchestration: /etc/puppetlabs/orchestration-services/conf.d/secrets/ LDAP (if applicable): /etc/puppetlabs/console-services/conf.d/secrets/keys.json 4. Restart Database Timers systemctl start pe_databases-catalogs.timer pe_databases-facts.timer pe_databases-other.timer systemctl status pe_databases-*.timer Puppet Master Server Restore Guide 1. Stop Database Timers systemctl stop pe_databases-*.timer 2. Uninstall PE on the Restore Target sudo /opt/pupp...

Auto Sign Puppet Agent Certificates Without Manual Approval

How to Enable Autosign for Puppet Agents If you're looking to automate the certificate signing process for Puppet agents, follow these tested and verified steps based on a real support case. Step 1: Enable Autosign on the Primary Server Edit the Puppet configuration file on your Primary Server : sudo nano /etc/puppetlabs/puppet/puppet.conf Add line "autosign = true" under the [main] section: [main] certname = puppet-master.example.com server = puppet-master.example.com user = pe-puppet group = pe-puppet environment_timeout = 0 module_groups = base+pe_only autosign = true Save and exit the file. Step 2: Clean the Agent Certificate on Primary To remove any conflicting certificate on the primary server, run: puppetserver ca clean --certname agent-node.example.com Replace agent-node.example.com with your actual agent's FQDN. Step 3: Clean SSL Certs on the Agent Node Now go to the agent node and clean its certificates: puppet ssl cl...

Why SELinux Is Blocking Your Service (and How to Fix It)

Troubleshooting SELinux-Related Issues in Linux Running services with SELinux enabled can sometimes lead to unexpected denials or blocked actions—especially if SELinux was previously disabled or the service is configured in a non-standard way. In most cases, SELinux issues are the result of misconfigured contexts, policies, or port bindings. Here's a step-by-step approach to diagnose and fix SELinux-related problems while keeping your system secure. Check SELinux Status and Mode Start by verifying that SELinux is active on your system: # sestatus Look for: SELinux status: enabled Current mode: enforcing or permissive To test if SELinux is causing a problem, temporarily switch to permissive mode: # setenforce 0 # Logs denials but doesn’t block actions # setenforce 1 # Reverts to enforcing mode If the issue disappears in permissive mode, SELinux is likely the root cause. Look for SELinux Denials i...

Double the Tomcat: Run Two Instances on One Server

Creating a Custom Unit File for a Second Instance of Apache Tomcat Need to run multiple instances of Tomcat on the same machine? You can set up a second instance by duplicating the configuration and customizing key parameters to avoid conflicts with the first instance. Step-by-Step Procedure 1. Duplicate the Tomcat Installation Start by copying the existing Tomcat directory (typically /opt/tomcat ): # cp -r /opt/tomcat /opt/tomcat-second This creates an independent folder for the second instance, which you can configure separately. 2. Customize Ports and Settings Edit /opt/tomcat-second/conf/server.xml and modify ports to avoid clashes with the default instance. For example: HTTP Connector: Change port="8080" to port="8081" Shutdown Port: Change port="8005" to something like port="8006" AJP Connector: Change port="8009" to port="8010" 3. Create a Custom systemd Un...

Step-by-Step Guide to Writing Custom systemd Unit Files

Creating Custom Unit Files in systemd Creating Custom Unit Files in systemd Systemd unit files give you the power to define, control, and manage background services on a Linux system. Whether you're building your own script-based service or integrating a third-party executable, creating a custom unit file ensures it behaves just the way you want it to. This guide walks you through the process step-by-step—from preparing your executable to activating your service in the systemd ecosystem. Step 1: Prepare Your Executable Every custom service begins with an executable — this could be a compiled binary or a script that performs a defined task. Think of this as the “engine” your systemd unit will start, stop, and monitor. Example: Suppose you have a simple shell script that backs up a directory every night. You might write something like this: #!/bin/bash tar -czf /var/backups/home_backup_$(date +%F).tar.gz /home/myuser/ ...

Fix EC2 NVMe Mount Issues After Reboot Using UUIDs (AWS EBS Guide)

Fix EC2 NVMe Mount Mismatch After Reboot — The UUID Way When working with multiple EBS volumes on an EC2 instance—especially NVMe-backed ones—you might notice that after a reboot, your mount points suddenly don't match the volumes you expected. This guide explains the root cause, shows how to fix it using UUIDs, and shares some patterns and pro tips around volume device names. The Problem After rebooting, your mount points might look correct on the surface but are actually pointing to the wrong disks. Before reboot: /dev/nvme1n1 → 300G → /opt/puppetlabs /dev/nvme2n1 → 50G → /var/log/puppetlabs /dev/nvme3n1 → 30G → /etc/puppetlabs After reboot: /dev/nvme1n1 → 30G → /opt/puppetlabs ❌ /dev/nvme2n1 → 50G → /var/log/puppetlabs ✅ /dev/nvme3n1 → 300G → /etc/puppetlabs ❌ This can cause services to fail or logs to be written in the wrong location. Root Cause: NVMe Device Name Reordering Device names like /dev/nvme1n1 are assigned dynamically by the Linux kern...

Puppet Recovery Playbook: Fix Code Deploy Failures, File Sync Issues, and PCP Connection Errors

After a UID/GID mismatch and misconfigured file permissions disrupted multiple Puppet services, including pe-puppetserver and file-sync on a replica node, we performed a comprehensive recovery. This post documents all issues, exact errors, and the steps we took to fix them. Issues & Errors Encountered 1. Puppet Server Restart Failure Error: Execution error (FileAlreadyExistsException): /opt/puppetlabs/server/data/analytics/analytics Fix: mv /opt/puppetlabs/server/data/analytics/analytics /opt/puppetlabs/server/data/analytics/analytics.bak_wrong mkdir -p /opt/puppetlabs/server/data/analytics/analytics chown pe-puppet:pe-puppet /opt/puppetlabs/server/data/analytics/analytics chmod 750 /opt/puppetlabs/server/data/analytics/analytics 2. puppet code deploy Failed Error: invalid or unknown remote ssh hostkey Fix: sudo -u pe-puppet ssh-keyscan -p 443 ssh.github.com >> /var/opt/lib/pe-puppet/.ssh/known_hosts chmod 600 /var/opt/lib/pe-puppet/.ssh/known_hosts ch...

Puppet Code Deploy Troubleshooting & Resolution Guide

Initial Problem I was setting up Puppet Enterprise and trying to push code using the trusted puppet code deploy command. But instead of a clean deployment, I got this surprise: puppet code deploy --dry-run And boom — this lovely error shows up: [POST /deploys][500] Errors while collecting a list of environments to deploy (exit code: 1). ERROR -> failed to stat '/var/opt/lib/pe-puppet/.gitconfig' So... what gives? Turns out this issue is part permissions, part SSH trust, and part “who ran this command and as which user.” But don’t worry — I’ll walk through all root causes and their real-world fixes step by step. Root Cause #1: Directory Permission What Happened: The directory /var/opt/lib was locked down with drwx------ (700) and owned by root . User pe-puppet couldn’t even step inside it. Since r10k runs as pe-puppet , it failed trying to read .gitconfig . The Fix: chmod 711 /var/opt/lib Why: This lets pe-puppet enter the folder ...

Linux Process Termination Signals Explained with Examples

Linux uses signals to communicate with processes, allowing them to terminate, pause, or handle specific events. Below are key termination signals, examples of how to use them, and their effects. 1. SIGINT (2) : Interrupt Signal Default Action : Terminate the process. Trigger : Press Ctrl+C in the terminal. Use Case : Gracefully stop a running command (e.g., a script or program). Example: # Start a long-running process sleep 100 # Press Ctrl+C to send SIGINT and terminate it Handling SIGINT in a Bash Script: #!/bin/bash trap 'echo "SIGINT caught! Exiting..."; exit' SIGINT echo "Running... Press Ctrl+C to test SIGINT" while true; do sleep 1 done Output: Running... Press Ctrl+C to test SIGINT ^CSIGINT caught! Exiting... 2. SIGQUIT (3) : Quit Signal Default Action : Terminate and generate a core dump. Trigger : Press Ctrl+\ in the terminal. Use Case : Debugging (generates a core dump for post-mortem analysis). ...

Linux Redirection Magic: Why command > /dev/null 2>&1 Works and command 2>&1 /dev/null Doesn’t

“Ever run a command in Linux and been swamped by an ocean of text flooding your terminal? Yeah, we've all been there.” But what if you don't want all that chatter? Linux has your back—enter the magic of redirections like /dev/null 2>&1 . But hold up, have you ever found yourself typing something like command 2>&1 /dev/null and wondered why your shell suddenly threw a tantrum? Well, today we're diving deep (but keeping it super simple!) into why one little twist in syntax matters so much. Let's clear up this confusion once and for all—trust me, your terminal will thank you. First Things First: Understanding the Basics In Linux or Unix-based systems, every command you run communicates through standard streams: Standard Input (stdin): Where your command takes input from (usually your keyboard). Standard Output (stdout): Where the command prints its normal output. Standard Error (stderr): Where the command sends error messages. Linux allows y...

Linux Command to Add a Prefix to Each Line

Adding a Prefix to Each Line in Linux Here are common methods to add a prefix to every line of a file: 1. Using sed Replace PREFIX with your desired string: sed 's/^/PREFIX/' input.txt Example: sed 's/^/Hello /' file.txt This turns: World Linux into: Hello World Hello Linux 2. Using awk Basic usage: awk '{print "PREFIX"$0}' input.txt Example: awk '{print "https://"$0}' domains.txt Output: https://example.com https://linux.org 3. In-Place File Modification Add -i to sed or perl to edit files directly: sed -i 's/^/PREFIX/' input.txt perl -i -pe 's/^/PREFIX/' input.txt 4. Handling Special Characters If the prefix includes slashes, change the sed delimiter: sed 's|^|/usr/local/|' paths.txt For quotes or spaces in awk, escape them properly: awk '{print "\"Quote: \""$0}' text.txt 5. Adding Prefix While Using Pipelines cat input.txt | sed 's/^...

Merging Two Files with AWK: A Guide

To merge two files using awk , you can use different strategies depending on whether you want to combine columns (e.g., join files based on a common key) or append rows (e.g., concatenate files vertically). Below are practical examples for both scenarios: 1. Merge Two Files Line-by-Line (Combine Columns) If both files have the same number of lines and you want to merge them side by side (like paste ), use awk to read both files sequentially. Example Files: file1.txt: Apple Banana Cherry file2.txt: 100 200 300 AWK Command: awk 'NR==FNR {a[NR]=$0; next} {print a[FNR], $0}' file1.txt file2.txt Output: Apple 100 Banana 200 Cherry 300 Explanation: NR==FNR : True while processing the first file ( file1.txt ). a[NR]=$0 : Store each line of file1.txt in an array. next : Skip to the next line (prevents processing the second file yet). print a[FNR], $0 : For file2.txt , print the line from file1.txt with the current line. 2. Merge Two Files Base...

Fix: SSH Permission Denied Issue | Real Solution

1. Verify SSH Command and Key Usage Use the correct private key with -i : ssh -i ~/.ssh/your_private_key user@host Ensure the key is not passphrase-protected (or enter the passphrase correctly). 2. Check Key Generation and Deployment Regenerate Keys (if unsure): ssh-keygen -t ed25519 -C "your_email@example.com" # Prefer ed25519 Ed25519 uses elliptic curve cryptography (Edwards-curve Digital Signature Algorithm) and provides 128-bit security (equivalent to RSA-3072 or RSA-4096) with a 256-bit key. Copy the Public Key to the Server: ssh-copy-id -i ~/.ssh/your_public_key user@host If ssh-copy-id isn’t available, manually append the public key to ~/.ssh/authorized_keys on the server. 3. Fix File/Directory Permissions On the Server: chmod 700 ~/.ssh chmod 600 ~/.ssh/authorized_keys chmod go-w ~ # Home directory should not be world-writable On the Client (Local Machine): chmod 600 ~/.ssh/your_private_key 4. Check SSH Server Configuration Edit /etc/ssh/s...