Puppet Recovery Playbook: Fix Code Deploy Failures, File Sync Issues, and PCP Connection Errors

-

After a UID/GID mismatch and misconfigured file permissions disrupted multiple Puppet services, including pe-puppetserver and file-sync on a replica node, we performed a comprehensive recovery. This post documents all issues, exact errors, and the steps we took to fix them.

Issues & Errors Encountered

1. Puppet Server Restart Failure

Error:

Execution error (FileAlreadyExistsException): /opt/puppetlabs/server/data/analytics/analytics

Fix:

mv /opt/puppetlabs/server/data/analytics/analytics /opt/puppetlabs/server/data/analytics/analytics.bak_wrong
mkdir -p /opt/puppetlabs/server/data/analytics/analytics
chown pe-puppet:pe-puppet /opt/puppetlabs/server/data/analytics/analytics
chmod 750 /opt/puppetlabs/server/data/analytics/analytics

2. puppet code deploy Failed

Error:

invalid or unknown remote ssh hostkey

Fix:

sudo -u pe-puppet ssh-keyscan -p 443 ssh.github.com >> /var/opt/lib/pe-puppet/.ssh/known_hosts
chmod 600 /var/opt/lib/pe-puppet/.ssh/known_hosts
chown pe-puppet:pe-puppet /var/opt/lib/pe-puppet/.ssh/known_hosts

3. UID/GID Mismatch

Observation:

id -a pe-puppet  # Showed incorrect UID/GID

Fix:

find / -xdev -uid <wrong_uid> -exec chown pe-puppet:pe-puppet {} +
find / -xdev -gid <wrong_gid> -exec chown :pe-puppet {} +

4. pe-puppetserver Failed to Start

Error:

Unable to create/set permissions for rundir: /run/puppetlabs/puppetserver

Fix:

chown -R pe-puppet:pe-puppet /run/puppetlabs/puppetserver
chmod 755 /run/puppetlabs/puppetserver

5. Restart Counter File Error

Error:

restartcounter is not readable/writable

Fix:

chown pe-puppet:pe-puppet /opt/puppetlabs/server/data/puppetserver/restartcounter

6. Serial File .tmp Permission Errors

Error:

Execution error (FileSystemException): /etc/puppetlabs/puppetserver/ca/infra_serials<rand>tmp: Operation not permitted

Fix:

chown pe-puppet:pe-puppet /etc/puppetlabs/puppetserver/ca/*.tmp

Validation:

id -a pe-puppet  # Verified GID was correct, not inherited from pe-host-action-collector

7. Token Expired During Replica Provisioning

Error:

The provided token has expired

Fix:

puppet access login

8. Replica Not Connected to PCP

Error:

The node <replica-server> is not connected to the primary via PCP

Fix:

  • Added entries in /etc/hosts on both nodes
  • Started the pxp-agent service:
systemctl start pxp-agent
  • Initiated CSR from replica:
puppet agent -t
  • Signed it on the primary:
puppetserver ca sign --certname <replica-server>

9. File Sync Not Running on Replica

Error:

Couldn't connect to server (https://puppet-master:8140/file-sync/v1/latest-commits): (Connection refused)

Fix:

  • Ensured correct UID/GID:
id -a pe-puppet
  • Corrected ownership on critical files/directories:
find / -uid <wrong_uid> -exec chown pe-puppet:pe-puppet {} +

Final Validation

puppet infrastructure status

Expected Output:

All services are fully operational on both primary and replica

Summary

  • Ownership and permissions were the root cause of cascading service failures.
  • Fixing UID/GID mismatches, cleaning up .tmp files, and ensuring valid certificates resolved all issues.
  • Documenting each error and its corresponding fix was essential for recovery and future-proofing.

Comments

Post a Comment

Popular Posts

Puppet Code Deploy Troubleshooting & Resolution Guide

-
Initial Problem I was setting up Puppet Enterprise and trying to push code using the trusted puppet code deploy command. But instead of a clean deployment, I got this surprise: puppet code deploy --dry-run And boom — this lovely error shows up: [POST /deploys][500] Errors while collecting a list of environments to deploy (exit code: 1). ERROR -> failed to stat '/var/opt/lib/pe-puppet/.gitconfig' So... what gives? Turns out this issue is part permissions, part SSH trust, and part “who ran this command and as which user.” But don’t worry — I’ll walk through all root causes and their real-world fixes step by step. Root Cause #1: Directory Permission What Happened: The directory /var/opt/lib was locked down with drwx------ (700) and owned by root . User pe-puppet couldn’t even step inside it. Since r10k runs as pe-puppet , it failed trying to read .gitconfig . The Fix: chmod 711 /var/opt/lib Why: This lets pe-puppet enter the folder ...
Read more

Fix: SSH Permission Denied Issue | Real Solution

-
1. Verify SSH Command and Key Usage Use the correct private key with -i : ssh -i ~/.ssh/your_private_key user@host Ensure the key is not passphrase-protected (or enter the passphrase correctly). 2. Check Key Generation and Deployment Regenerate Keys (if unsure): ssh-keygen -t ed25519 -C "your_email@example.com" # Prefer ed25519 Ed25519 uses elliptic curve cryptography (Edwards-curve Digital Signature Algorithm) and provides 128-bit security (equivalent to RSA-3072 or RSA-4096) with a 256-bit key. Copy the Public Key to the Server: ssh-copy-id -i ~/.ssh/your_public_key user@host If ssh-copy-id isn’t available, manually append the public key to ~/.ssh/authorized_keys on the server. 3. Fix File/Directory Permissions On the Server: chmod 700 ~/.ssh chmod 600 ~/.ssh/authorized_keys chmod go-w ~ # Home directory should not be world-writable On the Client (Local Machine): chmod 600 ~/.ssh/your_private_key 4. Check SSH Server Configuration Edit /etc/ssh/s...
Read more

Linux Process Termination Signals Explained with Examples

-
Linux uses signals to communicate with processes, allowing them to terminate, pause, or handle specific events. Below are key termination signals, examples of how to use them, and their effects. 1. SIGINT (2) : Interrupt Signal Default Action : Terminate the process. Trigger : Press Ctrl+C in the terminal. Use Case : Gracefully stop a running command (e.g., a script or program). Example: # Start a long-running process sleep 100 # Press Ctrl+C to send SIGINT and terminate it Handling SIGINT in a Bash Script: #!/bin/bash trap 'echo "SIGINT caught! Exiting..."; exit' SIGINT echo "Running... Press Ctrl+C to test SIGINT" while true; do sleep 1 done Output: Running... Press Ctrl+C to test SIGINT ^CSIGINT caught! Exiting... 2. SIGQUIT (3) : Quit Signal Default Action : Terminate and generate a core dump. Trigger : Press Ctrl+\ in the terminal. Use Case : Debugging (generates a core dump for post-mortem analysis). ...
Read more