Complete Guide to Backup and Restore Puppet Master Serve

-

Puppet Master Server Backup Guide

1. Prepare for Backup

Ensure you have working backups of the Primary, Replica, and Compilers. Before initiating the backup, stop the pe_databases module timers to prevent pg_repack from interfering:

systemctl stop pe_databases-*.timer

2. Create Backup

Run the backup command on the Primary server:

sudo puppet-backup create --dir=<BACKUP_DIRECTORY> --name=<BACKUP_NAME>

3. Backup Secret Keys

Secure the secret keys used by Orchestration and LDAP services:

  • Orchestration: /etc/puppetlabs/orchestration-services/conf.d/secrets/
  • LDAP (if applicable): /etc/puppetlabs/console-services/conf.d/secrets/keys.json

4. Restart Database Timers

systemctl start pe_databases-catalogs.timer pe_databases-facts.timer pe_databases-other.timer
systemctl status pe_databases-*.timer

Puppet Master Server Restore Guide

1. Stop Database Timers

systemctl stop pe_databases-*.timer

2. Uninstall PE on the Restore Target

sudo /opt/puppetlabs/bin/puppet-enterprise-uninstaller -p -d

Ensure data under /opt/puppetlabs/ and /etc/puppetlabs/ is removed.

3. Reinstall Puppet Enterprise

Install the same version of PE used for backup.

sudo ./puppet-enterprise-installer

4. Restore Backup

On your primary server, run the puppet-backup restore command to restore your PE infrastructure. The default command is:

export TMPDIR=/opt/puppetlabs/restore-backup ## Create directory under filesystem which has sufficient space
sudo puppet-backup restore <BACKUP-FILENAME>

5. Restore Secret Keys

  • Orchestrator: /etc/puppetlabs/orchestration-services/conf.d/secrets/
  • LDAP: /etc/puppetlabs/console-services/conf.d/secrets/keys.json
chown pe-orchestration-services:pe-orchestration-services /path/to/orchestration/keys
chown pe-console-services:pe-console-services /path/to/ldap/keys.json

6. Restart Services

systemctl restart pe-orchestration-services pe-console-services

7. Apply Puppet Configuration

puppet agent -t --no-use_cached_catalog

Run the above command twice.

It does NOT use the last cached catalog from /opt/puppetlabs/puppet/cache/state/, Instead fetches a fresh catalog from the Puppet Server

8. Validate Agent Connectivity

puppet agent -t --no-use_cached_catalog

9. Handle Replica Removal and Reprovisioning

puppet infrastructure forget --force <REPLICA_CERTNAME>
puppet agent -t

When you restore a Puppet primary server from backup, the restored state might have outdated or missing information about the replica, such as:

  • SSL certificates
  • RBAC tokens
  • CA metadata
  • Code sync states

This can cause mismatch or trust issues between the primary and the replica.

10. Uninstall Agent on Replica

/opt/puppetlabs/bin/puppet-enterprise-uninstaller
rm -rf /opt/puppetlabs /etc/puppetlabs

11. Validate if puppet RPM is removed.

rpm -qa | grep -i puppet

12. Install the agent on replica node

curl -k https://<primary_server>:8140/packages/current/install.bash | sudo bash
puppet agent -t

13. On the primary server, as the root user, run

puppet infrastructure provision replica <REPLICA_NODE_NAME> --enable

This command is used to provision a Puppet Primary Replica node.

Sets up the specified node as a replica of the primary server.

Automatically installs and configures necessary components:

  • Puppet Server
  • PuppetDB
  • Console services
  • Syncs code, SSL certificates, and RBAC settings from the primary.
  • Registers the replica in the infrastructure topology.
  • --enable makes the replica active immediately after provisioning.

14. Run Puppet Job

If you wish to immediately run Puppet on all your agents, you can do so with this command:

puppet job run --no-enforce-environment --query 'nodes {deactivated is null and expired is null}'

Validation Steps

1. Infra Status Check

puppet infra status

Ensure all services are running and syncing correctly.

2. Code Deployment Check

sudo puppet-code deploy --dry-run

If errors occur, check SSH configuration for Git access:

touch ~pe-puppet/.ssh/config
chmod 600 ~pe-puppet/.ssh/config
chown pe-puppet:pe-puppet ~pe-puppet/.ssh/config
cat ~pe-puppet/.ssh/config
host <git-host>
  HostName <git-host>
  IdentityFile /etc/puppetlabs/puppetserver/ssh/id-control_repo.rsa
  User puppet-batch
  StrictHostKeyChecking no
systemctl restart pe-puppetserver
sudo puppet-code deploy --dry-run

This guide helps maintain a reliable Puppet infrastructure with full backup, restore, and validation steps. Adjust all server names and paths according to your actual environment.

Comments

Post a Comment

Popular Posts

Puppet Code Deploy Troubleshooting & Resolution Guide

-
Initial Problem I was setting up Puppet Enterprise and trying to push code using the trusted puppet code deploy command. But instead of a clean deployment, I got this surprise: puppet code deploy --dry-run And boom — this lovely error shows up: [POST /deploys][500] Errors while collecting a list of environments to deploy (exit code: 1). ERROR -> failed to stat '/var/opt/lib/pe-puppet/.gitconfig' So... what gives? Turns out this issue is part permissions, part SSH trust, and part “who ran this command and as which user.” But don’t worry — I’ll walk through all root causes and their real-world fixes step by step. Root Cause #1: Directory Permission What Happened: The directory /var/opt/lib was locked down with drwx------ (700) and owned by root . User pe-puppet couldn’t even step inside it. Since r10k runs as pe-puppet , it failed trying to read .gitconfig . The Fix: chmod 711 /var/opt/lib Why: This lets pe-puppet enter the folder ...
Read more

Fix: SSH Permission Denied Issue | Real Solution

-
1. Verify SSH Command and Key Usage Use the correct private key with -i : ssh -i ~/.ssh/your_private_key user@host Ensure the key is not passphrase-protected (or enter the passphrase correctly). 2. Check Key Generation and Deployment Regenerate Keys (if unsure): ssh-keygen -t ed25519 -C "your_email@example.com" # Prefer ed25519 Ed25519 uses elliptic curve cryptography (Edwards-curve Digital Signature Algorithm) and provides 128-bit security (equivalent to RSA-3072 or RSA-4096) with a 256-bit key. Copy the Public Key to the Server: ssh-copy-id -i ~/.ssh/your_public_key user@host If ssh-copy-id isn’t available, manually append the public key to ~/.ssh/authorized_keys on the server. 3. Fix File/Directory Permissions On the Server: chmod 700 ~/.ssh chmod 600 ~/.ssh/authorized_keys chmod go-w ~ # Home directory should not be world-writable On the Client (Local Machine): chmod 600 ~/.ssh/your_private_key 4. Check SSH Server Configuration Edit /etc/ssh/s...
Read more

Linux Process Termination Signals Explained with Examples

-
Linux uses signals to communicate with processes, allowing them to terminate, pause, or handle specific events. Below are key termination signals, examples of how to use them, and their effects. 1. SIGINT (2) : Interrupt Signal Default Action : Terminate the process. Trigger : Press Ctrl+C in the terminal. Use Case : Gracefully stop a running command (e.g., a script or program). Example: # Start a long-running process sleep 100 # Press Ctrl+C to send SIGINT and terminate it Handling SIGINT in a Bash Script: #!/bin/bash trap 'echo "SIGINT caught! Exiting..."; exit' SIGINT echo "Running... Press Ctrl+C to test SIGINT" while true; do sleep 1 done Output: Running... Press Ctrl+C to test SIGINT ^CSIGINT caught! Exiting... 2. SIGQUIT (3) : Quit Signal Default Action : Terminate and generate a core dump. Trigger : Press Ctrl+\ in the terminal. Use Case : Debugging (generates a core dump for post-mortem analysis). ...
Read more