Puppet Code Deploy Troubleshooting & Resolution Guide

Initial Problem

I was setting up Puppet Enterprise and trying to push code using the trusted puppet code deploy command.

But instead of a clean deployment, I got this surprise:

puppet code deploy --dry-run

And boom — this lovely error shows up:

[POST /deploys][500] Errors while collecting a list of environments to deploy (exit code: 1).
ERROR -> failed to stat '/var/opt/lib/pe-puppet/.gitconfig'

So... what gives? Turns out this issue is part permissions, part SSH trust, and part “who ran this command and as which user.” But don’t worry — I’ll walk through all root causes and their real-world fixes step by step.

Root Cause #1: Directory Permission

What Happened:

The directory /var/opt/lib was locked down with drwx------ (700) and owned by root.
User pe-puppet couldn’t even step inside it.
Since r10k runs as pe-puppet, it failed trying to read .gitconfig.

The Fix:

chmod 711 /var/opt/lib

Why:

This lets pe-puppet enter the folder but not peek inside. Just how we like it: access, but no funny business.

Root Cause #2: SSH Host Key MIA

New Error After Fixing Permissions:

invalid or unknown remote ssh hostkey at /opt/puppetlabs/server/data/code-manager/git/ssh---ssh.github.com-443-<repo>.git

Explanation:

Puppet connects to ssh.github.com on port 443, not the usual 22.
pe-puppet didn’t have the SSH fingerprint for the host.
So... trust issues. Again.

The Fix:

Make sure to add each host key to the puppet_enterprise::profile::master::r10k_known_hosts parameter in the console.

Use the command below to generate the SSH key and append it to the known_hosts file:

sudo -u pe-puppet ssh-keyscan -p 443 ssh.github.com >> /var/opt/lib/pe-puppet/.ssh/known_hosts

Also ensure the known_hosts file is present and properly owned:

sudo -u pe-puppet mkdir -p /var/opt/lib/pe-puppet/.ssh
sudo chmod 700 /var/opt/lib/pe-puppet/.ssh
sudo chown -R pe-puppet:pe-puppet /var/opt/lib/pe-puppet/.ssh

sudo -u pe-puppet ssh-keyscan ssh.github.com >> /var/opt/lib/pe-puppet/.ssh/known_hosts
sudo chmod 600 /var/opt/lib/pe-puppet/.ssh/known_hosts
sudo chown pe-puppet:pe-puppet /var/opt/lib/pe-puppet/.ssh/known_hosts

Example for r10k_known_hosts parameter (JSON-style array):

[
  {
    "name": "github.com",
    "type": "ssh-ed25519",
    "key": "AAAACxxxxxxxx"
  }
]

Root Cause #3: Ownership Crisis

What Went Wrong:

known_hosts was owned by root due to an earlier ssh-keyscan run, and pe-puppet couldn’t modify it.

The Fix:

sudo chown -R pe-puppet:pe-puppet /var/opt/lib/pe-puppet/.ssh
sudo chmod 700 /var/opt/lib/pe-puppet/.ssh
sudo chmod 600 /var/opt/lib/pe-puppet/.ssh/known_hosts

Step-by-Step Verification

Test SSH fingerprinting:

sudo -u pe-puppet ssh -T -p 443 git@ssh.github.com

Expected:

Prompt to trust the host key
Permission denied (publickey) is fine — it means host key trust is set

Run Puppet code deploy again:

puppet code deploy --dry-run

Expected Output:

[
  { "environment": "production" }
]

Optional Best Practice: SSH Config

Create this file for better readability and behavior:

/var/opt/lib/pe-puppet/.ssh/config

Content:

Host github.com
    HostName ssh.github.com
    Port 443
    User git

Set Permissions:

sudo chown pe-puppet:pe-puppet /var/opt/lib/pe-puppet/.ssh/config
sudo chmod 600 /var/opt/lib/pe-puppet/.ssh/config

Don’t Forget

Each user has a separate .ssh/ configuration
Only pe-puppet matters for puppet code deploy
Fix SELinux context if enabled:

restorecon -Rv /var/opt/lib/pe-puppet

Final Checklist for Other Servers

/var/opt/lib has permission 711
.ssh directory exists under /var/opt/lib/pe-puppet/
SSH keyscan is run as pe-puppet
Ownership is pe-puppet, permissions are 700/600
Optional: SSH config file for GitHub access via port 443
puppet code deploy --dry-run completes successfully

That’s a wrap on debugging this Puppet tantrum.
If this saved your deploy today, drop a comment, share it with your team.

#EndOfLog — Happy deploying!

Search This Blog

FixnLogin