puppet client reporting to wrong host in Foreman

Question

This is my first post!

I have 100's of nodes managed by puppet/foreman. Everything is fine. I did something I already did without problem in the past: Change the hostname of a server.

This time I changed 2 hostnames:

Initially I had 'gate02' and 'gate03'.

I moved gate02 to 'gate02old' (with dummy IP, and switched the server OFF)

then I moved gate03 to gate02 ...

Now (the new) gate02 reports are updating the host called gate02old in foreman.

I did clean the certs in the puppetserver. I rm the ssl dir in the (new) gate02 and run puppet agent. I did not fing any reference to 'gate' in /var/lib/puppet. I changed the certname in puppet.conf and in hostname, and in sysconfig/network-script/ifcfg-xxxx.

The puppet agent run smoothly, and sends it to the puppetserver. But it updates the wrong host!

Anyone would have a clue on how to fix this ?

Thanks!

Foreman 2.0.3 Puppet 6

Answer 1

I do not accept that the sequence of events described led to the behavior described. If reports for the former gate03, now named gate02, are being logged on the server for name gate02old, then that is because that machine is presenting a cert to the server that identifies it as gate02old (and the server is accepting that cert). The sequence of events presented does not explain how that might be, but my first guess would be that it is actually (new) gate02old that is running and requesting catalogs from the server, not (new) gate02.

Fix it by

0. Ensuring that the machine you want running is in fact the one that is running, and that its hostname is in fact what you intend for it to be.

1. Shutting down the agent on (new) gate02. If it is running in daemon mode then shut down the daemon and disable it. If it is being scheduled by an external scheduler then stop and disable it there. Consider also using puppet agent --disable.

2. Deactivating the node on the server and cleaning out its data, including certs:

   puppet node deactivate gate02
   puppet node deactivate gate02old
   puppet node deactivate gate03
   

   You may want to wait a bit at this point, then ...

   puppet node clean gate02
   puppet node clean gate02old
   puppet node clean gate03
   

3. Cleaning out the nodes' certs. For safety, I would do this on both nodes. Removing /opt/puppetlabs/puppet/ssl (on the nodes, not the server!) should work for that, or you could remove the puppet-agent package altogether, remove any files left behind, and then reinstall.

4. Updating the puppet configuration on the new gate02 as appropriate.

5. Re-enabling the agent on gate02, and starting it or running it in --test mode.

6. Signing the new CSR (unless you have autosigning enabled), which should have been issued for gate02 or whatever certname is explicitly specified in in that node's puppet configuration.

Answer 2

Thanks for the answer, though it was not the right one. I did get to the right point by changing again the hostname of the old gateold02 to a another existing-for-testing one, starting the server and get it back in Foreman. Once that done, removing (again!) the certs of the new gate02 put it right, and its reports now updates the right entry in Foreman.

I still beleive there is something (a db ?) that was not updated right so foreman was sure that the host called 'gate02' was in the GUI 'gateold02'.

I am very sorry if you don't beleive me.

Not to say rather disappointed.

Cheers.

Answer 3

Unfortunately I had the same issue after renaming a node in the UI (Foreman Version 3.4.1) and adding a new node with the same hostname as the initial one from the renamed host.

cleaning and reissuing certs of both nodes did not help, what did help was

• delete the renamed host from foreman
• add it back with the new name (without build)
• clean certificates (rm -rf /etc/puppetlabs/puppet/ssl/) on the respective node and issue new certificates with the new hostname (the hostname of the node was already adjusted in the initial attempt to rename the node)

after that all reports were associated with the correct nodes.