Tuesday, December 24, 2013

Getting the latest 2.7.x stable Puppet on Ubuntu and Redhat.

Quick guide for installing the latest 2.7.x stable version of Puppet.

I have a need to get a bunch of Ubuntu 10.x and 12.x LTS servers all running the same version of Puppet community.   I tried having a mix-mash of Puppet versions 3.x 2.x and 0.25.x; this does not work well, lots of random problems.

To solve this issue I am going to install the latest 2.x stable version, at the time of this writing (2013/12/24) it is 2.7.22.



Install the Puppetlabs apt-repository, puppet software, and hold the version:

Ubuntu 10.04:

$ wget http://apt.puppetlabs.com/puppetlabs-release-lucid.deb
$ sudo dpkg -i puppetlabs-release-lucid.deb
$ sudo apt-get update
$ sudo apt-get install puppet=2.7.22-1puppetlabs1 puppet-common=2.7.22-1puppetlabs1
$ echo "puppet hold" | sudo dpkg  --set-selections
$ echo "puppet-common hold" | sudo dpkg  --set-selections

Ubuntu 12.04:

$ wget http://apt.puppetlabs.com/puppetlabs-release-precise.deb
$ sudo dpkg -i puppetlabs-release-precise.deb
$ sudo apt-get update
$ sudo apt-get install puppet=2.7.22-1puppetlabs1 puppet-common=2.7.22-1puppetlabs1
$ sudo apt-mark hold puppet puppet-common


NOTES: While the hold is set, no automatic security updates will be installed for Puppet.  Make sure you update as necessary.  



Additional Notes for Puppet server on Ubuntu:

12.04:

$ sudo apt-get install  puppet=2.7.22-1puppetlabs1 puppet-common=2.7.22-1puppetlabs1 puppetmaster=2.7.22-1puppetlabs1 puppetmaster-common=2.7.22-1puppetlabs1
$ sudo apt-mark hold  puppet puppet-common puppetmaster puppetmaster-common


Install Puppetlabs yum repository, puppet software, and exclude upgrades:

6:

$ sudo rpm -ivh http://yum-enterprise.puppetlabs.com/el/6/extras/x86_64/puppetlabs-enterprise-release-extras-6-2.noarch.rpm
$ sudo yum --showduplicates list puppet  # Put this here so I remember the command
$ sudo yum install puppet-2.7.22-1.el6
$ sudo vi /etc/yum.conf  # add exclude=puppet to [main] block


Monday, December 23, 2013

Fixing Rancid update problems (same diffs every day).

Rancid was email reports indicating the same changes were being made every day.  Basically new changes were not getting pushed into Rancid like they were supposed to.  

File permissions came to mind when I initially found the problem.   There is a FAQ that tells you how to resolve this.   We have had similar problem when someone ran Rancid as the 'root' user instead of 'rancid' like they should be.

This was not our problem, the problem was CVS was out of sync.


First check the status by running:

`cvs status`:

===================================================================
File: bn-001-0.nts.wustl.edu Status: Needs Checkout
   Working revision: 1.39 2013-12-22 22:43:15 -0600
   Repository revision: 1.39 /var/lib/rancid/CVS/wcdc/configs/bn-001-0.nts.wustl.edu,v
   Commit Identifier: boZWMtfaLtmxtsdx
   Sticky Tag: (none)
   Sticky Date: (none)
   Sticky Options: (none)
===================================================================
File: bn-002-0.nts.wustl.edu Status: Locally Modified
   Working revision: 1.18 2013-12-15 22:43:34 -0600
   Repository revision: 1.18 /var/lib/rancid/CVS/wcdc/configs/bn-002-0.nts.wustl.edu,v
   Commit Identifier: e6kEdcNzUdSp456x
   Sticky Tag: (none)
   Sticky Date: (none)
   Sticky Options: (none)


Run a update:

`cvs update`:

U bn-001-0.nts.wustl.edu
M bn-002-0.nts.wustl.edu


 Then run Rancid to see if this solved the problem:

`/var/lib/rancid/bin/rancid-run switches`


You will probably have to run it two times.   One to update and do a final check in.    One more run to make sure no more emails are generated.





Verify this fixed the problem:


`cvs status`:

===================================================================
File: bn-001-0.nts.wustl.edu    Status: Up-to-date
   Working revision:    1.39    2013-12-23 09:53:51 -0600
   Repository revision: 1.39    /var/lib/rancid/CVS/wcdc/configs/bn-001-0.nts.wustl.edu,v
   Commit Identifier:   boZWMtfaLtmxtsdx
   Sticky Tag:          (none)
   Sticky Date:         (none)
   Sticky Options:      -ko
===================================================================
File: bn-002-0.nts.wustl.edu    Status: Up-to-date
   Working revision:    1.19    2013-12-23 09:55:43 -0600
   Repository revision: 1.19    /var/lib/rancid/CVS/wcdc/configs/bn-002-0.nts.wustl.edu,v
   Commit Identifier:   maPVWYSoXBBu0hix
   Sticky Tag:          (none)
   Sticky Date:         (none)
   Sticky Options:      -ko




Monday, December 2, 2013

Using `strace` as a quick debugging tool.

The software SNAPP was throwing the error:

Failed to load dbi driver, please check db and/or dbi settings

In order to track down what was missing strace was used.  

By running the command with strace:

# strace snapp-collector -c /usr/local/etc/snapp_config.xml --foreground

The following was output:

execve("/usr/local/sbin/snapp-collector", ["snapp-collector", "-c", "/usr/local/etc/snapp_config.xml", "--foreground"], [/* 20 vars */]) = 0
brk(0)                                  = 0x1e5e000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f2c9ca12000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=63119, ...}) = 0
mmap(NULL, 63119, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f2c9ca02000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/usr/lib/libsnmp.so.15", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\27\1\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=620224, ...}) = 0
mmap(NULL, 2929152, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f2c9c526000
mprotect(0x7f2c9c5bb000, 2093056, PROT_NONE) = 0
.......
mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fca66878000
mprotect(0x7fca66878000, 4096, PROT_NONE) = 0
clone(child_stack=0x7fca668b7e30, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7fca668b89d0, tls=0x7fca668b8700, child_tidptr=0x7fca668b89d0) = 58461
openat(AT_FDCWD, "/usr/lib/dbd", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
futex(0x7fca6c229a68, FUTEX_WAKE_PRIVATE, 2147483647) = 0
stat("/usr/local/etc/snapp_config.xml", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
stat("/usr/local/etc/snapp_config.xml", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
stat("/usr/local/etc/snapp_config.xml", {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
open("/usr/local/etc/snapp_config.xml", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=210, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fca66877000
read(3, "SNAPP"..., 16384) = 210
read(3, "", 12288)                      = 0
lseek(3, 0, SEEK_CUR)                   = 210
close(3)                                = 0
munmap(0x7fca66877000, 4096)            = 0
write(2, "Failed to load dbi driver, pleas"..., 63Failed to load dbi driver, please check db and/or dbi settings
) = 63
tgkill(58447, 58461, SIGRTMIN)          = 0
tgkill(58447, 58449, SIGRTMIN)          = 0
tgkill(58447, 58450, SIGRTMIN)          = 0
tgkill(58447, 58451, SIGRTMIN)          = 0
tgkill(58447, 58452, SIGRTMIN)          = 0
.....


To find the error look around the output for the error: "Failed to load dbi driver....", then look above that line until something looks like the obvious problem.   In this example, we know that dbi failed to load.   Looking up a few lines we find the error:

openat(AT_FDCWD, "/usr/lib/dbd", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

Checking to see if this file exists:

ls -al /usr/lib/dbd
ls: cannot access /usr/lib/dbd: No such file or directory 

It does not.

Install the proper file:

apt-get install libdbd-mysql 

Run the program again and it works:

snapp-collector -c /usr/local/etc/snapp_config.xml --foreground
SNAPP-collector (version 3.0.7 (svn 9662)) started
collection_classes loaded=0, collections_loaded=0
config loaded
Snapp initialization complete 


Followers

Contributors