Wednesday, January 22, 2014

Using CACTI to Graph >60,000 Interface Statistics at 1 Minute Polling Interval

Here are some statistics and information from my CACTI server that is capable of polling > 60,000 interface statistics at a 1 minute polling interval.

The server is a HP ProLiant DL380p Gen8 with 32Gb/ram; 32 core Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz; 2x146G 15K raid1 disks; 4x600G 10K raid5 disks; 2x100G SSD disks.

The OS is a stock Ubuntu 12.04LTS installation.

CACTI is installed from Micah Gersten's 12.04 backports PPA.   The 8.8a release was need to get support for the Cacti Plugin Architecture.

CACTI plugins installed include: settings and thold

CACTI poller is spine with 20 threads.

There are >500 devices added into Cacti.   In order to add all servers and all the interfaces associated with them I wrote this script:

There are >60,000 RRA files all stored in /var/lib/cacti/rra, using up > 45G of space.   In order to test disk performance, these databases have been moved to all 3 different types of raid arrays.

First was the 2x146G/raid1 set:

When the RRA's were on this raid array the average utilization was around 20 - 70%.   The average polling cycle for Cacti took 15 - 30 seconds to complete all interfaces.  

Second was the 4x600G/raid5 set:

When the RRA's were on this raid array the disk utilization was pegged at 100% and the system was in I/O wait the entire time.   The average Cacti polling cycle took 30 - 45 seconds to complete all interface and the graphs started missing data.  

Finally was the 2x100G raid1 SSD set:

When the RRA's moved to this disk the average utilization is 1%.   The average Cacti polling cycle took 3 to 4 seconds to complete.   

SSD's make a significant improvement in performance.   

Load average also varied based on what raid array the RRA's were on.   (ie. day 16-17 2x146; day 17 - 21 4x600; day 21-22 2x100G SSD):

There are other plugins like boost that are supposed to help performance quite a bit.   At this point, when the RRA's are on solid state disks, there is no need yet.   

To conclude, CACTI is very capable of polling a large of number interfaces without a problem assuming the proper hardware is used.   

No comments:

Post a Comment