Using CACTI to Graph >60,000 Interface Statistics at 1 Minute Polling Interval

Here are some statistics and information from my CACTI server that is capable of polling > 60,000 interface statistics at a 1 minute polling interval.

The server is a HP ProLiant DL380p Gen8 with 32Gb/ram; 32 core Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz; 2x146G 15K raid1 disks; 4x600G 10K raid5 disks; 2x100G SSD disks.

The OS is a stock Ubuntu 12.04LTS installation.

CACTI is installed from Micah Gersten's 12.04 backports PPA.   The 8.8a release was need to get support for the Cacti Plugin Architecture.

CACTI plugins installed include: settings and thold

CACTI poller is spine with 20 threads.

There are >500 devices added into Cacti.   In order to add all servers and all the interfaces associated with them I wrote this script:



There are >60,000 RRA files all stored in /var/lib/cacti/rra, using up > 45G of space.   In order to test disk performance, these databases have been moved to all 3 different types of raid arrays.

First was the 2x146G/raid1 set:

When the RRA's were on this raid array the average utilization was around 20 - 70%.   The average polling cycle for Cacti took 15 - 30 seconds to complete all interfaces.  

Second was the 4x600G/raid5 set:

When the RRA's were on this raid array the disk utilization was pegged at 100% and the system was in I/O wait the entire time.   The average Cacti polling cycle took 30 - 45 seconds to complete all interface and the graphs started missing data.  

Finally was the 2x100G raid1 SSD set:

When the RRA's moved to this disk the average utilization is 1%.   The average Cacti polling cycle took 3 to 4 seconds to complete.   

SSD's make a significant improvement in performance.   

Load average also varied based on what raid array the RRA's were on.   (ie. day 16-17 2x146; day 17 - 21 4x600; day 21-22 2x100G SSD):

There are other plugins like boost that are supposed to help performance quite a bit.   At this point, when the RRA's are on solid state disks, there is no need yet.   

To conclude, CACTI is very capable of polling a large of number interfaces without a problem assuming the proper hardware is used.   

Public NTP server DoS mitigation

In recent weeks there has been a uptick in using NTP as a DoS amplification attack mechanism:

http://www.kb.cert.org/vuls/id/348126

https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2013-5211

https://www.us-cert.gov/ncas/current-activity/2014/01/10/Network-Time-Protocol-NTP-Amplification-Attacks

https://cert.litnet.lt/en/docs/ntp-distributed-reflection-dos-attacks

https://isc.sans.edu/forums/diary/NTP+reflection+attack/17300




If there is no need for the system to be a public NTP server, then configure the it to act as a client only.  Good documentation can be found here:

https://www.team-cymru.org/ReadingRoom/Templates/secure-ntp-template.html




To test if the server is vulnerable you can run:

ntpdc -n -c monlist unixhosts.us

If you receive any output, such has this your server is vulnerable:

remote address          port local address      count m ver rstr avgint  lstint
===============================================================================
2604:3e00:2:2::5       55949 2600:3c02::f03c:91ff:fe93:8906       11 7 2    590     28       0
172.56.29.211          44636 173.230.136.108        1 3 3    590      0       1
172.56.28.4            40338 173.230.136.108        1 3 3    590      0       1
208.54.90.242          10481 173.230.136.108        2 3 3    590      9       1
208.54.90.218          38450 173.230.136.108        1 3 3    590      0       2
208.54.90.249          22476 173.230.136.108        1 3 3    590      0       2
172.56.29.189          59132 173.230.136.108        1 3 3    590      0       2
172.56.29.237          44774 173.230.136.108        1 3 3    590      0       4
172.56.28.175          38340 173.230.136.108        1 3 3    590      0       5
172.56.28.10           21008 173.230.136.108        1 3 3    590      0       5
23.23.43.31              123 173.230.136.108       10 3 4    590   1065       6
208.54.90.187          37950 173.230.136.108        1 3 3    590      0       7
172.56.29.28           23414 173.230.136.108        2 3 3    590     26       7
95.94.128.234          53738 173.230.136.108        4 3 4    590      0       8

To disable the DoS mechanism add the following lines to the /etc/ntp.conf file:

restrict -4 default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery

Restart ntpd (it may take a little while for the ntp server to start responding after the restart).

Test to make sure the ntp server is working:

jemurray@janus:~$ sudo ntpdate unixhosts.us
13 Jan 14:40:05 ntpdate[30883]: adjust time server 173.230.136.108 offset 0.001280 sec 

Then test to make sure it can no longer be used as an amplification attack tool:

jemurray@janus:~$ ntpdc -n -c monlist unixhosts.us
unixhosts.us: timed out, nothing received
***Request timed out

Notes: Tested on Ubuntu 10.04.4 LTS running stock ntpd: 1:4.2.4p8+dfsg-1ubuntu2.1 - Network Time Protocol daemon and utility pro