Thursday, November 20, 2014

Bring up Google Now with bluetooth button (Jabra Motion + LG G2)

To make Google Now the default action when long pressing the Jabra bluetooth button when paired with an LG G2, follow these steps:



1) Disable Voice Record by going to Settings -> General -> Apps -> All -> Voice Recorder.  Then click Disable:




2) Download the Bluetooth Launch app from Google Play.

3) Start Bluetooth Launch.  Expand Google Search, then select com.google.android.googlequicksearchbox.VoiceSearchActivity: 


4) Long click the button on the bottom of the mic and Google Now will be the default action for all voice commands.   





Notes: Google Now is a MUCH better system for voice commands it actually works a majority of the time.   The stock actions were terrible.   

Friday, November 7, 2014

IPMI Monitoring of Dell Server Hardware with Nagios and iDRAC.

If you are looking for a dead simple way to monitor ALL hardware on Dell servers look no further than IPMI on the iDRAC cards.

The basic steps to get Nagios polling the iDRAC card for hardware sensors:
  • Boot the server with a keyboard and monitor attached.
  • Enter setup by pressing F2 in the BIOS POST.
  • Select the iDrac configuration.
  • Configure basic networking to get the iDRAC card online.

Once the iDrac card is online, go to the web interface by entering the URL:
http://your-idrac.example.com/


Under the network settings configure the IPMI settings:



In the User Configuration create a new user with IPMI User privileges:



Nagios setup:

In this example the Nagios server is running Ubuntu server 14.04.x LTS.   I will assume Nagios core and plugins are already installed.

You can try the stock check_ipmi_sensors script, but I could not get it to work.  It always failed with the following error:

jemurray@nagios:~$ /usr/lib/nagios/plugins/check_ipmi_sensor -H 192.168.28.12 -U lom -P lom-password -L user
ipmi_ctx_open_outofband: session timeout
-> Execution of ipmimonitoring failed with return code 1.
-> ipmimonitoring was executed with the following parameters:
   /usr/sbin/ipmi-sensors -h 192.168.28.12 -u lom -p lom-password -l user --quiet-cache --sdr-cache-recreate --interpret-oem-data --output-sensor-state --ignore-not-available-sensors

When I downloaded the lastest plugin from: http://www.thomas-krenn.com/en/wiki/IPMI_Sensor_Monitoring_Plugin - everything worked just fine:

jemurray@selleck:~/check_ipmi_sensor_v3-96ed86b$ ./check_ipmi_sensor -H 192.168.28.12 -U lom -P lom-password -L user -x 55 -x 84 -x 86 -x 87
IPMI Status: OK | 'Fan1 RPM'=2400.00 'Fan2 RPM'=2400.00 'Fan3 RPM'=2280.00 'Fan4 RPM'=2280.00 'Fan5 RPM'=2160.00 'Fan6 RPM'=2160.00 'Inlet Temp'=22.00 'Exhaust Temp'=31.00 'Temp'=35.00 'Temp'=39.00 'Current 1'=0.60 'Current 2'=0.20 'Voltage 1'=208.00 'Voltage 2'=208.00 'Pwr Consumption'=112.00 

Copy the newly downloaded plugin to /usr/local/bin.


Create the following Nagios plugin command.   They key thing to note here is the _HOSTIPMI_IP macro that is defined in the command and then in the host definition:

# Example Syntax: ./check_ipmi_sensor -H 192.168.0.120 -U lom -P lom-nagios-password -L user -x 32 -x 52 -x 82
define command{
        command_name    check_ipmi_sensor
        command_line    /usr/local/bin/check_ipmi_sensor -H $_HOSTIPMI_IP$ -U $ARG1$ -P $ARG2$ -L user $ARG3$
}


Create the following Nagios host and service definitions:

define host {
    host_name                          statseeker.example.com
    alias                              statseeker.example.com
    address                            192.168.28.49
    _ipmi_ip                           192.168.28.12
    parents                            dc-eps-0.example.com
    hostgroups                         server
    notifications_enabled              1
    event_handler_enabled              1
    flap_detection_enabled             1
    failure_prediction_enabled         1
    process_perf_data                  1
    retain_status_information          1
    retain_nonstatus_information       1
    check_command                      check-host-alive
    max_check_attempts                 10
    notification_interval              0
    notification_period                24x7
    notification_options               d,u,r
    contact_groups                     systems
    icon_image                         base/linux40.png
    statusmap_image                    base/linux40.png
    icon_image_alt                     Server
}
define service {
    service_description             IPMI-statseeker.example.com
    host_name                       statseeker.example.com
    check_command                   check_ipmi_sensor!lom!my-lom-password!-x 55 -x 84 -x 86 -x 87
    notification_interval           0
    active_checks_enabled           1
    passive_checks_enabled          1
    parallelize_check               1
    obsess_over_service             1
    check_freshness                 0
    notifications_enabled           1
    event_handler_enabled           1
    flap_detection_enabled          1
    failure_prediction_enabled      1
    process_perf_data               1
    retain_status_information       1
    retain_nonstatus_information    1
    is_volatile                     0
    check_period                    24x7
    normal_check_interval           5
    retry_check_interval            1
    max_check_attempts              4
    notification_period             24x7
    notification_options            w,u,c,r
    contact_groups                  systems
}

Tuesday, November 4, 2014

Yum failure (again).

Once again, Redhat/CentOS has problems with yum corruption:

[root@unixhosts /home/jemurray]# yum update
rpmdb: Thread/process 24996/140588668987136 failed: Thread died in Berkeley DB library
error: db3 error(-30974) from dbenv->failchk: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db3 -  (-30974)
error: cannot open Packages database in /var/lib/rpm
CRITICAL:yum.main:
Error: rpmdb open failed


I have been running Debian and Redhat servers for a very long time.   I have also used auto security updates for a long time.

Debian has never failed me.   Redhat fails more times than I want to count.    For services as critical as automatic updates this can't fail.

There are times I must use Redhat servers for a handful of enterprise applications.   So, here is the fix:

rm -f /var/lib/rpm/__db*
rpm --rebuilddb
yum clean all
yum update

Friday, July 18, 2014

Puppet module to manage users, groups, and ssh keys (updated).

This is an updated version of my old post.  There were some errors, confusion, and typos.

Everything has been moved to a GitHub repository with an updated README:

    https://github.com/duxklr/manageusers


The goal of this setup was to create a very easy module that is capable of adding and removing user accounts as well as managing the groups and ssh keys.

Thursday, July 17, 2014

Example of HTTP POST and PUT using RESTful interface with JSON data being sent and returned.

Here is what is sent over the wire when you HTTP POST or PUT JSON data via a RESTful (like) API.   The example has been mocked up using Python and Flask.  The goal of this exercise was to show how the data is transmitted as well as all the HTTP headers in use for the transaction.   




Command to initiate the POST:
curl -i -H "Content-Type: application/json" -X POST -d '{"variable1":"data1", "variable2":"data2"}' http://localhost:5000/api/2

Packet capture of POST using Wireshark to follow the TCP stream:
POST /api/2 HTTP/1.1
User-Agent: curl/7.30.0
Host: localhost:5000
Accept: */*
Content-Type: application/json
Content-Length: 42
{"variable1":"data1", "variable2":"data2"}

Packet capture of data returned from POST:
HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 96
Server: Werkzeug/0.9.6 Python/2.7.5
Date: Fri, 18 Jul 2014 01:10:51 GMT
{
  "post_id": 2,
  "variable1": "data1",
  "variable2": "data2"
}




Command to initiate the PUT:
curl -i -H "Content-Type: application/json" -X PUT -d '{"variable3":"data3", "variable4":"data4"}' http://localhost:5000/api/3

Packet capture of PUT using Wireshark to follow the TCP stream:
PUT /api/3 HTTP/1.1
User-Agent: curl/7.30.0
Host: localhost:5000
Accept: */*
Content-Type: application/json
Content-Length: 42
{"variable3":"data3", "variable4":"data4"}

Packet capture of data returned from PUT:
HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 68
Server: Werkzeug/0.9.6 Python/2.7.5
Date: Fri, 18 Jul 2014 01:17:39 GMT
{
  "post_id": 3,
  "variable3": "data3",
  "variable4": "data4"
}

You don't have to use CURL to send the data.  Here is an example of using telnet instead.  After you connect, paste in the data starting at the PUT and ending at the {...}:

jemurray@dsg:~ $ telnet localhost 5000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
PUT /api/3 HTTP/1.1
User-Agent: telnet/0.1
Host: localhost:5000
Accept: */*
Content-Type: application/json
Content-Length: 42
{"variable3":"data3", "variable4":"data4"} 


Wednesday, May 14, 2014

X11 forwarding on RHEL 6

In order to get X11 forwarding working properly on a RHEL 6 server make sure the following options are installed and configured:


  • Install the following package - xorg-x11-xauth.x86_64:
sudo yum install xorg-x11-xauth


  • Make sure sshd has X11 forwarding enabled by editing /etc/ssh/sshd_config:
X11Forwarding yes


  • Log into the server with the X11 Forwarding option:
ssh -X server.example.com


  • Verify X11 is working after you log in:

[jemurray@server01 ~]$ echo $DISPLAY

localhost:10.0








Tuesday, March 25, 2014

Twenty years in this field and never knew awk was so great...

I needed to find lines that look like this:

Mar 25 06:38:58 192.268.29.141 syslog_smtp: Info: New SMTP ICID 447042464 interface public (192.268.29.141) address 157.56.110.254 reverse dns host mail-bn1un0254.outbound.protection.outlook.com verified yes
Mar 25 06:38:58 192.268.29.141 syslog_smtp: Info: ICID 447042464 REJECT SG BLACKLIST match sbrs[-10.0:-3.0] SBRS -6.5
Basically I need to match anything that belongs to outbound.protection.outlook.com and grab the ICID number.   Then continue to look though the log file to find another match of the ICID and BLACKLIST.   If these are found then print both lines.


You can accomplish this by this awk script:


/outbound.protection.outlook.com/ {
  icid = $10
  lastline = $0
}
($8 ~ icid && $11 ~ /BLACKLIST/ ) {
  printf "First line: %s\n",  lastline
  printf "Next line: %s\n", $0
  print ""
}

Running it yields:

jemurray@syslog:~$ cat /var/log/ironport.log.1 | awk -f ./blacklist.awk

First line: Mar 24 06:42:29 192.168.29.141 syslog_smtp: Info: New SMTP ICID 446816558 interface public (192.168.29.141) address 207.46.100.250 reverse dns host mail-by2hn0250.outbound.protection.outlook.com verified yes
Next line: Mar 24 06:42:29 192.168.29.141 syslog_smtp: Info: ICID 446816558 REJECT SG BLACKLIST match sbrs[-10.0:-3.0] SBRS -6.7
First line: Mar 24 06:42:29 192.168.29.10 syslog_smtp: Info: New SMTP ICID 462989143 interface public (192.168.29.10) address 207.46.100.250 reverse dns host mail-by2hn0250.outbound.protection.outlook.com verified yes
Next line: Mar 24 06:42:29 192.168.29.10 syslog_smtp: Info: ICID 462989143 REJECT SG BLACKLIST match sbrs[-10.0:-3.0] SBRS -6.7
First line: Mar 24 06:47:30 192.168.29.141 syslog_smtp: Info: New SMTP ICID 446817160 interface public (192.168.29.141) address 207.46.100.246 reverse dns host mail-by2hn0246.outbound.protection.outlook.com verified yes
Next line: Mar 24 06:47:30 192.168.29.141 syslog_smtp: Info: ICID 446817160 REJECT SG BLACKLIST match sbrs[-10.0:-3.0] SBRS -6.7
First line: Mar 24 06:47:31 192.168.29.10 syslog_smtp: Info: New SMTP ICID 462989826 interface public (192.168.29.10) address 207.46.100.247 reverse dns host mail-by2hn0247.outbound.protection.outlook.com verified yes
Next line: Mar 24 06:47:31 192.168.29.10 syslog_smtp: Info: ICID 462989826 REJECT SG BLACKLIST match sbrs[-10.0:-3.0] SBRS -6.5
First line: Mar 24 06:52:10 192.168.29.141 syslog_smtp: Info: New SMTP ICID 446817705 interface public (192.168.29.141) address 157.56.112.247 reverse dns host mail-am1hn0247.outbound.protection.outlook.com verified yes
Next line: Mar 24 06:52:10 192.168.29.141 syslog_smtp: Info: ICID 446817705 REJECT SG BLACKLIST match sbrs[-10.0:-3.0] SBRS -8.1
First line: Mar 24 06:52:10 192.168.29.10 syslog_smtp: Info: New SMTP ICID 462990429 interface public (192.168.29.10) address 157.56.112.247 reverse dns host mail-am1hn0247.outbound.protection.outlook.com verified yes
Next line: Mar 24 06:52:10 192.168.29.10 syslog_smtp: Info: ICID 462990429 REJECT SG BLACKLIST match sbrs[-10.0:-3.0] SBRS -8.1 


In the past I have always used a higher level language like Perl or Python to accomplish this.

Tuesday, March 18, 2014

Compiling and installing (almost) all Nagios Core plugins on Centos6.x with EPEL.

Typically I try to use vendor supplied packages when possible.   In this case I need the features of Nagios 4.x.   As of 2014/03 the only way to get this software was to compile Nagios Core and Nagios Plugins by hand.  Before installing the plugins, make sure Nagios is installed by following the installation directions (outside the scope of this document).

The Nagios Plugins 2.0 plugins have numerous package dependencies.  Many of these can be resolved by installing the EPEL repository and using yum to install them.   The hardest part of this project was finding what packages resolve these dependences.

Install the basic compiler tools:
sudo yum groupinstall "Development tools"

Install the EPEL repository:
sudo rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm

Install the following extra packages.  There will be LOTS of dependencies that need to be satisfied. Installing the following should cover everything:
sudo yum install openssl-devel gnutls-devel fping qstat openldap-devel mysql-devel postgresql-devel libdbi-devel radiusclient-ng-devel perl-Net-SNMP libsmbclient-devel samba-client net-snmp-devel 
Download the software:
wget http://nagios-plugins.org/download/nagios-plugins-2.0.tar.gz

Enter the dir:
cd nagios-plugins-2.0 

Run configure:
./configure --with-nagios-user=nagios --with-nagios-group=nagios

Watch the output of the ./configure command.   Look for any plugins that don't compile.  You may see something like this:
...
configure: WARNING: Get smbclient from Samba.org to monitor SMB shares
...
configure: WARNING: Get fping from http://www.fping.com in order to make check_fping plugin
...
configure: WARNING: Tried /usr/bin/perl - install Net::SNMP perl module if you want to use the perl snmp plugins
... 
While I believe the above list of packages is comprehensive, I may have missed something.   Based on the errors, it is pretty easy to figure out what package that needs to be installed.   For example, the fping warning above could be solved by running:
yum search fping

Find the package and install it:
sudo yum install fping

Once configure is completed, install the plugins:
sudo make install

That is it.  There should be about 72 plugins:
ls -1 /usr/local/nagios/libexec/ | wc -l
72 

I know the following plugins didn't get built.   You have to go to external sources to satisfy their requirements.  I didn't feel like doing that:

  • flexlm license manager
  • Qmail qstat

Wednesday, January 22, 2014

Using CACTI to Graph >60,000 Interface Statistics at 1 Minute Polling Interval

Here are some statistics and information from my CACTI server that is capable of polling > 60,000 interface statistics at a 1 minute polling interval.

The server is a HP ProLiant DL380p Gen8 with 32Gb/ram; 32 core Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz; 2x146G 15K raid1 disks; 4x600G 10K raid5 disks; 2x100G SSD disks.

The OS is a stock Ubuntu 12.04LTS installation.

CACTI is installed from Micah Gersten's 12.04 backports PPA.   The 8.8a release was need to get support for the Cacti Plugin Architecture.

CACTI plugins installed include: settings and thold

CACTI poller is spine with 20 threads.

There are >500 devices added into Cacti.   In order to add all servers and all the interfaces associated with them I wrote this script:



There are >60,000 RRA files all stored in /var/lib/cacti/rra, using up > 45G of space.   In order to test disk performance, these databases have been moved to all 3 different types of raid arrays.

First was the 2x146G/raid1 set:


When the RRA's were on this raid array the average utilization was around 20 - 70%.   The average polling cycle for Cacti took 15 - 30 seconds to complete all interfaces.  


Second was the 4x600G/raid5 set:


When the RRA's were on this raid array the disk utilization was pegged at 100% and the system was in I/O wait the entire time.   The average Cacti polling cycle took 30 - 45 seconds to complete all interface and the graphs started missing data.  


Finally was the 2x100G raid1 SSD set:


When the RRA's moved to this disk the average utilization is 1%.   The average Cacti polling cycle took 3 to 4 seconds to complete.   

SSD's make a significant improvement in performance.   


Load average also varied based on what raid array the RRA's were on.   (ie. day 16-17 2x146; day 17 - 21 4x600; day 21-22 2x100G SSD):





There are other plugins like boost that are supposed to help performance quite a bit.   At this point, when the RRA's are on solid state disks, there is no need yet.   


To conclude, CACTI is very capable of polling a large of number interfaces without a problem assuming the proper hardware is used.   


Monday, January 13, 2014

Public NTP server DoS mitigation

In recent weeks there has been a uptick in using NTP as a DoS amplification attack mechanism:

http://www.kb.cert.org/vuls/id/348126

https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2013-5211

https://www.us-cert.gov/ncas/current-activity/2014/01/10/Network-Time-Protocol-NTP-Amplification-Attacks

https://cert.litnet.lt/en/docs/ntp-distributed-reflection-dos-attacks

https://isc.sans.edu/forums/diary/NTP+reflection+attack/17300




If there is no need for the system to be a public NTP server, then configure the it to act as a client only.  Good documentation can be found here:

https://www.team-cymru.org/ReadingRoom/Templates/secure-ntp-template.html




To test if the server is vulnerable you can run:
ntpdc -n -c monlist unixhosts.us


If you receive any output, such has this your server is vulnerable:
remote address          port local address      count m ver rstr avgint  lstint
===============================================================================
2604:3e00:2:2::5       55949 2600:3c02::f03c:91ff:fe93:8906       11 7 2    590     28       0
172.56.29.211          44636 173.230.136.108        1 3 3    590      0       1
172.56.28.4            40338 173.230.136.108        1 3 3    590      0       1
208.54.90.242          10481 173.230.136.108        2 3 3    590      9       1
208.54.90.218          38450 173.230.136.108        1 3 3    590      0       2
208.54.90.249          22476 173.230.136.108        1 3 3    590      0       2
172.56.29.189          59132 173.230.136.108        1 3 3    590      0       2
172.56.29.237          44774 173.230.136.108        1 3 3    590      0       4
172.56.28.175          38340 173.230.136.108        1 3 3    590      0       5
172.56.28.10           21008 173.230.136.108        1 3 3    590      0       5
23.23.43.31              123 173.230.136.108       10 3 4    590   1065       6
208.54.90.187          37950 173.230.136.108        1 3 3    590      0       7
172.56.29.28           23414 173.230.136.108        2 3 3    590     26       7
95.94.128.234          53738 173.230.136.108        4 3 4    590      0       8


To disable the DoS mechanism add the following lines to the /etc/ntp.conf file:
restrict -4 default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery

Restart ntpd (it may take a little while for the ntp server to start responding after the restart).

Test to make sure the ntp server is working:
jemurray@janus:~$ sudo ntpdate unixhosts.us
13 Jan 14:40:05 ntpdate[30883]: adjust time server 173.230.136.108 offset 0.001280 sec 

Then test to make sure it can no longer be used as an amplification attack tool:
jemurray@janus:~$ ntpdc -n -c monlist unixhosts.us
unixhosts.us: timed out, nothing received
***Request timed out



Notes: Tested on Ubuntu 10.04.4 LTS running stock ntpd: 1:4.2.4p8+dfsg-1ubuntu2.1 - Network Time Protocol daemon and utility pro


Followers

Contributors