Thursday, August 18, 2016

How to properly enable NSF / Graceful-Restart in OSPF between IOS and NX-OS


OSPF routes get withdrawn on all neighbors (NXOS 7K) when a VSS supervisor (IOS 6807) switch over is initiated by any of the following methods: 
  1. Pulling active supervisor out
  2. redundancy force-switchover
  3. Rebooting chassis with active supervisory. 

Hardware / Software

Cisco 6807

  • 2x Cisco 6807 
  • 2x Sup6T (in each chassis)
  • Firmware: 15.3(1)SY
  • VSS

Nexus 7K

  • 3x Cisco Nexus 7009
  • 2x SUP1 (in each chassis)
  • Firmware: 6.2(10)

Topology Overview

Two Cisco 6807's in VSS mode are peering with three different NS-OX 7K's via routed point-to-point port-channels from both VSS members to a single 7K.  


By default NX-OS does not support nsf cisco mode.   The IOS device must be configured with nsf ietf.  See example below:

router ospf 1
 nsf ietf
 redistribute static subnets route-map STATIC_TO_OSPF
 passive-interface default
 no passive-interface Port-channel9
 no passive-interface Port-channel10
 no passive-interface Port-channel11
 no passive-interface Port-channel12
 no passive-interface Vlan899

Reference documents

Thursday, March 24, 2016

High CPU and Latency to VSS 4500X Control Plane

While troubleshooting a high latency issue on a Cisco 4500X we determined the problem to be a 30Mb/sec stream of UDP syslog traffic streaming into a host that was shutdown and therefor the ARP entry was removed:

4500x-switch#show ip arp vrf yellow-zone
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet           0   Incomplete      ARPA

We saw the CPU go high:

4500x-switch#show processes cpu sorted
Core 0: CPU utilization for five seconds: 99%; one minute: 87%;  five minutes: 49%
Core 1: CPU utilization for five seconds: 1%; one minute: 13%;  five minutes: 51%
PID    Runtime(ms) Invoked  uSecs  5Sec     1Min     5Min     TTY   Process
8609   1518439     20262550 392    50.56    50.40    50.32    0     iosd

Cisco says, when there is no ARP entry:

When CEF cannot locate a valid adjacency for a destination prefix, it punts the packets to the CPU for ARP resolution and, in turn, for completion of the adjacency. In rare cases, the adjacency persists in an incomplete state. For example, if the ARP table already lists a particular host, then punting it to the process level does not trigger an ARP.

This can be determined by looking for L3 Glean:

4500x-switch#show platform cpu packet statistics | inc Glean
L3 Glean                    2283852361     12913     14103     11978       8839
L3 Glean                     767303902      7732      8469      7606       6976

If the L3 Glean is high, the packets are getting punted to the CPU for processing.  As of 2016/03/24 we are checking to see if this is a bug.  This can be used as a DoS attack avenue.   

Sunday, November 29, 2015

Fun and open workplace culture.

Geared towards programming, but similar takeaways can be had for engineering culture in general.   Building open and fun technology.    Not just buying off the shelf shit that is no fun to build or work on.      I think this is why my department is so fun.   This is the kind of culture I would like to continue to foster.  

Coding is boring, unless

Thursday, October 29, 2015

Direct link for Android adb and fastboot (ie. platform-tools) for OSX

Here is the link straight from Google (no 3rd party sources) for Androids adb and fastboot:

You can't view the contents of this directory, if you want to make sure you have the latest keep increasing the number by one until you find the most recent.  

Wednesday, October 28, 2015

Create an Archive button just like Gmail in Outlook 2016 (and probably other versions) on OSX.

If you are like you me the 'archive' button in Gmail is a great way to fly though your email.   Microsoft Outlook for OSX lacks this native feature.

However, it can added it in just a few clicks.

This has been tested this on:

  • OSX 10.11.x (El Capitan)
  • Microsoft Outlook 2016

  1. Open Outlook
  2. Create an 'archive' folder if you don't already have one
  3. Select a message you want to archive
  4. Select Message -> Move -> Select Folder...
  5. Select the 'archive' folder and move the message
  6. Go to Message -> Move > Select Folder again...  It should look like this now (remember the exact name of the folder in this menu, you will need it later):
  7. Open System Preferences
  8. Select Keyboard -> Shortcuts -> App Shortcuts
  9. Click +
  10. Select Microsoft Outlook
  11. Select your folder in the "Menu Title" input box.   This MUST be the exact folder name you selected earlier (I told you to remember it).   It should look like this:
  12. Go back to Outlook
  13. Select Message -> Move
  14. WaaaLaaa - There is your new shortcut:
  15. In my example, press CTRL+A and the messages goes into the Archive (Wustl) folder.

Tuesday, October 27, 2015

OSX Desktop Setup for Network/Systems Engineers

I enjoy seeing how other people configure their desktops, so here is my setup...

These are the notes I would use if I were to start building a new laptop.  It is not all inclusive, but does cover the minimum things I like to have.

First thing to do, enable the firewall (System Preferences -> Security and Privacy):

Second, check for and configure updates (System Preferences -> App Store):

Third, enable full disk encryption (System Preferences -> Security and Privacy):

Swap around the Caps-Lock and CTRL key.   Why do this you ask?  I use the control key a lot.   Having to move my pinky from to the CTRL key requires me to either bend my finger in a way in which is not comfortable or to move my entire hand.  Switching the CTRL key where Caps-Lock is fixes this problem (System Preferences -> Keyboard -> Modifier Keys...):

Enable software installation from all sources.  There are many pieces of software I install from sources other then the App Store.  If you don't installed 3rd party software like this don't enable it. (System Preferences -> Security and Privacy):

Install iTerm2 as a replacement for the built in terminal program.   Why you ask?  The primary reason I use it is the "Paste Slowly" function.  If you use serial consoles this is a must.  There are many other great features such as saving window arrangements, split panes, profiles, broadcast input, column selection, etc...  Just get it, you won't be disappointed:

Install Xcode tools.   OSX 10.11.x has made this much easier.   Open a terminal and type 'make', it will detect the tools are not installed and ask if you would like to install them:

Install Homebrew the missing package manager for OSX:

Enable lots of desktops.   I sort my desktop layout in the following windows:
  1. Email
  2. Web Browsers
  3. Terminal Windows
  4. Communication (Slack, IRC, Instant Messenger)
  5. Note Taking / TODO List
  6. General Purpose (left open)
  7. Windows 10 in a VM

Configure CTRL+# to switch between desktops.  Remember why we switched the Caps Lock and CTRL key around?  This helps move between tasks quickly (System Preferences -> Keyboard -> Shortcuts):

Display the date on the task bar.  It is good to know what day it is (System Preferences -> Date and Time -> Clock):

Replace the command+<space> with Alfred2.  Why?  To send email, run terminal commands, execute applications, perform a search, plus a whole lot more:

Install Dropbox (personal files) and Box (work files).   I store 99% of my files in the cloud.   It is so convenient to get a new computer out of the box, install this software, and have all your files back in a few minutes (errr... hours) of sync time:

Install mosh.  What is mosh you ask?   A remote terminal application that allows roaming, supports intermittent connectivity, and provides intelligent local echo and line editing of user keystrokes.  Homebrew is a good way to install this software:

Install some type of VM software.   We use VMware Fusion at work.  While I use this software to test many different software packages, its primary use case is a Windows 10 desktop.  There are many pieces of software that we must use such as vSphere and Visio.   Additional thoughts after running Windows 10 for a while; It does NOT suck down the battery like it used to.  

Install Java.  Yes, another necessary evil.  Too many "enterprise" software packages require it.

Adjust the power settings in your Windows 10 to keep the VM from going to sleep:

To make it easier to connect to remote SSH hosts, I created what I call a SSH Function File.   Every unix, linux, switch, and router is in this file.  The file is stored on a cloud sync'ed file system with Box.  The following line is added to the .bashrc:

if [ -f ~/box/shared/ssh_functions ]; then
        source ~/box/shared/ssh_functions

The functions file (ssh_functions) looks like:

function host1() { ssh -t -A "ssh $" ;}
function host2() { ssh -t -A "ssh $" ;}
function host3() { ssh -t -A "ssh $" ;}
function host4() { ssh -t -A "ssh $" ;}
function host5() { ssh -t -A "ssh $" ;}
function jump_ssh() { ssh -t -A "ssh $@" ;}

To connect to a host:
$ host1

Or tab complete.   This is the primary reason we use "functions", we can tab complete our SSH connections.   For example, if you group all our devices by something like 'asa-' you can type 'asa-' hit the tab a few times and get the full list of all your ASA devices:
$ hos<tab>

This is an absolute minimum ~/.vimrc, I use many other plugins, but this is a great start:

filetype plugin indent on
syntax on
set modeline
set background=dark
set tabstop=4
set expandtab
set softtabstop=4
set shiftwidth=4

This is all.   There are probably many other little things I have left off.   I encourage you to post in the comments with suggestions and software you find useful in your day-to-day network/system engineering jobs.  

Sunday, March 1, 2015

Setting up a Cisco ASA 5505 on 9.x Code for Home Use with Charter Internet

This is an overview of how simple it is to setup a Cisco ASA 5505 for home use with Charter (although most any Internet provider should work).   The end goal is to have a system that will:
  1. Obtain an IP address from the cable modem by using DHCP
  2. PAT/NAT translate all inside addresses to the outside Charter address
  3. Act as a firewall protecting your inside network from the public Internet

The device we are using is the Cisco ASA 5505 running firmware: 9.1(5)21 with 1024Mb RAM, 128Mb flash, and a 500Mhz processor.

Cable up the device as follows:

  1. Connect port 1 to your cable modem, this is the outside network.  
  2. Connect port(s) 2 - 8 to your access points, computers, printers, etc.  This is the inside network. NOTE: ports 7 and 8 provide Power over Ethernet.   If your access points support this, you don't have to plug them into a wall outlet.   They can obtain power from these ports.   

Lets start with a blank canvas by erasing all existing configurations:

  1. Log into the device though the serial console.  Typically the password is blank, pressing enter will take you to the command prompt.   If password is lost, go through the password recovery here.
  2. Enter enable mode: enable (again, the password should be blank, press enter when prompted for a password).
  3. Clear the old configuration: write erase
  4. Reload the device to start with a fresh config: reload (do not save the config before reloading)

  1. Log into the device through the serial console.
  2. Enter enable mode: enable
  3. Enter configuration mode: configure terminal
  4. Use the Cisco factory default command to perform the basic setup: configure factory-default (this is the most important step, it does all the basic configuration)
  5. Setup user accounts, enable passwords, etc:
    1. Set a login password: passwd MySecretPassword
    2. Set an enable password: enable password MySecretPassword
    3. Setup a user account and password: username MyUserName password MySecretPassword
  6. (optional) enable remote SSH management on the local/inside network only:
    1. Turn on user authentication: aaa authentication ssh console LOCAL
    2. Enable SSH: ssh inside
  7. Save the configuration: write
That is it, the system should be working.  Any devices connected to ports 2-8 should be getting an IP address though DHCP in the range of -

Configuration Gotch-Ya: On my cable modem only 1 MAC address can be configured at a time.   I had to clone the MAC address from my former router on the ASA before I was able to obtain a DHCP address from the Charter modem.  Configuration example:
interface Vlan2
  mac-address 3408.0408.1234

Important Note: As with any device connected to the public Internet, the latest vendor recommended software should always be installed.   Do this now!   Instructions can be found here.