Monitoring a connection via IP Sla, Tracking & Syslog

It's been nearly a week before I managed to learn something new at work.  It encompasses a few bits though!  Here goes:

We have 2 connections at branch sites.  One connection heads back to the core network and the other is a connection to our organisations version of the Internet.  This Internet connection is managed by a third party and we have to log calls to prompt any action if it is suspected it is at fault - we have no access to the router.

Recently it has been at fault, its just proving it!

We statically route a long list of addresses out of the Internet connection on Cisco 1841 routers, rather than them being routed internally.  Its more efficient to use the connection back to the hub site for internal data and the internet connection is funded centrally, so why not use it for itnernet traffic?

The problem seems to be that every 'now and then' user computers will 'hang' for a period of a few minutes, or substantially longer when they are accessing services through this connection.  Our desktop engineers have ruled out the PC's - and I was called in.  The syptoms seem familiar to a home user accessing a web-page with some inbuilt java or scripting, then their connection dies and thus the PC sits there with an egg-timer until the router/connection is restored and off they go again. 

The router and switches that we can manage look fine.  No error messages of any sort and the interface counters seem relatively solid. 

As we use static routing defined like so...:

ip route 0.0.0.0 0.0.0.0 192.168.242.22
ip route 20.130.152.0 255.255.255.0 10.128.99.32

... The defined route will only check its interface and line status to the device known as 10.128.99.32 - which is the 3rd party router connected via a crossover cable next to it.  If the crossover breaks, or both interfaces go down, then the route will be changed to the default route listed above.  Its clear that this isn't happening, the interface counters and syslog don't show it to have dropped and internet traffic hasn't been routed to our core firewalls, so the issue must be on the wider 3rd party connection.  but how to prove that...

We have IOS 15.1(3)T - so I can't vouch for older IOS revisions in terms of the commands used here, just for info.

Find an I.P
First I found a 'pingable' I.P address of one of the systems accessed over the connection.  Its suprising how many of the services we use wouldn't respond, but luckily the most important system allowed ICMP echo's.  You may question how pinging one I.P address will confirm the connection is down - and I'd agree with you that it doesn't.  But... we do access this system from 100+ other sites and it has an up time of 99.9% so, in my book, its as solid an address as I can find.

Configure IP SLA
Configuring the IP SLA manually will allow you to set definable thresholds, frequency and timeout values.  But I did actually stumble across Solar Winds IP SLA monitor.  Its free and therefore its not a polished product that can be tweaked to the endth degree, but it works ok for the task.  Ensure you have write snmp details for the device and that you know the string, otherwise you'll get zero results!
After configuring the SLA through the application, I did have to amend the configuration in the CLI, just to tell the SLA that it should use the interface connected to the internet connection, to do the pinging!
It ended up looking like this:

ip sla 1
 icmp-echo 20.130.152.151 source-interface FastEthernet0/1
 owner SW_IpSla_FreeTool_test
 frequency 30
ip sla schedule 1 life forever start-time now
I like how the application has given itself an owner tag.  In any case, the little desktop application shows a green light - so the SLA is a success at the minute.  Also running show ip sla statistics, proves that to be the case.  The frequency field is defined by the application (boo) and amending it doesn't allow the SLA to be monitored through the application (frequency is how often the ping occurs, in seconds).

Configure a Track
I then setup a track on this object so that we could assess its reachability state and base some actions on it.  The track has to be the easiest series of commands:

track 1 ip sla 1

Show track, gives the state, changes and time of last change etc.

Configuring Event Manager
Rather than looking through the router logs after logging in, we'd like events to be referenced in Syslog so we can view them through our network management software.

Event manager is the way to do this.  You can set actions (such as log to syslog with a message) based on the state changes of our Tracked object.  In turn, we can build a log of how often the internet connection appears to drop.  We can compare this log with another site router (performing all the same tricks as this one) to approach the 3rd party with something comparable - doing their job for them ideally.

Here are the commands to configure event manager:

event manager applet internet-down
 event tag pingdown track 1 state down
 trigger
  correlate event pingdown
  attribute tag pingdown occurs 1
 action 1 syslog msg "*********Internet DOWN**********"

event manager applet internet-up
 event tag pingup track 1 state up
 trigger
  correlate event pingup
 action 1 syslog msg "*********Internet UP**********"
We have two actions based on the 'state' of the tracked object.  One logs the internet as down and the other as being up.  Its important to log when it comes back up, so we know how long the outage was for.
You may also notice a couple of irrelevant commands - such as the trigger and correlate events commands.  Originally during testing I created two event tags for each applet.  This allowed two IP addresses to be pinged (I had two IP SLA's and two Tracks) and the correlate would take both into account - its just a slightly more accurate way to say the connection was down/up basing it on more than one I.P address.  I subsequently removed the IP SLA 2 and Track 2, but left the trigger and correlate there for you to see what is possible with event manager.

And there we have it.  A free desktop application keeps an eye on our IP Sla's as well as logged data recorded from the router.  If we have to go a step further in the future, it would be to tweak the SLA to suit our parameters and then base the routing decisions via PBR (policy based routing) on the reachability of some service I.P addresses (or other SLA probes). 

But for the time being, management just want a log of how often the problem occurs, so in a month or so, I hope we'll have some results - or not... the poor users will be the ones affected if I do have something to show!

High CPU on Cisco 1841 Router - ARP or SNMP?

And so the learning continues today...

Another network manager has reported an issue with some legacy routers.  The CPU is over 80% and the SNMP engine seems to be partly responsible.  The manager has turned off the SNMP engine to prevent the router from melting - but unsure why the problem has occurred.

Another network administrator has already found some information that may help.  He discovered that the ARP tables on the 'busy' routers were large.  In fact they were extremely large for a remote site/branch router.  50,000 entries on one of them, is some serious amount of Arp requests.

Delving further into the problem it appears that the router is proxy-arp'ing everything.  Running a show arp summary (on slightly new IOS's that some of ours have) give some indication as to which interfaces the Arp'ing was being conducted.  In this case it was a Vlan (the router was using an etherswitch card) that connected the remote site to its main/hub site.  It was also the route for traffic that didn't have a specific route in the routing table - namely a default route (ip route 0.0.0.0 0.0.0.0 vlan128).  The arp table showed requests to sites on the internet as well as internal ranges.

The Cisco documentation regarding troubleshooting high CPU utilization does cover this aspect as highlighted here:

************************************

ARP Input

High CPU utilization in the Address Resolution Protocol (ARP) Input process occurs if the router has to originate an excessive number of ARP requests. The router uses ARP for all hosts, not just those on the local subnet, and ARP requests are sent out as broadcasts, which causes more CPU utilization on every host in the network. ARP requests for the same IP address are rate-limited to one request every two seconds, so an excessive number of ARP requests would have to originate for different IP addresses. This can happen if an IP route has been configured pointing to a broadcast interface. A most obvious example is a default route such as:

ip route 0.0.0.0 0.0.0.0 Fastethernet0/0

In this case, the router generates an ARP request for each IP address that is not reachable through more specific routes, which practically means that the router generates an ARP request for almost every address on the Internet. For more information about configuring next hop address for static routing, see Specifying a Next Hop IP Address for Static Routes.

Alternatively, an excessive amount of ARP requests can be caused by a malicious traffic stream which scans through locally attached subnets. An indication of such a stream would be the presence of a very high number of incomplete ARP entries in the ARP table. Since incoming IP packets that would trigger ARP requests would have to be processed, troubleshooting this problem would essentially be the same as troubleshooting high CPU utilization in the IP Input process.

************************************

So it appears that specifying the next hop as an interface may not be good practice.  Luckily we already have some knowledge of this and have implemented IP addresses of remote router interfaces at most sites (usually wise to base next hop on a reachable IP in-case you have more than one route to that I.P rather basing it on an interface state), but obviously some legacy configs are still around.

It must be that the SNMP engine was then processing all this ARP data and thus adding to the CPU load for an already busy router processing arp requests!

By changing the next hop interface to an I.P address, the router calmed and normal service was resumed - although there is still some monitoring to be done to see if this is a long term cure.  One suspects that turning proxy-arp off, is not only a good security lesson, but may help the router out in future.