Showing posts with label pbr. Show all posts
Showing posts with label pbr. Show all posts
Friday, 27 January 2012
IP GRE Tunnel + PBR (Policy Based Routing)
A little project that has taken up a few days at work recently was: How to route traffic on a remote network out of an internet connection on a local network? The closest idea that could be invented within the department was to turn off the remote networks internet connection (it is 1/10th the capacity of the local one in question) and let the routing tables take over.
I thought there was a better way and suggested PBR as the answer. This lead to another network adminstrator and myself drawing up how it would work. Cue whiteboard and a bit of head scratching. We only wanted certain traffic to NOT use the remote internet connection - even though it was slower than the local one, it was still perfectly servicable.
I might use local and remote in the wrong context here, but the LOCAL internet connection is the one I'm sat next to and the REMOTE network is at the other side of the city - just for reference!
PBR
The idea to use PBR is to detect traffic on the remote network (matching them by destination and source) and policy routing them to the local internet router. Whilst this sounds a solid enough idea, PBR relies on the next hop being locally connected (which it definately isn't in our case - the traffic must traverse a core network, approximately 4 hops or so). Also the core network uses HP Layer 3 switches and this model is not capable of PBR (or a great deal else... but that is another story), so its not like we can simply keep PBR'ing all the way to the local network! PBR would play its part in matching the traffic and assigning a next hop value, we just needed to create some form of connection that appeared local between the two sites to then assign PBR to it.
IP Tunnel
What we needed to do was create an IP tunnel from the remote network to the local one. To accomplish this, we could have used a VPN, but encryption on the internal network was overkill, so a simple GRE IP Tunnel was investigated. It stands for 'general routing encapsulation' and does the job we want of logically creating a point-to-point tunnel between two remote devices. I must say that it is super easy to setup. We setup the remote site router to run keepalives, as to not use the connection if the tunnel was down - a cheap form of disaster recovery you could say! This forced the router to pass traffic through the normal routing table if the other end of the tunnel was offline.
Stuff I forgot about
Whilst setting up the PBR to state a next hop for the 'matched' traffic was easy and the tunnel was also easy to create, It still didn't work.
On the local router the only configuration commands required were that of the B-end of the tunnel. Nothing more. It was this oversight that proved to be the problem.
Of course to traverse the router to access the internet we use NAT. You guessed it already, I didn't add the tunnel interface to be an inside NAT interface - lesson 1.
Lesson 2 was also NAT related. We use ACL's to determine which traffic can access the internet - the remote site's I.P range was not specified and that concludes our lessons for today.
Learnt today?
All of it really. I did have an idea about PBR already but didn't know it was only local gateways that could be used as next hops. I even thought the recursive command would be useful but it turns out that the router performs a recursive lookup for the remote gateway and routes the packet TOWARDS it, rather than encapsulating it etc - this obviously was still an issue with the HP switches.
I also learnt the GRE tunnel isn't intelligent. The remote ends make no difference to routing decisions if it is down and thus creating a routing black hole. I though about using an IP Sla and tracking to check the remote interface and base the PBR on that, but keepalives did the job on the remote router where the traffic originates - this keepalive actually affects whether PBR is used or not - news to me.
Also, did you know that when using PBR you are actually process switching? unless you state (at interface level) - ip route-cache policy.
PBR also was a bit confusing when using a loopback interface to test the tunnel - it didn't work correctly and thus this is because router generated traffic is not influenced by PBR unless you turn it on globally using - ip local policy route-map *map-tag*.
The most important thing learnt today was that working with others has its benefits when drafting up solutions like this. Brain-storming and bouncing ideas of each other is a form of learning, even if the outcome isn't as expected!
Monday, 23 January 2012
Monitoring a connection via IP Sla, Tracking & Syslog
It's been nearly a week before I managed to learn something new at work. It encompasses a few bits though! Here goes:
We have 2 connections at branch sites. One connection heads back to the core network and the other is a connection to our organisations version of the Internet. This Internet connection is managed by a third party and we have to log calls to prompt any action if it is suspected it is at fault - we have no access to the router.
Recently it has been at fault, its just proving it!
We statically route a long list of addresses out of the Internet connection on Cisco 1841 routers, rather than them being routed internally. Its more efficient to use the connection back to the hub site for internal data and the internet connection is funded centrally, so why not use it for itnernet traffic?
The problem seems to be that every 'now and then' user computers will 'hang' for a period of a few minutes, or substantially longer when they are accessing services through this connection. Our desktop engineers have ruled out the PC's - and I was called in. The syptoms seem familiar to a home user accessing a web-page with some inbuilt java or scripting, then their connection dies and thus the PC sits there with an egg-timer until the router/connection is restored and off they go again.
The router and switches that we can manage look fine. No error messages of any sort and the interface counters seem relatively solid.
As we use static routing defined like so...:
ip route 0.0.0.0 0.0.0.0 192.168.242.22
ip route 20.130.152.0 255.255.255.0 10.128.99.32
... The defined route will only check its interface and line status to the device known as 10.128.99.32 - which is the 3rd party router connected via a crossover cable next to it. If the crossover breaks, or both interfaces go down, then the route will be changed to the default route listed above. Its clear that this isn't happening, the interface counters and syslog don't show it to have dropped and internet traffic hasn't been routed to our core firewalls, so the issue must be on the wider 3rd party connection. but how to prove that...
We have IOS 15.1(3)T - so I can't vouch for older IOS revisions in terms of the commands used here, just for info.
Find an I.P
First I found a 'pingable' I.P address of one of the systems accessed over the connection. Its suprising how many of the services we use wouldn't respond, but luckily the most important system allowed ICMP echo's. You may question how pinging one I.P address will confirm the connection is down - and I'd agree with you that it doesn't. But... we do access this system from 100+ other sites and it has an up time of 99.9% so, in my book, its as solid an address as I can find.
Configure IP SLA
Configuring the IP SLA manually will allow you to set definable thresholds, frequency and timeout values. But I did actually stumble across Solar Winds IP SLA monitor. Its free and therefore its not a polished product that can be tweaked to the endth degree, but it works ok for the task. Ensure you have write snmp details for the device and that you know the string, otherwise you'll get zero results!
After configuring the SLA through the application, I did have to amend the configuration in the CLI, just to tell the SLA that it should use the interface connected to the internet connection, to do the pinging!
It ended up looking like this:
ip sla 1
icmp-echo 20.130.152.151 source-interface FastEthernet0/1
owner SW_IpSla_FreeTool_test
frequency 30
ip sla schedule 1 life forever start-time now
I like how the application has given itself an owner tag. In any case, the little desktop application shows a green light - so the SLA is a success at the minute. Also running show ip sla statistics, proves that to be the case. The frequency field is defined by the application (boo) and amending it doesn't allow the SLA to be monitored through the application (frequency is how often the ping occurs, in seconds).
Configure a Track
I then setup a track on this object so that we could assess its reachability state and base some actions on it. The track has to be the easiest series of commands:
track 1 ip sla 1
Show track, gives the state, changes and time of last change etc.
Configuring Event Manager
Rather than looking through the router logs after logging in, we'd like events to be referenced in Syslog so we can view them through our network management software.
Event manager is the way to do this. You can set actions (such as log to syslog with a message) based on the state changes of our Tracked object. In turn, we can build a log of how often the internet connection appears to drop. We can compare this log with another site router (performing all the same tricks as this one) to approach the 3rd party with something comparable - doing their job for them ideally.
Here are the commands to configure event manager:
event manager applet internet-down
event tag pingdown track 1 state down
trigger
correlate event pingdown
attribute tag pingdown occurs 1
action 1 syslog msg "*********Internet DOWN**********"
event manager applet internet-up
event tag pingup track 1 state up
trigger
correlate event pingup
action 1 syslog msg "*********Internet UP**********"
We have two actions based on the 'state' of the tracked object. One logs the internet as down and the other as being up. Its important to log when it comes back up, so we know how long the outage was for.
You may also notice a couple of irrelevant commands - such as the trigger and correlate events commands. Originally during testing I created two event tags for each applet. This allowed two IP addresses to be pinged (I had two IP SLA's and two Tracks) and the correlate would take both into account - its just a slightly more accurate way to say the connection was down/up basing it on more than one I.P address. I subsequently removed the IP SLA 2 and Track 2, but left the trigger and correlate there for you to see what is possible with event manager.
And there we have it. A free desktop application keeps an eye on our IP Sla's as well as logged data recorded from the router. If we have to go a step further in the future, it would be to tweak the SLA to suit our parameters and then base the routing decisions via PBR (policy based routing) on the reachability of some service I.P addresses (or other SLA probes).
But for the time being, management just want a log of how often the problem occurs, so in a month or so, I hope we'll have some results - or not... the poor users will be the ones affected if I do have something to show!
We have 2 connections at branch sites. One connection heads back to the core network and the other is a connection to our organisations version of the Internet. This Internet connection is managed by a third party and we have to log calls to prompt any action if it is suspected it is at fault - we have no access to the router.
Recently it has been at fault, its just proving it!
We statically route a long list of addresses out of the Internet connection on Cisco 1841 routers, rather than them being routed internally. Its more efficient to use the connection back to the hub site for internal data and the internet connection is funded centrally, so why not use it for itnernet traffic?
The problem seems to be that every 'now and then' user computers will 'hang' for a period of a few minutes, or substantially longer when they are accessing services through this connection. Our desktop engineers have ruled out the PC's - and I was called in. The syptoms seem familiar to a home user accessing a web-page with some inbuilt java or scripting, then their connection dies and thus the PC sits there with an egg-timer until the router/connection is restored and off they go again.
The router and switches that we can manage look fine. No error messages of any sort and the interface counters seem relatively solid.
As we use static routing defined like so...:
ip route 0.0.0.0 0.0.0.0 192.168.242.22
ip route 20.130.152.0 255.255.255.0 10.128.99.32
... The defined route will only check its interface and line status to the device known as 10.128.99.32 - which is the 3rd party router connected via a crossover cable next to it. If the crossover breaks, or both interfaces go down, then the route will be changed to the default route listed above. Its clear that this isn't happening, the interface counters and syslog don't show it to have dropped and internet traffic hasn't been routed to our core firewalls, so the issue must be on the wider 3rd party connection. but how to prove that...
We have IOS 15.1(3)T - so I can't vouch for older IOS revisions in terms of the commands used here, just for info.
Find an I.P
First I found a 'pingable' I.P address of one of the systems accessed over the connection. Its suprising how many of the services we use wouldn't respond, but luckily the most important system allowed ICMP echo's. You may question how pinging one I.P address will confirm the connection is down - and I'd agree with you that it doesn't. But... we do access this system from 100+ other sites and it has an up time of 99.9% so, in my book, its as solid an address as I can find.
Configure IP SLA
Configuring the IP SLA manually will allow you to set definable thresholds, frequency and timeout values. But I did actually stumble across Solar Winds IP SLA monitor. Its free and therefore its not a polished product that can be tweaked to the endth degree, but it works ok for the task. Ensure you have write snmp details for the device and that you know the string, otherwise you'll get zero results!
After configuring the SLA through the application, I did have to amend the configuration in the CLI, just to tell the SLA that it should use the interface connected to the internet connection, to do the pinging!
It ended up looking like this:
ip sla 1
icmp-echo 20.130.152.151 source-interface FastEthernet0/1
owner SW_IpSla_FreeTool_test
frequency 30
ip sla schedule 1 life forever start-time now
I like how the application has given itself an owner tag. In any case, the little desktop application shows a green light - so the SLA is a success at the minute. Also running show ip sla statistics, proves that to be the case. The frequency field is defined by the application (boo) and amending it doesn't allow the SLA to be monitored through the application (frequency is how often the ping occurs, in seconds).
Configure a Track
I then setup a track on this object so that we could assess its reachability state and base some actions on it. The track has to be the easiest series of commands:
track 1 ip sla 1
Show track, gives the state, changes and time of last change etc.
Configuring Event Manager
Rather than looking through the router logs after logging in, we'd like events to be referenced in Syslog so we can view them through our network management software.
Event manager is the way to do this. You can set actions (such as log to syslog with a message) based on the state changes of our Tracked object. In turn, we can build a log of how often the internet connection appears to drop. We can compare this log with another site router (performing all the same tricks as this one) to approach the 3rd party with something comparable - doing their job for them ideally.
Here are the commands to configure event manager:
event manager applet internet-down
event tag pingdown track 1 state down
trigger
correlate event pingdown
attribute tag pingdown occurs 1
action 1 syslog msg "*********Internet DOWN**********"
event manager applet internet-up
event tag pingup track 1 state up
trigger
correlate event pingup
action 1 syslog msg "*********Internet UP**********"
We have two actions based on the 'state' of the tracked object. One logs the internet as down and the other as being up. Its important to log when it comes back up, so we know how long the outage was for.
You may also notice a couple of irrelevant commands - such as the trigger and correlate events commands. Originally during testing I created two event tags for each applet. This allowed two IP addresses to be pinged (I had two IP SLA's and two Tracks) and the correlate would take both into account - its just a slightly more accurate way to say the connection was down/up basing it on more than one I.P address. I subsequently removed the IP SLA 2 and Track 2, but left the trigger and correlate there for you to see what is possible with event manager.
And there we have it. A free desktop application keeps an eye on our IP Sla's as well as logged data recorded from the router. If we have to go a step further in the future, it would be to tweak the SLA to suit our parameters and then base the routing decisions via PBR (policy based routing) on the reachability of some service I.P addresses (or other SLA probes).
But for the time being, management just want a log of how often the problem occurs, so in a month or so, I hope we'll have some results - or not... the poor users will be the ones affected if I do have something to show!
Subscribe to:
Posts (Atom)