Friday, 1 November 2013

Wow, has it been that long + basic WCCP

Its been over a year since I wrote the last blog update.  Where has the time gone.
Since then, life has been a little crazy.  A new job for starters.  I've learnt and experienced too much to list here - maybe one day, I'll get around to it.  But for now, I thought I'd drop a quick WCCP config in here to reference later.
The scenario is that guest wifi is being rolled out.  The guests are tunnelled into the DMZ and once there, they can access the internet.  The customer would like some relatively simple transparent proxy or filtering capability and so WCCP was common across both the layer 3 switch and proxy appliance.
I've cut and paste most of this from my own documentation, so apologies if its not 'blog friendly'.

What is all this about?


WCCP stands for Web Cache Communication Protocol.  It is a Cisco developed content-routing protocol that provides the mechanism to redirect traffic in real-time.  In RDaSH’s network environment, this is used to redirect Guest WiFi traffic to a proxy server – without the need to manually configure a proxy server in each device (practically impossible with Guest WiFi access).  The technology can work one of two ways; either redirecting using a GRE tunnel, or rewriting the MAC to that of the local engine (called Layer 2 redirect).


There are two versions of WCCP (v1 and v2 surprisingly).  Version 1 only supports HTTP (TCP port 80) traffic flows, whereas v2 supports up to 255 different service groups (such as HTTPs).  Version 2 should be used where possible.


Configuring a Cisco 3750x Layer 3 switch


Ip wccp version 2

Ip wccp web-cache password guest redirect-list 45 group-list 44

Ip wccp 70 password guest redirect-list 45 group-list 44

Ip wccp web-cache password guest


Here we are turning on WCCP and setting the version.  We then tell the device which ‘services’ to run with WCCP.  Normal port 80 traffic is called ‘web-cache’ so we define that.  We then define an additional service called ‘70’ which is actually an https-cache service.  We set the password to use with the WCCP service – this is defined on the proxy/filter as well, so they both have some relatively simple security.  The final parts are ‘redirect-list’ which references an ACL in order to match traffic we want re-directed. This is typically the client range.  The ‘group-list’ also references a different ACL, which is matching defined cache engines – or in our case, specifying the proxy/filter.


Int vlan666

Ip wccp web-cache redirect in

Ip wccp 70 redirect in


At this point we need to define the interface on which WCCP should begin to ‘redirect’ traffic that we’ve asked to be sent to a proxy.  Remembering that ‘inbound’ traffic is the direction specified, as in order to leave VLAN 666, traffic will be coming into its L3 interface before heading out to the internet.  We also define both services to redirect, that we configured earlier.


access-list 44 permit

access-list 44 remark **Sophos appliance for guest WCCP**


access-list 45 permit

access-list 45 remark **DHCP Guest WiFi range for WCCP**


The ACL’s as specified earlier contain the client range to redirect and the address of the appliance/cache engine.




Unsuprisingly this solution did not work out of the box on the 3750x and hopefully these pointers may assist you.  I’m not laying the blame totally at Cisco’s door however, there was some Sophos research required as well.


SDM prefer

By default the 3750x uses ‘desktop default’ template.  In short, the Switch Database Management Template (SDM) doesn’t allocate resources under this default template to do any form of WCCP (it’s actually PBR via the TCAM which is responsible for WCCP).  Thus you need to change the template using ‘SDM prefer routing’.  This requires a switch reboot.


IP routing

Obviously make sure IP routing is enabled!


Re-direction method

The 3750x only supports L2 and not GRE – so make sure your appliance is set to L2.


Things in different subnets

I’d read that you need to have clients, cache servers etc in different subnets for WCCP re-direction to work – but this seems unfounded, at least when using L2 as the redirect and return method.  Maybe this is only evident when using GRE, which the 3750x doesn’t support, so I couldn’t test it anyway.


Fast Timers (Headache – take note)

In other words, keepalives – time interval.  The Sophos web appliance didn’t like this feature which I’m informed is standard on cisco firmware later than 2012 or WCCP v2 rev1.  There was no indication that this was to blame on the switch and no indication on the appliance either.

What you will see, is that traffic isn’t redirected.  Instead when issuing ‘show ip wccp’ Total packets Unassigned will increment.  Cisco documentation just states that this means the ‘cache engine’ can’t be contacted for whatever reason.  Not much to go on.

To fix the problem, you must issue the command (it’s a hidden command, so using ? won’t help) ‘no ip wccp variable-timers’.  As if by magic, it all works fine after that!  Maybe a Sophos related problem…. But there you go.


Further information

When viewing debugs on WCCP you will see a lot of packets sent with HIA and ISY I the description.  If you research these, you’ll see this is the fundamental way in which WCCP communications is established.  HIA = here I am & ISY = I see you.
Should the need arise to create/filter different WCCP service identifiers then use the table below.  YOU MUST also ensure that the filter/proxy/appliance is capable of reading these identifiers as well,  otherwise redirection will not

Monday, 13 August 2012

Cisco Smart Logging Telemetry (SLT) on a 3560G - Netflow Trial

A while ago, I researched the possibility of using/enabling Netflow on a couple of Cisco 3560G switches we'd purchased.  After much head scratching it was deemed you'd need IOS revision 12.2(58)SE to get any form of netflow - and its not even net flow. Hmmmm....

So... out of hours, I upgraded one of the switches to this new firmware to give it a go.
After a nervous wait for it to restart (Logging in remotely from home) I checked to see if the commands are available - which they were.
So off we go:
Create the exporter
flow exporter test-collector
 description "collection of data for the boss"

This creates an exporter profile in which I can describe what its for and where the data should go.

Then configure smartlog
Conf t
Logging smartlog
logging smartlog exporter test-collector
logging smartlog packet capture size 1024
We then need to give it something to report about.  As a test I created an ACL which permits everything:
access-list 97 permit any smartlog
and then assigned it to an inteface
interface GigabitEthernet0/3
 description Monitoring port for the boss

 ip access-group 97 in

And thats it - pretty easy to setup but to be honest its limitations are immediately visible when using Scrutinizer (or other net/s/flow related applications).  I've just shown you a test example of the ACL here - I amended it later to monitor a specific DENY on a particular protocol on a specific port.

This is netflow but with restrictions.  The data has to be event based and as such - you can view the data when an event has occurred - hence it has been logged.  That in itself limits what you can see.
Don't get me wrong, its far easier than trawling through syslog data and you can drill down into the raw data to at least see some of the packet, but other than assessing security concerns, I'm struggling to see how I'd use it.

In terms of satisfying the original request to view netflow data, I will put the question back to the originator with a politically cryptic hint of sarcasm.... "What netflow data????"

Thursday, 19 July 2012

Netflow NoGo on a Cisco Catalyst 3560 switch

In our haste to purchase switches for an already over-run project, it would appear we overlooked the product features of the 3560G. 
I have been asked to enabled netflow on one of the two switches we use in a production environment, but after much head scratching - the 3560 doesn't support it.

Looking further into it, the whole 3000 series doesn't.  Unless you either buy the uplink 10G modules (but then I'm sure it'll only allow you to monitor uplinks) or use a trimmed down version of netflow exporting (appearinging in 12.2(58)SE) from a later revision firmware.  Of course, we are a few releases behind that - but I think it might be worth giving it a go anyway to try and satisfy the request.

So, its a netflow no go.  For now.

Wednesday, 13 June 2012

IPSEC over GRE VPN? or maybe its the other way around...

Its been a while since I've sunk my teeth into something new.  I've had the time consuming tasks of updating CV's and interview skills, for the impending shake up of the organisation I work for.  CCNP study has taken a slight sidewards step (better than backwards) during recent weeks - but I've got a holiday booked shorty and there is no better way to learn than in the sun.

Anyway, we have the following situation:

Without concentrating on the lines and arrows, there are two sites depicted in the diagram.  The top one (the top blue oblong that covers the LAN and two routers) is what we'd call a hub site.  It is one of 5 sites that all connect together to form the core of the network.  As you can see with the 3 black lines coming from the Cisco 3745.
The other site (the bottom blue oblong) is what we call a remote site.  They typically connect back to their closest hub site, (based on legacy geographical cost) and we support about 60 of these remote sites - so the diagram is obviously cut down.
Both sites have an internet connection.  The router that manages the internet connection is controlled by a third party - so we have no access to it.  It has no firewall though, so we run CBAC on our routers.

The green lines at the remote site show traffic flow.  The 1841 makes the decision (based on static routes) which path to take based on the destination address.  If it is one of a pre-defined IP range for a website, then off it goes out of the local internet connection.  If it is an internal address (server, other remote site etc) then the packets are routed through the default route and back to the hub site via a 2mb serial connection - WHICH IS OVERSUBSCRIBED.

And thus a question has been asked:

"If the 2mb connection goes down (often), can there be a backup route using the internet connections?"


"Could the two routes be used in tandem to increase throughput?"

The diagram (if possible) might look like this:

The answer is yes. To both questions.  Although I can't see this being implemented.

We already use GRE tunnels (see my previous posts) but that is internal.  With this setup, we'd need some form of encryption when tunnelling through the internet. 

I'll explain the tunnel setup first and just gloss over the other parts.  As I mentioned, this probably wont get implemented and its purely just answering the question to detail a business case to make it happen to then not make it happen.  Does that make sense?  No - thats politics for you.

On with the technical.

IPSEC over GRE/GRE over IPSEC...
Its one or the other, but I don't know/don't need to know which - google is your friend, if you're bothered.  Technically the tunnel would be accomplished using general routing encapsulation, but yet we'd run IPSEC to encrypt GRE packets.  But then, you need IPSEC to create the encryption to run GRE in the first place.  But you need the tunnel to then use the transport protection of IPSEC.  Chicken & Egg, I suppose.

I've configured the remote site router first:

crypto isakmp policy 10
  authentication pre-share
crypto isakmp key CISCO address

We define an ISAKMP policy (Good info about what this is and what it does is here) (and give it a priority number) and then instruct the policy to use pre-shared keys as the authentication method for this policy.  Two peers must have a common policy or they wont connect.
We then specify what the key will be, in this example CISCO is used - passwords and IP's are abviously made up for the purposes of this post! We also define the 'other end' I.P address with whom the key exchange will take place.

crypto ipsec transform-set Transformers esp-3des esp-sha-hmac
  mode transport
crypto ipsec profile MyProfile
  set transform-set Transformers

Transform sets are then used (in this case we called ours transformers) to detail what encryption and authentication methods are used for the ESP protocol (Encapsulating security payload) which is what establishes the IPsec tunnel encryption and authentication between the peers.  They must be the same at both ends or the IPSEC tunnel will not form.
The ipsec profile defines the parameters to be used between the two routers which we will reference in the tunnel protection command later.  It references the previously created transform set - You might find this confusing, but remember you can use different transform sets with different profiles and even both of them can be referenced with different ISAKMP policies!  As our example is using a simple point to point, some commands can seem overkill, but they are necessary.

interface Tunnel1
  ip address
  tunnel source
  tunnel destination
  tunnel mode ipsec ipv4
  tunnel protection ipsec profile MyProfile!

And here we have one end of the tunnel.  This one is referenced locally as 'tunnel1'.
The ip address is a completely private range and we've assigned a subnet mask so that only the two peers are useable.
The tunnel source is the interface IP for the interface you want to create the tunnel on.  Similarly the destination is the interface IP of where the end of the tunnel should be.
Rather than using GRE, we specify the tunnel mode to be ipsec ipv4 and then introduce the tunnel protection command to reference the ipsec profile - which in turn references the transform set with encryption and authentication information.

Thats pretty much it.  That creates one end of the tunnel on the remote site router and the process of creating the other end is just a reversal of the IP addresses used above - oh, and the ip address of the tunnel being the other useable one in that subnet!

We configured the above in a test environment and guess what?  It didnt work.

Why it didn't work
I mentioned earlier that we use CBAC (or ip inspect as its referenced) and thus inbound access lists prevent any nasties from the internet making their way into the corporate network.  We had to add a couple of earlier sequence numbers to an extended access list to ensure that both UDP and ESP were permitted from the respective peers.

Like so:

ip access-list extended INBOUND
  8 permit udp host eq 500 host eq 500
  9 permit esp host host


Remember that because the list is INBOUND and applied with the IN statement, the source and destination fields of the ACL must reflect the peer at the other end.  I'm only making this point because I'm used to amending ACL's for outbound access and thus I did it wrong.
Default keepalives were at 10, so we thought we'd adjust them to 5, but its user preference and not the reason why it didn't work.
Be aware:
Show int tun1 - should tell you if both the interface and protocols are up.  Check the logs too, to ensure the tunnels are passing the authentication and encryption stages of the exchange.  Ping the private addresses from each device to ensure connectivity is established.  If you add static routes to test end to end Ip ranges, then ensure you specificy the source interfaces when trace routing or pinging.
Using it as a backup route - question 1
So the question was asked about using the tunnel as a backup.  Given the 2mb serial is referenced with a: ip route then we can simply add another static route but with a higher cost, to form a 'floating' static.
Load balancing - question 2
I didn't look into this option (I know, I know) but its simply because I can't replicate it in a lab environment.  BUT... because we use RIPv2 then technically the tunnel would have to have the same hop count to reach the subnets in question as it would using the 2mb circuit.  If so, then by default the Cisco router would load balance with 2 entries in the routing table.  See my earlier posts about process and packet switching to understand how this load balancing would occurr.  It would likely be load balanced per destination and not per packet - even so, with the 2mb serial connection saturated, a little breathing space would be created if only temporarily!

Hope it helps you if you need to create a point to point tunnel - securely.

Monday, 16 April 2012

Symantec Antivirus SAV 10.x goes End of life (well EOSL) sort of..

7ish years ago SAV 10.x was at the forefront of corporate AV protection.  I personally thought it was great and we happily upgraded a year after its launch from the 9.x version on our network.

An announcement has been made that it is now going EOSL in July 2012.  Its interesting to note that it will go end of support life rather than end of life - yet the message we have received and seen can be a bit worrying...

Firstly, the use of 'support' life will cause many admins to raise an eyebrow.  Does this mean that we can still get definition files/updates?  if not, we'll have to hastily migrate to the SEP 12.1 product within 3 months - or switch vendors all together.

Secondly, there is no mention of this in the emails sent to customers and/or on the website.  By not announcing eitherway whether definitions are allowed passed July 2012 will potentially scare customers into going with SEP 12 anyway - a clever marketing ploy I think.  This could backfire though, like I said, its a choice to upgrade to SEP 12.1 and its just as much of a change than it would be to switch vendors altogether.

The news from our camp (after contacting Symantec) is that XDB updates will not be available online for SAV 10.x installations.  We rely on manually updating the definition files daily - whilst many employ the LiveUpdate application to do this for them - we're old fashioned and like to see the file download + run.  BUT.... if you continue to use LiveUpdate for SAV 10.x installations after July 2012 then you 'should' still be ok.

Make of that what you will.

I noticed that you can pay to have definitions available beyond July too!  So Symantec would charge you for packaging the update and hosting it for download - when LiveUpdate will continue to work for free?  Money for old rope? - I think so.

Now, many people will question why you're still on a legacy AV product anyway - what with security and all, being a top priority in most organisations.  I do have some sympathy for the folk left behind - I've managed migrations to SEP and it is a task - one that should be project managed if you're doing a large scale deployment and its features and functions are massively different from the 10.x platform to boot.  Smaller organisations might have employed better edge protection and thus have decided to stick with 10.x or there might be financial reasons.  The best case I've heard is of a small business running 10.x because when they paid for contracted resource to install it (in 2008) it has just 'worked' ever since.

I still think that Symantec could have/can do this better.  Their clever/not-so-clever decision to omit the definitions question with either alienate or force the hand (neither make good customer relations).  Why not announce that defintions will continue to be available for 6 months after the EOSL period or tie in with the renewal period of current licence holders?  Sack off the fee for updates idea (that just makes you look greedy) and distribute trials of SEP to current 10.x customers so they can use the 6 months to familiarise?

Anyway, goodbye to SAV - thanks for all the phish(ing) emails you've not protected me against - only joking Symantec.  Can I have a job please?

Wednesday, 11 April 2012

EIGRP Route filtering (from CCNP study)

I have managed to mock up the 'work' network using GNS3 = great success.
As such, I can apply the CCNP knowledge learned through study on the work based topology to enhance my comfort levels with the various commands.

Today (and the past few weeks) have been focusing on EIGRP which we do not use at work (from the ICND1 days, you'll know that EIGRP is Cisco's proprietory routing protocol) because we have some HP (peh) equipment at the core.

Route Filtering with EIGRP
Filtering inbound routes by source (using a route-map)

I set myself a small task of filtering EIGRP routing updates on an inbound interface.  I inserted the following commands:

acccess-list 99 permit then:route-map test deny 10
match ip address 99
route-map test permit 20
and then (under the router eirgrp <process number>:
distribute-list route-map test in

Explaining the above: I wanted to block routing updates from a device with I.P listed in the ACL 99 I created first.  I then created a route-map called 'test' and denied anything that matched the ip address defined in the ACL 99.  Route-maps end with an explicit deny, so a permit statement with a later sequence number (20) was added to permit everything else.
I then inserted the distribute-list command referencing the route-map name and specified it as inbound.

Sounds good - but it didn't work! LOL.

The answer was with the match statement in the route-map:

match ip route-source 99

With this in place, routing updates from were not in the routing table - leason learnt.

Filtering inbound routes by what is inside the route advertisement (Using an ACL)

Now lets say we wanted to receive routing updates from the router but within its route advertisement it was advertising a route to and we would like to remove that.

Distribute lists do not work with extended ACL's so its best to create an standard IP ACL.

ip access-list standard test
 permit any

Like so.  We have created a standard ACL called 'test' and denied our subnet we want out of the routing update.
Under the router process we define the distribute list:

distribute-list test in

To check the ACL is matching packets you can run a show access-list command and the show ip route command should confirm that the route is being filtered.

I will state as a mini-disclaimer that I do not like route filtering inbound unless it is a real requirement for complex networks.  Route re-distribution plays a role in inbound filtering but for most networks the inbound filtering component can be a headache to manage and understand (especially if not documented correctly).  Also in most enterprises a branch site router over some WAN link would want a route filtered before it traverses the WAN link!  otherwise the bandwidth is being used, only to the have the route removed at destination.
The stages of learning early CCNP text reference route filtering on outbound interfaces only - obviously to keep you sane! but also, its far easier to define what is being filtered out than in (in my opinion) at this early stage of learning.

Monday, 26 March 2012

Break Sequence to access Rommon on a Cisco Router

I was handed a lovely new laptop by work, but it didn't have a pause/break key.  I needed to perform password recovery on a router we'd have for too long and thus I though I was stuck for a minute.

But you can initiate a break sequence by doing the following:
  • Obviously connect to the router via the console port
  • Open a hyperterminal/terminal session (I use Putty.exe) and set the connection to use serial and a speed of 1200.
  • Power off the router and back on again, whilst pressing the keyboard every second, for 10-15 seconds.  You will just see a load of crap on the screen, but thats not a problem.
  • Close the terminal session (do not power cycle the router or anything) and open a new one with the correct baud speed of 9600.  You may have to press enter, but it'll put you at the Rommon prompt.