The purpose of this communication is to explain the tasks we are undertaking with Updata ltd in order to improve the stability of the network and return the service to our previously enjoyed high levels of availability and stability.
Denial of Service (DOS)
As we are all too aware the networks performance over recent months has been extremely poor.
To provide maximum flexibility to schools and to try and restrict as little innovation and flexibility as possible, the security policy applied to the schools service is as light touch as we can make it without compromising safety. To that end we try and support all possible service and traffic types.
Our primary defence against the insecurities of the Internet is our firewall. The firewall on the HICS network is really 2 large chassis based high capacity Cisco units. These are based in the LD4 and LD5 data centres.
As with all such networks we are a target for DOS attacks. Our recent instability has been caused by the firewall devices grinding to a halt. Despite the amount of traffic, they are handling being well below their theoretical maximum capability. Extensive investigations by Updata, supported by the equipment’s manufacturer (Cisco), have identified that the firewalls deployed are suffering all the classic symptoms of a DOS attack.
To reduce the frequency of the outages we have agreed with Updata to block some types of traffic on the upstream ISP routers. This has helped reduce the frequency and its resultant impact but has not in itself fixed the issue. These changes will inevitably have reduced some of the services that may be used over the network. However to date we have had no complaints about loss of specialist services.
Solution to be implemented
Dealing with such situations is complex and time consuming with code uplifts and testing regimes. This is clearly not a direction of travel we wish to take and have agreed with Updata that a more beneficial approach for the Schools would be to deploy a newer firewall solution. This is to be achieved by deploying some seriously enhanced market leading ASA firewall blades.
Cisco are assisting with technical support and some financial support, however Updata are making a major investment and are determined to rectify the issue. They value the relationship and reputation for quality that has been established with the schools of Hertfordshire and they are working hard to fix the issue.
All works and testing will be carried out beyond normal working hours and will be controlled via a strict change management regime.
With any network there are always upgrades and code releases and ours is no different. However we are doing this work in and around the upgrade solution to reduce overall impact.
Here is a summary of the changes that are being implemented:
The Cisco 6506 (the HICS outside firewalls) have been upgraded to a new IOS. This was done in preparation for the new firewall solution that is being deployed. The IOS was upgraded on the 24th February (LD4) and the 26th February (LD5). Both changes went through successfully out of hours.
Updata are to load test the proxy solution by artificially creating traffic on the network to test the recently upgraded proxy component (Authent). We are keen to make sure the network can cope when failed over to the one data centre. This will be completed out of hours, on a time and date that has not yet been agreed.
Deployment of new ASA firewall modules. On the recommendation of Cisco TAC, an entirely different type of firewall module will be deployed into the network. LD4 was completed successfully on the 5th March (out of hours). LD5 is scheduled for the 12th March (out of hours).
There are six Cisco 3600s that are located on the network and they are located in the exchanges around the 10gbps core link. They were upgraded to a more recent release of firmware. This was completed on the 27th February.
Because the firmware on the 3600s has been upgraded, Updata can now arrange some reconfiguration work on them. This will enable additional memory to be dedicated to the IP table space. This should increase performance levels of the network.
There are other changes too, including: extra resilience for the VPN solution and enhanced network monitoring of the network (TBC).
If you wish to discuss any of this with me, please get in touch.