February 22nd performance issues

On the 22nd of February 2017 at approximately 08:56, The HICS support team started to receive calls reporting slow speeds for multiple sites across the network. This was logged with Updata as a priority 1 incident and escalated within Updata and their Major Incident process was invoked. This impacted performance for schools as well as HCC Corporate sites.  Significant packet loss was identified across the network and the cause of which was subsequently identified as an MPLS issue between LD4 and the Hemel Hempstead exchange. Updata worked to improve the flow of traffic across the Herts network. Work was carried out to reroute traffic away from Hemel Hempstead and this reduced packet loss, leading to an improved experience for users at some sites. Updata subsequently diverted all traffic away from the Hemel Exchange and this further improved performance.

Meanwhile, the HICS Support team started to receive reports that the Netsweeper service (filtering platform) was unavailable. Attempts to access the internet were met with “Internet Filtering Service is Unavailable” error message. Proxy servers were restarted without success. The Virtual Machines on which the proxies reside were then restarted which did restore service.

The Updata hardware at both the Hemel Hempstead and LD4 ends were checked and found to be operating correctly. The root cause of the incident has been attributed to a hardware failure of a Small Form-factor Pluggable transceiver unit at the Hemel Hempstead end of the link to LD4. After diagnosis BT engineers identified and replaced this faulty SFP unit at the Hemel Hempstead exchange. Traffic was then rerouted back to its primary path. Testing was carried out and no further packet loss identified.

Posted in Service | Leave a comment

Procurement

The HICS network currently provides internet and connectivity services to over 520 schools and academies in Hertfordshire.  The service is strategically managed by HfL and the ICT in Schools Partnership Working Group, which is made up of representatives from the headteacher and governor communities.  The service has enabled all schools in Hertfordshire to receive an excellent connectivity service, wherever they are situated, whilst also ensuring that they get a value for money, educational service providing secure links in and between schools and where appropriate, the LA.

HICS is a tailored education service, led by educational staff.  This means that we are able to proactively manage safeguarding concerns for schools, reflecting not only OfSTED requirements but also changes to national and international internet practice.  Our filtering options enable schools to take varying levels of control, based on their own staffing and expertise, whilst also ensuring that pupils are protected from emerging online dangers.

In order to ensure that we are able to continue to provide the very best level of service, maximising the impact of aggregated purchase and new technological developments, HICS has recently reproduced the service.  This means that our supplier is due to change.  This should provide service advantages to our evolving school community, whilst protecting the benefits of the HICS network that has existed in Hertfordshire for the last 8 years.

Q What is the key change to the HICS service?

A HfL’s contract with the current provider runs out on September 30th 2017. We have a new supplier contract from October 1st 2017 which will enable us to offer you an improved level of service.  From a customer facing perspective, you will continue to receive a full wrap around service from HfL, with the same contact details and level of service.

By carrying out a full tendering process, HfL have been able to negotiate a better price and improved service for schools, which maximises the impact of technological changes for schools.  We have engaged school headteachers and governors throughout the process, ensuring that the new service meets all requirements in terms of OfSTED and safeguarding, as well as ensuring that schools and academies do not have to carry out their own tendering processes.

Q What happens if I decide I want to resubscribe to HICS after this date?

A We will never turn away a school/academy, although you will run the risk of having a period of time without a broadband service.

Q What happens if we decide we do not want to continue to subscribe to HICS?

A Of course, there is no compulsion to resubscribe to the HICS network.  However, we believe we have negotiated the best price for the level of service a school needs.  We would therefore urge you to ensure that any new service provides the full education cover and filtering that your school needs, so please do check that you have a like for like cost comparison. You will need to have the new provision live by October 1st 2017 as your current provision will cease at midnight on 30th September 2017.

Q How long will the school/academy have to subscribe to HICS for?

A In order to negotiate a good financial deal for schools, we have had to offer a longer period of delivery with our supplier.  Therefore we will be asking schools to confirm their subscription for three years.  Of course, this also means that schools are able to budget ahead for three years, and protect themselves from price rises throughout this period.

Q How have HfL decided on who the broadband provider will be?

A In 2016, HfL commenced a full OJEU (Official Journal of the European Union)procurement to identify who the contact will be awarded to, this included researching the market and speaking to schools/academies for their feedback through various means such as the partnership working group. This was to ensure best value and improved functionality to schools/academies.

Q Will the service improve and if so, how?

A We have chosen a proven provider that specialises in delivering broadband services to education and their existing core network has availability of 99.99%. As such, the new HICS service will deliver a range of improvements and benefits to your school:

A proven web filtering service  – Going beyond the latest DfE and Ofsted guidance and duties for online safety

Improved performance and results – A ‘market leading’ SLA guaranteeing reliable connectivity for your staff. The service will be designed and implemented to provide 99.99% availability 24 hours a day, 7 days a week

Greater security and safeguarding – Improved measures to ensure business continuity and protection against the latest online threats, like Malware and DDoS attacks

More flexibility and choice – The opportunity, as the need for bandwidth increases, to scale up your service. Also, the option to take greater control of a wider range of service elements, e.g. your filter policies, DNS settings and firewall configuration

Greater confidence – More feedback for your Designated Safeguarding Leads, who will receive scheduled reports to show when attempts have been made to access inappropriate online content

Q How much will the service cost?

A The costs will be in line with what you are currently paying, should similar numbers of schools/academies commit to HICS in line with what we currently have. We expect discounts in the second and third year if the current commitment is the same.

Q I am a Secondary School/Academy Network Manager and I would like further technical information, how do I get this?

A Further communications will follow about times and dates of a Q&A session that HfL will be hosting alongside the new provider. If you are interested in attending, please contact Kevin Crawley (kevin.crawley@hertsforlearning.co.uk). The new provider will also be holding free training sessions on certain services such as the filtering that you can attend in due course.

Q I work in a primary school/academy and I would like my third party IT support engineer to familiarise themselves with the new service prior to launch. Is this possible?

A If there is a demand HfL will be happy to host an event and invite primary schools support engineers in to discuss the new service; this will be attended by HfL and the new provider. This will be organised closer to migration, allowing the knowledge learnt to remain current and up-to-date.

Q How will HfL manage this contract on our behalf?

A HICS is, and continues to be, steered and overseen by the ICT PWG – which is made up of heads and governors. The service will be underpinned by a market leading SLA which HfL will manage on behalf of subscribers. All service impacting credits will be refunded to your HfL account. We will be your one stop shop and the existing HfL support teams will remain in place. Regular Network Manager forums will take place allowing school representatives to ask questions direct to HfL, as well as the broadband provider.

Q Will it continue to be an educationally driven service?

A Yes, it will. The new service will proactively respond to safeguarding/OFSTED concerns, changes in Government legislation etc.

Q I am a rural school/academy struggling on my existing bandwidth. Will I be able to receive an acceptable level of service, at an affordable cost?

A Absolutely. One of the key benefits of our central contract is that we are able to use our purchasing power to drive best value for all of our schools, and ensure all schools receive an excellent level of connectivity.

Q If I leave HICS, will I be able to access HCC applications such as SOLERO and SEAM?

A HCC manage these connections.  HfL can, as the largest supplier of internet services in Hertfordshire, negotiate these centrally rather than individually as part of its wrap around service.  HfL is not able to negotiate on behalf of other internet providers.  If a school was not part of the HICS provision, these interfaces would need to be managed directly by the new supplier and HCC; please contact the technology team at HCC on p&tpolicy&processteam@hertfordshire.gov.uk .  Clearly, any changes to HCC interfaces are managed by HICS in a proactive manner.  Alternative suppliers would manage these directly.

Q What is the expected timeline?

A Further communications, along with timelines and costs will follow shortly..

Posted in Service, Service Improvements | Leave a comment

Major Incident report 6th February to 9th February

As explained previously, whenever there’s a major incident, Updata provide a report. Here is a summary from the week prior to half term – any queries, please get in touch with me.

Description of incident and customer impact

On the 6th of February at approximately 21:07, Capita Reigate 24/7 Network Operations Centre (NOC) Engineers were proactively alerted by their network monitoring tools to the loss of numerous devices in LD4 and LD5 datacentres. The incident was raised as a priority 1 and assigned to the oncall Technical Escalations Team engineer to conduct initial investigations. All customer traffic traversing these links was down for the duration of the incident. Herts Corporate and Herts Education sites suffered Network connectivity issues of varying degrees for the duration of the incident.

Resolution details

Resolution to this incident comprised numerous activities described below:

The initial LD4 6513 hardware (HICS core router) failure was resolved by the replacement of the failed supervisor card and reboot of the device.

The downstream routing issues were resolved by the Technical team trouble shooting and bouncing MPLS tunnels and local PE devices.

The high CPU usage on the LD5 6513 was attributed to a corrupted table that came about as the result of the initial unscheduled reboots of that device. This was corrected under the direction of Cisco Technical Assistance Centre (TAC) by a clean restart of the device which rebuilt the corrupt routing table.

Root cause analysis

The root cause of the incident has been attributed to catastrophic Network failure caused by the failure of the supervisor card in the LD4 6513 device. All subsequent impacts are directly related to this failure. It was noted that a scheduled change was being implemented just prior to the hardware failure. When the Internet was brought back up there was a lot of routing convergence happening. It is possible that this sudden flood of routing requests could have caused a failure on the supervisor card.

Summary of incident

The Capita Reigate 24/7 NOC were alerted to a hardware failure in the LD4 & LD5 datacentres relating to the Herts Network. They engaged the Major Incident (MI) Manager who assembled the MI team. A field engineer was tasked to go to the LD4 datacentre. Upon arrival he identified that the onsite 6513 device had failed and would not reboot. It was subsequently identified that the supervisor card had failed.

A Cisco TAC case was raised and a spare part ordered Cisco advised that the service agreement for this device was for next business day delivery. At this point a failover was attempted however due to configuration issues this was not possible.

In parallel the MI manager got in contact with the Capita Tannochside support team and was able to agree the sourcing of a spare supervisor card. This was expedited and couriered to the Datacentre arriving at approximately 09:40 on 07/02. The card was configured and fitted bringing the device back up. Some network availability was restored however a number of routing issues persisted. The support teams worked on these issues providing network connectivity for the majority by 19:00 on 06/07.

A downstream impact was later identified whereby some schools were experiencing slow internet connectivity. The issue was investigated and after engaging Cisco TAC was found to be an issue with a routing table on the LD5 6513 which was causing the device to run at extremely high CPU usage. An emergency change was implemented to reboot the LD5 6513 and normal internet connection speed was restored to Herts schools.

Updata’s Observations and Corrective Actions

Contracted support for the LD4 & LD5 6513 devices is next business day. For critical devices this is not adequate  

When devices are brought into service the appropriate level of support should be considered based upon criticality of the device. Additionally when amendments are made to service the support requirements should be reconsidered.

When the device failed a TAC case was raised with Cisco and an Replacement arranged. Going forward this activity should be coordinated via Tannochside 

Support documentation to be updated to ensure that the Tannochside Service Desk are engaged immediately when it is identified that there may be a requirement to procure spares.

The 6513 devices are old and various components are very close to end of life  

The Service Architect team have already made the customer aware that there are exposures with the current end of life hardware. They are in consultation to plan the potential decommissioning of the aged 6513 devices. This will involve a re-engineering of the Network.

Spare parts were not readily available and had to be ordered from Cisco

Updata system’s stocklists to be checked to ensure that information is current and also that the location of any spare parts is recorded correctly

The attempt to fail services to LD5 was not successful due to a reliance on the LD4 datacentre

It has been identified that over the course of time and with numerous projects that have changed the network topology the resilience that was once available has been diminished. As part of the ongoing discussions with the customer the Service Architect team will be reviewing network resilience and the  failover process as an integral part of the proposed network redesign.

During the course of the major incident it was difficult to gain a definitive customer impact

In order to manage incidents appropriately a definitive customer impact is vital. The process for gaining impact needs to be reviewed with the customer to ensure that it is recorded accurately. Additionally all issues experienced need to be recorded in the Incident Management tool for completeness and also to ensure that these issues are taken into account when assessing the impact.

              

Posted in Internet access, Service, Uncategorized | Leave a comment

Prevent alerts now available on HICS

We are pleased to inform you that HICS now has an added layer of protection for its users. Schools can now opt in to receive emailed alerts notifying them when local sessions have accessed websites with extremist related content, in line with the Government’s Prevent Strategy. The alert will be sent out within a few minutes to an email address of your choice. The event gets captured in real-time, then queued for screen-shot generation, and finally it might take a minute or two for the email itself to be delivered.

The list of words that generate these alarms are a closely guarded secret because if they are readily available, it will defeat the object. The words are ever changing and research is on-going meaning that when new information and words come to light they are added. The words are based on detailed research into terrorist groups and their propaganda. It includes the names of people, concepts, ideas, places (such as routes used to travel to Syria) and also the media arms and publications that terrorist groups use.

The content on the webpage is scanned in real-time by the HICS filtering platform. It should be noted that only specific content types are scanned – basically anything text-based, which includes HTML, Javascript and Stylesheets – so images (jpegs and png files etc) are not scanned – nor are executables. When a match is found, the proxy then queues up a screen-shot request to the screen-capture service (which runs on the same server as the proxy). The screen-capture service will then log the event and send an email out.

In order for the scanning engine to be able to detect the words, it must have access to the data-stream – this means that SSL inspection must be set up. For now, activity will only be logged against internal IP address and not users from Active Directory. Hopefully as the alerts are generated in real time this will not be a huge issue but rest assured that we are investigating ways to sync this with Active Directory.

To get this set up, all you need to do is to contact the HICS Service Desk and request this gets deployed.

Posted in Web Filtering | Leave a comment

Procurement update

Lots of you are understandably very keen to hear about the next HICS provision (October 2017 onwards). The procurement has taken a lot longer than we had hoped but we are nearly there to answer your questions. Trust me I am very keen to open dialogue.

Just to keep you all in the loop, a very high level overview:

  •  The HfL HICS marketing campaign will be starting next week – more information will then become apparent.
  • Lots of service improvements  are on the horizon– improved filtering platform, more proactive filtering alerts, app based filtering etc. School can manage their DNS, firewall requirements through HICS, rather than logging this with us. (although we can still do this if you’d prefer). Lots of emphasis on resiliency, backup circuits,  scheduled fail over testing. Traffic shaping/prioritisation. Plus lots more.
  • Prices will be in line what they currently are – possibly with savings in year 2 and 3, dependant on the take up. These are currently being worked on to make sure they are competitive
  • Change of Broadband provider – Updata will no longer be the network supplier (can’t name who the new supplier is just yet)
  • I will be hosting an event for a Q&A session with the new provider in the coming weeks ,which secondary schools are welcome to attend. Date TBC
  • If there’s demand, I will also invite companies in who support primary schools for another session.

Further details next week…. I really appreciate your patience.

 

Posted in Service, Service Improvements | Leave a comment