There is an IOS upgrade on the ACE module tonight in LD5. The work is set to go ahead 10pm onwards and there is chance of some downtime, of no more than 30 minutes. This will affect the schools connected to the LD5 data centre. Which will of course mean anyone trying to connect into the school remotely, could experience connectivity issues.
Any queries, please get in touch.
For those of you trying to connect remotely into a school in St Albans, there are currently power issues. I am told that UK Power are hoping for this to be resolved by 11:30am. Just to confirm this is NOT a HICS issue.
On Thursday evening an emergency change was deployed on the transparent proxy network. Updata are now “highly confident” that they have found the root cause of the recent issues and applied the relevant fixes. Monitoring of this platform on Friday suggests that performance is once again stable.
When transparently handling HTTPS requests there were approximately 100 bytes of memory that was being leaked (allocated, but never released). Over time, as more and more HTTPS connections were being processed, these bytes began to build up and start consuming all the available RAM in the system. This then caused the operating system to start swapping to disk in a vain attempt to make more memory available. Following Thursday’s change, the memory utilisation is once again stable and no longer increasing.
Please accept our apologies for the unsatisfactory time it has taken for Updata to fix this. Senior Management from HCC and HfL are meeting senior management from Updata next week to voice concerns over the recent performance .
At approximately 3pm today there was a spike in response times on the HICS network. This would have affected schools connected to the LD5 data centre (if you are unsure we can tell you what data centre you go out to the internet from).
Within each data centre, Updata are constantly polling various and popular websites such as BBC, Google and Wikipedia. The chart below shows you that the response times increased considerably around this time. So if you noticed it, the issue was on the HICS core – not your LAN. I have asked Updata to provide more information on this and I will add further comments to this post. In the meantime, please get in touch with me if you wish to discuss this in more detail.
The transparent proxy is down. Updata are dealing with this and I am told they will hopefully be failing over all transparent proxy traffic to the corresponding data centre shortly.