Load-shedding, the South African term for scheduled, rolling electricity blackouts by the national power utility, Eskom, has wreaked havoc on the South African economy, with major consequences for business and industry, education and healthcare. Load-shedding has further revealed the stark inequalities in the country: those individuals and organisations that can afford to pay for private solutions do, and those that cannot, simply go without. The inequalities felt in South Africa are also keenly felt among TENET’s research and education partners, primarily public universities, which see an enormous disparity when it comes to access to resources. While some institutions are able to fund alternative electricity generation and keep the lights on and classes running, others simply are not.
Resource inequities aside, South African public universities and research institutions are interconnected and interdependent when it comes to network & Internet connectivity. All are connected to the South African National Research Network (SANReN), as the connectivity needs of these institutions are unique and specific, and cannot be met by commercial Internet providers and infrastructure.
“Collegiality and shared ownership of the NREN (national research and education network) has been a model that has served us well for the better part of the last two decades,” says Guy Halse, head of trust and identity at TENET. “We have relied on reciprocal arrangements with universities to host the necessary infrastructure, as opposed to using commercial data centres, as this lowers costs, reduces dependence on commercial operators, and builds a sense of ownership.”
But unfortunately load-shedding has upset this balance in cases where institutions hosting infrastructure do not have as much power redundancy as those who depend on it.
Impact of load-shedding on the SANReN network
The SANReN network comprises a national backbone and several metropolitan rings as well as extensive long-haul circuits to important research installations. This infrastructure connects all the partner institutions reliant on the SANReN network.
Load-shedding has had a major impact on this network, as, with it being interconnected and interdependent, when one site goes down it can impact the others, and when multiple sites go down the consequences are severe.
“In order to understand the impact of load-shedding on the network it is important to understand how the network is set up,” says Maria Mvungi, service support manager at TENET. “Often in metropolitan areas the network will run in a ring around the region, connecting campuses of multiple institutions.”
She explains that in a ring, if there is an issue such as a fibre break or power failure, traffic can flow in the other direction to ensure other campuses are not affected. The trouble with frequent load-shedding is that it happens all too often that there will be a break at one segment, and loadshedding at another site along the ring. This completely isolates sites between the two problem areas.
“In the best case scenario, this kind of isolation lasts only for the two or so hours of load-shedding,” says Mvungi, “but sometimes there are municipality power failures resulting in blackouts for several hours more.”
This causes understandable frustration for those institutions who have been able to implement alternative solutions for power, especially in a post-COVID world where online teaching and learning is widely used.
Another challenge for Mvungi and her team is that it is not always possible to tell when there is a fibre break that needs to be repaired on the network, or where the network has simply gone down due to load-shedding.
“Each time there’s a network outage, we have to try and reach out to institutions to get a sense of what is happening,” says Mvungi, “but sometimes we cannot reach colleagues, sometimes it is after hours. It takes up a lot of time just to figure out if we need to send someone out to the site to investigate the break or not.”
Load-shedding also impacts the devices used in the network, significantly shortening their usable life span.
“These devices are not meant to turn on and off all the time, this wears them out,” explains Mvungi. “So with a shortened life of our devices we are also seeing more network outages than we otherwise would.”
Many municipalities are struggling with a similar impact on their electrical reticulation systems, which exacerbates the problems.
Scrambling for solutions
Load-shedding is here to stay for the foreseeable future. Eskom recently announced South Africans should brace for another six to 12 months of load-shedding, but many predict it will be much longer. This means solutions need to be found, but it is no small task.
The TENET network operations team have been looking into where it is possible to bypass potential problem sites, to build a fibre network that runs past sites that regularly have power (back up) problems or other issues, so those areas at least won’t affect the connectivity of the other sites on the ring. Where possible, TENET has been putting in place infrastructure to bypass sites.
“To bypass a site we need to send a technician to the area to connect up the actual fibre,” explains Mvungi. “But more importantly, there needs to be additional fibre to allow the bypass. Not all metropolitan areas have this extra infrastructure. In many of the rings where we see major problems there is only one fibre provider which means alternate solutions and routes require more expensive, custom-built solutions that are often unaffordable.”
One potential solution Mvungi is positive about however is installing monitoring devices into the TENET/ SANReN devices to directly communicate with the team whether a device is down because of a power cut, or if there is another problem that an engineer needs to see to. “This at least should be able to shorten our response time if the issue is something we are able to fix,” she says.
TENET have also tried putting down batteries, uninterrupted power supply (UPS), down in certain key sites where it is needed. But in periods of frequent load-shedding the power goes down so frequently and for so long, the batteries drain and do not have time to charge again. There are also major problems with theft of batteries during the blackouts.
“Given the size and scope of the network it is just not feasible or sustainable to put in alternative power solutions for every device.” explains Mvungi. “While the institutions who host nodes on the network infrastructure do have contractual obligations to keep that infrastructure operational, there are also legitimate reasons why some are unable to do so,” she says. “TENET has done what they can to pull out reserve funds and support such institutions where needed, but the challenge is a large one.”
Mvungi notes they also do need to factor in the disparities among institutions.
“TENET is also investigating options of working more carefully to balance requirements of sites where infrastructure is hosted with their ability to keep the infrastructure running. Different expectations of the facilities will depend on if the site is only connected to one other site or if it connects two or more.”
In the spirit of collegiality, it may also be possible for institutions to help each other. For example when an institution that does have the skills and resources to keep running is dependent on another that doesn’t, it might make sense for them to lend their colleagues a helping hand. TENET is aware of at least one university funding power solutions at another simply to keep their own link up.
Planning for the future
Just as TENET was established because universities and research institutions recognised the service they needed went beyond what one or even a handful of institutions were able to do by themselves, load-shedding’s impact on the SANReN network is an issue that needs a collective response.
“We need to find creative ways to minimise the impact of load-shedding on the research and institutions’ teaching and learning activities,” says Mvungi. “TENET will continue to engage with our partners. Key to successfully weathering the load-shedding storm is to remember we are all in the same boat and we need to seek and find solutions for the collective good.”
