SEACOM Outage: 5 July 2010 - 23 July
Document Actions
Current status: service restored at 06h30 23 July
Timeline:
5 July:
1) At 11h20 all TENET institutions experienced a break in international traffic.
2) At 11h49 TENET issued the following REN-news announcement:
A fault on SEACOM has caused a complete failure of international bandwidth to all TENET sites. The nature of the fault is not yet known, though it is not suspected to be a cable break; nor is the estimated time to repair. As soon as further information is available it will be shared on this list.
3) At 16h02 TENET issued a further REN-news announcement:
SEACOM report that the fault is between Mumbai and Mombasa, and that a repeater has failed. More information is expected shortly.
4) At 21h55 TENET posted to REN-news a customer announcement from SEACOM, reading as follows:
At 09:19 GMT, 5 July 2010, SEACOM experienced a submarine failure resulting in service downtime between Mumbai and Mombasa. Current investigations indicate that a repeater has failed on segment 9 of the SEACOM cable, which is offshore to the north of Mombasa. This unexpected failure affects traffic towards both India and Europe. Traffic within Africa is not affected.
SEACOM has initiated emergency repair procedures to replace the repeater. Once mobilised, the repair ship is deployed to the location of the fault to pick up the cable. The cable is then brought on board to undergo the repair – the faulty element is replaced with a new repeater - before being put back in the water.
Whilst the repair process itself will only take a few hours, the overall process may last a minimum of 6-8 days. The actual duration is unpredictable due to external factors such as transit time of the ship, weather conditions and time to locate the cable. For this reason, the estimated duration of this repair remains uncertain.
SEACOM, in co-operation with individual clients, is actively seeking alternatives to restore service whilst the repairs are undertaken.
SEACOM has initiated emergency repair procedures to replace the repeater. Once mobilised, the repair ship is deployed to the location of the fault to pick up the cable. The cable is then brought on board to undergo the repair – the faulty element is replaced with a new repeater - before being put back in the water.
Whilst the repair process itself will only take a few hours, the overall process may last a minimum of 6-8 days. The actual duration is unpredictable due to external factors such as transit time of the ship, weather conditions and time to locate the cable. For this reason, the estimated duration of this repair remains uncertain.
SEACOM, in co-operation with individual clients, is actively seeking alternatives to restore service whilst the repairs are undertaken.
5) Two SMS announcements were also issued on the afternoon of 5 July summarising the information available to that point.
6 July:
6) At 07h27 TENET issued the following REN-news announcement:
While we have no official news on this yet, it seems that SEACOM has managed to restore some connectivity to the cable system. Things have been fairly intermittent and it remains to be seen how long before the system is stabilized, however, at around 12:53am this morning we saw traffic restore on the network. This then went down again at around 1:50am and was again restored at 3:05am. This was followed by a series of outages where the system was up and down till around 6:10am this morning when it last cleared and came up.
Currently the system is up and working as of the time of sending this update.
We will send out more information as soon as we get it.
Currently the system is up and working as of the time of sending this update.
We will send out more information as soon as we get it.
7) At 10h30 international connectivity was lost once more. TENET issued the following REN-news announcement:
Unfortunately SEACOM has gone down again, we are attempting to verify what is actually going on, but this is a rather difficult situation since no one could explain why we were up in the first place. Hopefully they manage to get it back up and working again.
We will send out further updates as soon as we get some
We will send out further updates as soon as we get some
8) At 13h05 TENET issued a management advisory to IT directors of institutions, warning them to prepare to operate under a degraded service that would follow the procurement of alternative international bandwidth, unavoidably at a lower capacity than the SEACOM circuit.
Our Seacom circuit is up again and has been stable for about an hour and a half. As per the other announcements, we currently have no indication that this will remain stable or how long it will remain but, but we will continue to monitor the situation.
7 July:
10) Service was lost again at about 02h00 the following morning, and resumed at 05h00 on 7 July.
11) Two short breaks in SEACOM service have taken place since the last update (10h20-10h45 and 13h30-13h35).
12) Service failed again at 14h20 and resumed at 15h30.
13) Service failed again at about 17h00.
8 July:
14) In the course of the previous two days, TENET entered into agreements with other South African ISPs for use of some emergency capacity when it became available. At around 16h30 the first such capacity was released to TENET. At 17h52 TENET sent the following announcement to REN-news:
We have managed to bring up some extremely limited international bandwidth. Currently, this is only 100mbit/second pipe for the entire network so things will be extremely slow, but the connectivity is there.
We appeal to everyone to be cognizant of their usage during this period and limit usage to business critical applications for the benefit of the entire community.
We hope to have some further bandwidth up by tomorrow morning, but this is dependent on a number of beyond our control, however, as we expand the capacity we will send out further updates.
We appeal to everyone to be cognizant of their usage during this period and limit usage to business critical applications for the benefit of the entire community.
We hope to have some further bandwidth up by tomorrow morning, but this is dependent on a number of beyond our control, however, as we expand the capacity we will send out further updates.
This was supplemented later in the evening by a further 600 Mbps. At 23h30 TENET issued the following further REN-news announcement:
We have brought some more emergency capacity online onto the network, and now have just over half a gigabit of international capacity available to us at this time. This represents about 5/13ths of our normal peak utilization bandwidth, so while things are still going to be fairly tight in terms of bandwidth, it should be reasonably useable.
I need to stress that there may be some additional work needed to ensure the load balancing is 100% between the various sources, so there may be some optimization work done tomorrow when we can do a proper analysis during day time usage periods as to how the loads are looking. This should not have any direct impact on availability of service however.
I need to stress that there may be some additional work needed to ensure the load balancing is 100% between the various sources, so there may be some optimization work done tomorrow when we can do a proper analysis during day time usage periods as to how the loads are looking. This should not have any direct impact on availability of service however.
Once again, TENET would like to thank all of you for your patience during this trying time, we fully realize the frustration levels during an event like this and will continue to endeavor in every way to minimize the impact this is having on the services we provide. We also once again appeal to the community to be cognizant of the limited amount of capacity available and to be circumspect with bandwidth usage to ensure a better experience for everyone while we run on a degraded service.
9 July:
15) Service on the temporary bandwidth appears to have been stable and useable. At 11h10 TENET issued the following REN-news announcement:
A brief update.
We have had confirmation from SEACOM that the repair ship is on route to the cable break, though there is currently no estimated time of arrival or repair. The break point itself has been pinpointed to an exact location however.
SEACOM has also informed us that the break is in extremely deep water, and this could cause some slight delay in the repair process (the failed repeater is at a depth of around 4700 meters below sea level)
With regards to our current bandwidth situation, we seem to be running at reasonable traffic loads and performance at this point is looking reasonable, though we are keeping an extremely close eye on utilization over the limited capacity. To all the institutions who have assisted us by locking down non-essential traffic or letting us perform this function on their behalf where the capabilities were not available on site, our sincere thanks.
Once again, please rest assured that we are monitoring the situation on a constant basis and will do all in our power to ensure that the service provided during this time is of as high a standard as we can possibly achieve.
I also at this point need to extend a sincere thank you to the partners we are working with in the industry for their assistance. It has amazed me to see just how much is being achieved right now through the collaboration of the various entities, and it is proof in my mind that in times of crisis, the industry has shown an amazing resilience and an ability to band together to improve the situation for all concerned.
We have had confirmation from SEACOM that the repair ship is on route to the cable break, though there is currently no estimated time of arrival or repair. The break point itself has been pinpointed to an exact location however.
SEACOM has also informed us that the break is in extremely deep water, and this could cause some slight delay in the repair process (the failed repeater is at a depth of around 4700 meters below sea level)
With regards to our current bandwidth situation, we seem to be running at reasonable traffic loads and performance at this point is looking reasonable, though we are keeping an extremely close eye on utilization over the limited capacity. To all the institutions who have assisted us by locking down non-essential traffic or letting us perform this function on their behalf where the capabilities were not available on site, our sincere thanks.
Once again, please rest assured that we are monitoring the situation on a constant basis and will do all in our power to ensure that the service provided during this time is of as high a standard as we can possibly achieve.
I also at this point need to extend a sincere thank you to the partners we are working with in the industry for their assistance. It has amazed me to see just how much is being achieved right now through the collaboration of the various entities, and it is proof in my mind that in times of crisis, the industry has shown an amazing resilience and an ability to band together to improve the situation for all concerned.
18 July:
16) TENET secured an additional 400 Mbps of temporary bandwidth for use between 07h00 and 17h30 on weekdays only. This brought the total temporary international bandwidth to 1.15 Gbps, which is very close to normal daytime utilisation. On the basis of this, institutions were advised to consider relaxing self-imposed bandwidth restrictions.
22 July:
17) At 12h40 SEACOM advised as follows:
SEACOM is pleased to confirm that the physical repairs to the fault on the submarine cable are in the final stages of completion with the entire system currently undergoing testing before the cable is lowered back into the water. Our technical teams are actively working with all customers to reinstate their SEACOM traffic to pre-outage configurations and all connectivity going out of Africa is expected to be fully restored on 23 July 2010.
23 July:
SEACOM reported that the cable repair activity completed successfully. TENET institutions' international traffic is running once more on SEACOM, and the service is being closely monitored.
Current position: