tag:status.vorco.net,2005:/historyVorco Status - Incident History2024-03-29T20:53:11+13:00Vorcotag:status.vorco.net,2005:Incident/191444192023-11-17T16:44:09+13:002023-11-23T16:14:27+13:00Unplanned Event - International Connectivity<p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>16:44</var> NZDT</small><br><strong>Resolved</strong> - This was resolved at 3:04pm. We'll update this incident with a root cause when available.</p><p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>15:30</var> NZDT</small><br><strong>Update</strong> - We've confirmed this is due to an upstream transit fault which for the time being has resolved itself. We're awaiting confirmation on whether a permanent fix is in place and will update this incident when we receive more details.</p><p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>14:34</var> NZDT</small><br><strong>Identified</strong> - Start Time: 1434<br />Service(s) Affected: International internet connectivity<br />Customer Impact: Customers may be unable to reach websites and services hosted internationally, particularly those that route via the USA. This appears to be a fault upstream and we are liaising with our upstream transit for more information. Connectivity to Australia appears to be working still.<br /><br />We apologise for any inconvenience this may cause.</p>tag:status.vorco.net,2005:Incident/187683282023-10-13T10:08:33+13:002023-10-13T10:10:21+13:00Unplanned Event - SkyPhone Outage<p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>10:08</var> NZDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>09:06</var> NZDT</small><br><strong>Update</strong> - Services should be fully restored, but sometimes phones may not automatically re-register with the network. If your phone is still not working, please power cycle it by removing the power and network from the phone for a moment, then plugging back in.</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>08:11</var> NZDT</small><br><strong>Monitoring</strong> - Power has been restored and checks are being done by engineers to confirm services are working as expected again.</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>07:54</var> NZDT</small><br><strong>Update</strong> - Electricians are on-site working to resolve power issues. No ETR at this time but this is being treated with the highest priority.</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>06:45</var> NZDT</small><br><strong>Identified</strong> - Start Time: 0645<br />Service(s) Affected: SkyPhone Standard<br />Customer Impact: Due to a datacentre power fault SkyPhone handsets will be unable to register and make calls.<br /><br />We apologise for any inconvenience this may cause.</p>tag:status.vorco.net,2005:Incident/182431932023-09-05T06:00:06+12:002023-09-05T06:00:06+12:00Planned Event - AKL4 Core Router Software and Firmware Upgrades<p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>06:00</var> NZST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'> 4</var>, <var data-var='time'>23:00</var> NZST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>20:16</var> NZST</small><br><strong>Scheduled</strong> - Service(s) Affected: Internet access<br />Customer Impact: Customers on single homed internet services will experience approximately 30 minutes downtime, twice during the outage window.<br />Details: We need to upgrade software and firmware on core routers which requires multiple full chassis reboots. Customers who are multi-homed (e.g. Chorus + Vector fibre) will only experience brief impact while BGP reconverges to our AKL2 site. Customers who are single-homed to AKL4 (most Chorus fibre) will be impacted for the duration of each reboot.<br /><br />We apologise for any inconvenience this may cause.</p>tag:status.vorco.net,2005:Incident/182430782023-09-05T06:00:05+12:002023-09-05T06:00:05+12:00Planned Event - Chorus Handover Migration<p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>06:00</var> NZST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'> 4</var>, <var data-var='time'>23:00</var> NZST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>20:02</var> NZST</small><br><strong>Scheduled</strong> - Service(s) Affected: Chorus fibre services<br />Customer Impact: Customers on Chorus fibre services will experience approximately 30 minutes downtime during the outage window.<br />Details: Chorus are migrating our handover (and 15 others) to new equipment. This requires physical re-patching as well as migration of configuration on Chorus' side.<br /><br />We apologise for any inconvenience this may cause.</p>tag:status.vorco.net,2005:Incident/163928722023-03-31T12:30:13+13:002023-03-31T12:30:13+13:00Planned Event - Enable Christchurch Fibre Handover Migration<p><small>Mar <var data-var='date'>31</var>, <var data-var='time'>12:30</var> NZDT</small><br><strong>Completed</strong> - Enable have had to cancel this work and will reschedule it at a later time.</p><p><small>Mar <var data-var='date'> 9</var>, <var data-var='time'>09:41</var> NZDT</small><br><strong>Scheduled</strong> - Service(s) Affected: Enable fibre services in Christchurch<br />Customer Impact: Loss of service for 20-30 minutes during the outage window<br />Details: Enable are replacing core network equipment and need to cut over Vorco's services to their new core equipment.<br /><br />We apologise for any inconvenience this may cause.</p>tag:status.vorco.net,2005:Incident/158146842023-01-11T14:15:40+13:002023-01-11T14:48:49+13:00Unplanned Event - National Backhaul Issues<p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>14:15</var> NZDT</small><br><strong>Resolved</strong> - Backhaul vendor has advised that remedial actions to restore stability have been undertaken and our monitoring is reflecting the same.</p><p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>13:25</var> NZDT</small><br><strong>Identified</strong> - We've confirmed there is a backhaul vendor fault and are awaiting further details on the nature of the fault/estimated time to repair.</p><p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>12:40</var> NZDT</small><br><strong>Investigating</strong> - Start Time: 1240<br />Service(s) Affected: Internet Connectivity<br />Customer Impact: Customers outside of Auckland on non-Chorus fibre services appear to be flapping. We're looking into this with the highest priority and will provide a further update soon.<br /><br />We apologise for any inconvenience this may cause.</p>tag:status.vorco.net,2005:Incident/110986782022-09-13T04:15:44+12:002022-09-13T04:15:44+12:00Planned Work - Urgent Software Updates<p><small>Sep <var data-var='date'>13</var>, <var data-var='time'>04:15</var> NZST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'>13</var>, <var data-var='time'>00:15</var> NZST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>11</var>, <var data-var='time'>14:49</var> NZST</small><br><strong>Scheduled</strong> - We need to urgently install software updates on equipment at our AKL2 & AKL3 sites to patch a software bug causing incorrect data to be provided to our network monitoring system.<br /><br />During the maintenance window several equipment restarts will take place which will interrupt service for between 15 and 40 minutes at a time.</p>tag:status.vorco.net,2005:Incident/108944502022-08-24T22:32:09+12:002022-08-24T22:32:09+12:00International Transit Degradation to Australia and Southeast Asia<p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>22:32</var> NZST</small><br><strong>Resolved</strong> - Primary Southern Cross Cable connectivity has been restored.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>20:31</var> NZST</small><br><strong>Update</strong> - Southern Cross Cables have revised the estimated repair time to 10pm. Our upstream international transit has advised they are running very close to capacity and to expect degraded international performance until the evening peak period passes.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>17:28</var> NZST</small><br><strong>Update</strong> - We've been advised by our upstream transit they are running close to maximum capacity on the failover path and traffic to international destinations may begin to experience packet loss as the evening peak period approaches. Emergency changes are being made to offload traffic as much as possible and to bring up an additional link to Australia.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>14:50</var> NZST</small><br><strong>Update</strong> - Southern Cross Cables have advised that repair work is now underway and it will take at least another five hours to complete. Estimated time of repair 8pm NZ time.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>14:22</var> NZST</small><br><strong>Update</strong> - Southern Cross Cables have advised corrosion has been found in a joint box and it will need to be replaced. Repair work is commencing. No ETA for completion at this time.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>12:41</var> NZST</small><br><strong>Update</strong> - Southern Cross Cables have confirmed a break in the cable approx. 1.7km from the Sydney landing station. No ETA for repair yet.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>10:20</var> NZST</small><br><strong>Update</strong> - We have been advised there is a possible submarine cable break in the vicinity of the Alexandria landing station and all of Southern Cross segment A is impacted as a result.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>08:04</var> NZST</small><br><strong>Monitoring</strong> - We observed a loss of connectivity to some international locations (Australia and Southeast Asia) between 8:04am and 8:10am. Since connectivity was restored, we are seeing higher latency to those parts of the world.<br /><br />We have since been advised this is due to an unplanned outage in an upstream vendor's network and the loss of connectivity was due to failover occurring after losing several core links between NZ and Australia.<br /><br />The higher latency is due to the failover path between NZ and Australia being longer, and latency will return to normal once the shorter primary path is restored.</p>tag:status.vorco.net,2005:Incident/105735272022-08-03T06:00:33+12:002022-08-03T06:00:33+12:00Planned Work - Transit Maintenance<p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>06:00</var> NZST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>00:00</var> NZST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>15:31</var> NZST</small><br><strong>Scheduled</strong> - Our upstream international transit will be undertaking maintenance which will result in a loss of international connectivity at one of our sites (AKL2) for approximately 30 minutes during the maintenance window.<br /><br />We have redundant international connectivity, however BGP routing reconvergence will occur which will cause packet loss and service impact as international connectivity shifts to backup paths, and then again when international connectivity shifts back to primary paths.</p>tag:status.vorco.net,2005:Incident/106573992022-07-21T19:46:37+12:002022-07-21T19:46:37+12:00Inbound calling to Microsoft Teams intermittent<p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>19:46</var> NZST</small><br><strong>Resolved</strong> - Microsoft have advised Teams is now operational.</p><p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>15:51</var> NZST</small><br><strong>Monitoring</strong> - We're now seeing calls connect quickly again. We've moved this incident to monitoring and will keep it open until Microsoft close out their incident.</p><p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>15:43</var> NZST</small><br><strong>Update</strong> - We're seeing calls succeeding now, although they may appear to ring for up to a minute as the Microsoft Teams service is still responding slowly and not accepting calls immediately. Microsoft 365 administrators can follow updates from Microsoft directly here: https://admin.microsoft.com/Adminportal/Home#/servicehealth/:/alerts/TM402718</p><p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>14:57</var> NZST</small><br><strong>Identified</strong> - This is not a Vorco fault. Microsoft Teams is rejecting most inbound calls. We do see a small number of calls succeeding but large numbers being rejected. Microsoft support have advised us this is related to the global Teams outage earlier today and we will continue to follow up regularly for updates and update this incident as new information comes to hand.</p><p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>14:31</var> NZST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>13:20</var> NZST</small><br><strong>Investigating</strong> - Inbound calls to Microsoft Teams are not always connecting presently. We are investigating the cause.</p>tag:status.vorco.net,2005:Incident/103641312022-06-21T14:35:24+12:002022-06-21T15:18:04+12:00Unplanned Outage - Backhaul to Northpower Whangarei<p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>14:35</var> NZST</small><br><strong>Resolved</strong> - Backhaul between Auckland and Northpower Fibre in Whangarei has now been restored. We are awaiting a reason for outage and will provide this upon request when available.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>14:34</var> NZST</small><br><strong>Monitoring</strong> - The backhaul vendor has rolled back a change and service has been restored. We are monitoring the service.<br /><br />If your service has not been restored please restart your router, then contact Vorco Support on 0800 486 726 if it doesn't come back online.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>13:42</var> NZST</small><br><strong>Update</strong> - Backhaul vendor engineers have identified where the fault is and are working on it now. We're expecting more information soon regarding the nature of the fault and estimated time to repair.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>13:20</var> NZST</small><br><strong>Identified</strong> - We've lost connectivity between Auckland and Northpower Fibre in Whangarei. We're investigating now and will provide an update soon.</p>tag:status.vorco.net,2005:Incident/102579142022-06-21T06:00:08+12:002022-06-21T06:00:08+12:00Planned Work - Transit Maintenance<p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>06:00</var> NZST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>00:00</var> NZST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>16:31</var> NZST</small><br><strong>Scheduled</strong> - Our upstream international transit will be undertaking maintenance which will result in a loss of international connectivity at one of our sites (AKL4) for approximately 30 minutes during the maintenance window.<br /><br />We have redundant international connectivity, however BGP routing reconvergence will occur which will cause packet loss and service impact as international connectivity shifts to backup paths, and then again when international connectivity shifts back to primary paths.</p>tag:status.vorco.net,2005:Incident/98543432022-05-05T04:15:23+12:002022-05-05T04:15:23+12:00Planned Work - Software Upgrades - Night 3<p><small>May <var data-var='date'> 5</var>, <var data-var='time'>04:15</var> NZST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'> 5</var>, <var data-var='time'>00:15</var> NZST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>14:19</var> NZST</small><br><strong>Scheduled</strong> - We need to install updated software across our core network devices to ensure our network remains up to date, secure and covered by vendor support. Multiple devices will be restarted to apply updates during the maintenance window. Each restart will impact service for between 5 minutes and 40 minutes.</p>tag:status.vorco.net,2005:Incident/98543292022-05-04T04:15:22+12:002022-05-04T04:15:22+12:00Planned Work - Software Upgrades - Night 2<p><small>May <var data-var='date'> 4</var>, <var data-var='time'>04:15</var> NZST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'> 4</var>, <var data-var='time'>00:15</var> NZST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>14:16</var> NZST</small><br><strong>Scheduled</strong> - We need to install updated software across our core network devices to ensure our network remains up to date, secure and covered by vendor support. Multiple devices will be restarted to apply updates during the maintenance window. Each restart will impact service for between 5 minutes and 40 minutes.</p>tag:status.vorco.net,2005:Incident/98543222022-05-03T03:25:09+12:002022-05-03T03:25:09+12:00Planned Work - Software Upgrades - Night 1<p><small>May <var data-var='date'> 3</var>, <var data-var='time'>03:25</var> NZST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'> 3</var>, <var data-var='time'>00:15</var> NZST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>14:14</var> NZST</small><br><strong>Scheduled</strong> - We need to install updated software across our core network devices to ensure our network remains up to date, secure and covered by vendor support. Multiple devices will be restarted to apply updates during the maintenance window. Each restart will impact service for between 5 minutes and 40 minutes.</p>tag:status.vorco.net,2005:Incident/98540972022-04-26T20:11:36+12:002022-04-26T20:11:36+12:00Planned Work - Megaport Auckland Peering<p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>20:11</var> NZST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>19:32</var> NZST</small><br><strong>Update</strong> - Scheduled maintenance is still in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>13:18</var> NZST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>13:17</var> NZST</small><br><strong>Scheduled</strong> - We're carrying out maintenance on the equipment facing our Megaport Auckland peering. Peering sessions will be shut down while maintenance is undertaken. No user impact is expected as traffic will be re-routed via other paths.</p>tag:status.vorco.net,2005:Incident/94400842022-03-02T14:02:26+13:002022-03-02T14:42:33+13:00National backhaul outage<p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>14:02</var> NZDT</small><br><strong>Resolved</strong> - We've been advised the fault is fixed as of 14:02. Root cause analysis is ongoing.</p><p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>14:01</var> NZDT</small><br><strong>Update</strong> - We're seeing traffic returning on our Hamilton and Dunedin backhaul paths now. No vendor confirmation yet.</p><p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>13:54</var> NZDT</small><br><strong>Identified</strong> - We've narrowed this down to specifically our backhaul paths to Hamilton and Dunedin. Our other backhaul areas (Whangarei, Wellington, Christchurch) have normal levels of traffic. We're working with the backhaul vendor now.</p><p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>13:44</var> NZDT</small><br><strong>Investigating</strong> - We are investigating an issue with national backhaul impacting fibre services outside of Auckland.</p>tag:status.vorco.net,2005:Incident/94229702022-02-28T16:38:42+13:002022-02-28T16:38:42+13:00Partial DNS resolution issues<p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>16:38</var> NZDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>12:00</var> NZDT</small><br><strong>Monitoring</strong> - Due to an outage on the Southern Cross cable between Auckland and Sydney one of our transits lost connectivity for a short while and then became congested due to re-routing around the outage which led to some websites and DNS requests not working as expected, particularly those hosted in Australia and Singapore.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>10:33</var> NZDT</small><br><strong>Investigating</strong> - We are investigating reports of some DNS hosts not resolving and issues with connectivity to some websites.</p>tag:status.vorco.net,2005:Incident/94020752022-02-25T10:17:54+13:002022-02-25T10:30:06+13:00Northpower Fibre Services Down<p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>10:17</var> NZDT</small><br><strong>Resolved</strong> - We have received confirmation the fault has been repaired and traffic has returned to normal levels as of 10:17am.</p><p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>09:57</var> NZDT</small><br><strong>Update</strong> - We've received an update to confirm the Whangarei side of the Auckland-Whangarei backhaul path is currently isolated and fibre technicians are on route to repair.</p><p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>09:40</var> NZDT</small><br><strong>Identified</strong> - Our backhaul from Whangarei to Auckland is up but delivering no traffic. The carrier that operates the Whangarei end has confirmed they have received multiple reports of a similar nature and are investigating this now.</p><p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>09:20</var> NZDT</small><br><strong>Investigating</strong> - We are aware of a fault affecting UFB customers in Northland and are investigating the cause.</p>tag:status.vorco.net,2005:Incident/85858212021-11-22T16:30:12+13:002021-11-25T20:28:47+13:00Slow customer portal and SkyPhone provisioning<p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>16:30</var> NZDT</small><br><strong>Resolved</strong> - The DDoS has subsided so we are closing this.</p><p><small>Nov <var data-var='date'>18</var>, <var data-var='time'>11:30</var> NZDT</small><br><strong>Monitoring</strong> - An ongoing DDoS attack is impacting access to our customer portal, voice API's and SkyPhone provisioning (including making changes to handsets/users). The attack is being actively mitigated but changes will continue to be slow or need multiple attempts until the attack ceases.</p>tag:status.vorco.net,2005:Incident/86536132021-11-21T23:25:24+13:002021-11-21T23:29:45+13:00Loss of Vector Fibre services<p><small>Nov <var data-var='date'>21</var>, <var data-var='time'>23:25</var> NZDT</small><br><strong>Resolved</strong> - Handover port restored. We will follow up with Vector to ensure notifications are received in future regarding this port and any planned maintenance.</p><p><small>Nov <var data-var='date'>21</var>, <var data-var='time'>23:25</var> NZDT</small><br><strong>Identified</strong> - This handover port is down due to unnotified planned maintenance work that was meant to cover specific customer circuit ID's only. Vector are investigating how this got missed but we're expecting service restoration in the next 15 minutes.</p><p><small>Nov <var data-var='date'>21</var>, <var data-var='time'>23:04</var> NZDT</small><br><strong>Investigating</strong> - Our Vector handover port has gone down, we're investigating.</p>tag:status.vorco.net,2005:Incident/82152662021-10-13T16:43:28+13:002021-10-13T16:46:37+13:00Wellington and South Island connectivity down<p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>16:43</var> NZDT</small><br><strong>Resolved</strong> - The card has been replaced and we are seeing services restoring as of 4:43pm</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>16:38</var> NZDT</small><br><strong>Update</strong> - Replacement parts have arrived on site and technicians are preparing to replace the failed card and re-patch.</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>16:03</var> NZDT</small><br><strong>Identified</strong> - We've confirmed with our backhaul vendor that a line card has failed in Albany. Parts have been dispatched already and are expected on-site in the next 60 minutes, plus another 60 minutes to swap the failed card and re-patch all ports. We are estimating restoration of service by 6pm.</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>15:28</var> NZDT</small><br><strong>Investigating</strong> - We've lost backhaul connectivity for customers on Chorus fibre services in the lower North Island and customers on Chorus fibre services in the South Island. This is not affecting other fibre services in those areas.</p>tag:status.vorco.net,2005:Incident/79053272021-09-03T23:33:05+12:002021-09-03T23:33:05+12:00SkyPhone phone registration and calling issues<p><small>Sep <var data-var='date'> 3</var>, <var data-var='time'>23:33</var> NZST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'> 3</var>, <var data-var='time'>16:32</var> NZST</small><br><strong>Monitoring</strong> - Full service is restored now. The DDoS attack is ongoing but being scrubbed. We will continue to monitor the situation closely.</p><p><small>Sep <var data-var='date'> 3</var>, <var data-var='time'>16:07</var> NZST</small><br><strong>Update</strong> - Service is partially restored but is under an ongoing DDoS attack which is preventing full restoration. Work is ongoing to fully restore service.</p><p><small>Sep <var data-var='date'> 3</var>, <var data-var='time'>13:08</var> NZST</small><br><strong>Update</strong> - We're seeing some restoration but also receiving reports that devices are showing "URL calling disabled". This is still being treated with the highest priority.</p><p><small>Sep <var data-var='date'> 3</var>, <var data-var='time'>10:17</var> NZST</small><br><strong>Identified</strong> - The fault has been identified as being with an upstream provider, we have lodged a ticket with them and they have advised their team is working on it. No ETA is available at present. <br /><br />We will be keeping in close contact with the upstream provider to request updates and will post more information as soon as it comes available.<br /><br />Due to the nature of the fault, we are unable to forward calls to mobiles for affected users as the inbound calls are not reaching the point in the system where the forwarding occurs.</p><p><small>Sep <var data-var='date'> 3</var>, <var data-var='time'>08:51</var> NZST</small><br><strong>Investigating</strong> - Multiple customers have reported that some of their inbound phone numbers are not connecting. The fault is being investigated by the engineering team.</p>tag:status.vorco.net,2005:Incident/79066282021-09-03T15:41:23+12:002021-09-03T15:52:27+12:00IPv4 international connectivity issues<p><small>Sep <var data-var='date'> 3</var>, <var data-var='time'>15:41</var> NZST</small><br><strong>Resolved</strong> - This incident is resolved. The cause was a DDoS attack against one of our international transit providers. Domestic traffic and peering was not affected, and IPv6 international transit was not affected.</p><p><small>Sep <var data-var='date'> 3</var>, <var data-var='time'>15:11</var> NZST</small><br><strong>Monitoring</strong> - Our AKL2 site now has international connectivity again and both paths are restored. We're leaving this incident open for the time being until we confirm stability.</p><p><small>Sep <var data-var='date'> 3</var>, <var data-var='time'>14:55</var> NZST</small><br><strong>Update</strong> - We've implemented a workaround to re-route all traffic via our AKL4 site and all customers should now have connectivity again. We're continuing to work on getting AKL2 international connectivity restored.</p><p><small>Sep <var data-var='date'> 3</var>, <var data-var='time'>14:13</var> NZST</small><br><strong>Update</strong> - International transit to our AKL4 site has returned but there is still no connectivity to our AKL2 site.</p><p><small>Sep <var data-var='date'> 3</var>, <var data-var='time'>13:22</var> NZST</small><br><strong>Identified</strong> - We have lost our international transit IPv4 BGP sessions. This has been confirmed as a fault on the upstream side.<br /><br />IPv6 is working, and our other IPv4 transit/peering sessions are working.</p>tag:status.vorco.net,2005:Incident/72696642021-06-18T01:05:20+12:002021-06-18T01:05:20+12:00Emergency Maintenance - AKL4 Software Updates<p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>01:05</var> NZST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>00:03</var> NZST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>17</var>, <var data-var='time'>20:43</var> NZST</small><br><strong>Scheduled</strong> - We need to deploy software upgrades to try and effect a permanent resolution to the software faults encountered earlier today. The software upgrades will involve multiple outages of around 15-minutes as equipment reloads.</p>