A joint operation by Telstra Business Technology Solutions, Mangano IT and Datto to deploy backup and disaster recovery across a Queensland council’s 64 servers kicked into overdrive earlier this year, when Cyclone Debbie was forecast to hit the mainland right near the council’s main site.
The region regularly faces intense weather, which prompted the Whitsunday Regional Council to reassess its IT environment. Telstra BTS won a tender process and proposed a Datto solution, supplied by Mangano IT, for the council's headquarters at Proserpine and a Bowan site, with work taking place in March.
“The Datto solution at Proserpine is a physical device; it's designed to capture all the servers locally at that site,” Mangano IT managing director Paul Mangano said.
“At the Bowen site, we used a virtualised Datto appliance which lives on the client’s infrastructure. This captures all of the servers over at the Bowen site.”
Work was tracking along nicely when a cyclone warning sounded across the region.
Forming as a tropical low on 23 March before intensifying to a cyclone on 26 March, Cyclone Debbie made landfall near Airlie Beach, north of Proserpine, on 28 March. At the time, ABC News reported the cyclone was producing destructive winds of 120kph and gusts of up to 165kph.
Mangano said there was a "mad rush" to get the project finished in time.
“Telstra did their bit by fattening the link from the client site up to the internet, the Datto team made their support guys all available for us, and we were working frantically to ensure that all the last issues were resolved in our project. And everything got done in time before disaster struck.”
In the days leading up to the storm’s landfall, the partners held twice-daily meetings, including Datto’s US support team, to make hour-to-hour decisions on when the final backup would be sent and when a virtualised environment would be switched on.
“Once we ensured that all the servers had replicated to the cloud, we worked with the Datto team and said: ‘Okay, let's break out some scenarios here. So if we need to turn this on, at what time do we stop pushing new backups to the cloud?’,” Mangano said.
With the council sites evacuated and the community bracing for the worst, the team made final preparations and pushed the most recent backup to the cloud, ready for a full replication switch over.
“We got to the point, probably by about Wednesday [29 March] and we'd called it, because by that stage there was obviously very little local data changes happening up at the site at Whitsundays,” Mangano said.
As the cyclone blew through the region, the council’s Proserpine site was all but wrecked, except, astonishingly, the server room. Mangano said the services continued to operate throughout the storm, and no switchover to the cloud was necessary.
“It's just as important to decide when to cut over as when not to cut over,” he said.
“Once you go, it's quite an effort to then come back. So if you push to the cloud unnecessarily, that becomes your live production environment, you're operating out of that environment, you've got to then think, ‘Well, how does all that data then come back?’
“Once the event's over, and everyone's safe and well, you need to bring all the data back, and Datto have quite a good process in doing that. But nothing's an insignificant event when you're dealing with moving around that amount of data.”
Jason Law, national manager for local government at Telstra Business, told CRN the three IT stakeholders delivered a positive outcome in a difficult situation and that work has continued with the council in the wake of the weather event.
"We have been very pleased with the solution to date and although it was not completely in place prior to Cyclone Debbie hitting the region, we were able to work with the Council and put measures in place to help them be as prepared as possible," he said.
"The local Telstra team has helped the council during and immediately after Cyclone Debbie, providing help such as a temporary satellite system to restore vital communication services for the SES, other network changes, loan equipment and temporary services to get the council back up and running as soon as possible.
"We’re currently helping the council move its IT infrastructure to the cloud, update its telephony requirements to an IP system and providing network services to their temporary offices."
Datto strategic accounts manager Peter Eldon told CRN the vendor prided itself on working with partners who cared about customer data, and that Mangano IT and Telstra were prime examples of that kind of partnership.
"From the moment the threat was assessed, Datto, Mangano IT, Telstra and the council worked together as a team toward one goal – no loss of data and minimal downtime for the council. Lead by Mangano principals Paul and Robert Mangano, plans were formulated, executed, reassessed and executed again," Eldon said.
"This project was a great example of the client, vendor and reseller working together and the Datto team are extremely proud to have been a part of it. Datto’s technology ensured that Whitsunday Regional Council did not lose any critical data, but it was what everyone did around it that made the difference in this disaster situation.”
.png&h=142&w=230&c=1&s=1)
.png&w=100&c=1&s=0)
