Who Is

watching out for you?

In today's world you need to understand a few more things...

Who Is

knocking on your virtual front door?

It could be someone down the block or from the Bloc

When your hard drives are 2 meters underwater

image of hard disks in a bag

24-Jul-2014 - Weatherman says, “Slim chance of flooding”

On the night of the June 20, 2013 staff at Calgary City Hall were told flooding might cause a power loss. While nobody expected City Hall to be flooded, the possibility existed.

Kelly Brown is a Telecom Engineer, Voice and Video Solutions for The City of Calgary. As part of the process for managing critical call centers, she called two technicians to camp out in the basement, in case the building lost power. She also notified a backup center in Calgary.

In the City Hall basement were a non-VoIP main Nortel PBX phone switch that served over 12,000 analog or digital lines, LRT, 20+ call centers, workforce management, and voicemail, along with a GUI application for programming the phone switch. 

“The next morning around 8am the techs informed me water was entering the basement,” said Brown. “After alerting our security team, the techs removed the hard drives from the phone switch and main applications, put them in anti-stat bags, and took them upstairs.”

When security staff finally allowed her to go to the first platform half way down the basement staircase, Brown saw the water was about 5ft deep. The next day it was approximately 2 meters or 6ft deep.

“Three days of pumping later, our focus was on restoring basic phone service,” she said. “We weren’t even looking at the applications yet. I didn’t think we could recover anything in the basement, because the water was so dirty and deep.”

One of the techs retrieved the remaining drives and placed them in what he had available, which were standard clear plastic – not anti-static – bags, with paper plates in an attempt to absorb moisture.

There were 49 drives in total – 30 small profile drives and 19 full-size drives, ranging in capacity from 9GB up to 100GB.

On July 4 firstonsite Restoration referred Brown to LWG Consulting, who referred her to Kroll Ontrack for the possibility of recovering data from the waterlogged drives. The first set of drives arrived at Kroll Ontrack July 18.

“In the analysis phase Kroll Ontrack identified whether or not the data was recoverable, which let me decide whether or not we wanted that data recovered,” said Brown. “Of the 16 different systems we did two emergency restorations, and the rest were standard restorations. The folks at Kroll Ontrack were very good about providing detailed reports on every drive, and how many files they found and restored.”

Only one application wasn’t fully recovered. City technicians were able to use backups to restore those minor files that were missing.

The technicians received the first set of restored drives on July 23. Main applications were prioritized, and were up and running within days of receiving the recovered data from Kroll Ontrack.

“The crazy thing is we’re still not fully restored,” Brown said. “We didn’t restore some applications, because we weren’t happy with those applications in the first place.”

Rebuilding city communications required a juggling act between big-picture restoration and business unit restoration. In the process of rebuilding the 12,000 lines, inventive city techs redirected calls from the main city line, pool, parks, and other city facility numbers, to cell phones.

Moving the operation to higher ground required infrastructure updates. From City Hall 40 remote sites were connected via either fiber or T1. All of those had to be pathed to the new location. An alignment to Calgary Parking Authority remains uncompleted.

“We were in the process of making any of our critical applications geographically redundant – so if we lose one facility the other is able to take over immediately,” said Brown. “We also have a new process for backups and rotate the media between locations.”

The incident demonstrates important lessons…
•    Natural disasters occur, so have backup and restoration plans before you need them
•    Even after submersion in dirty water for days, your data is recoverable by those with the right knowledge and experience
•    Dedicated employees can reduce your restoration time

“People were working very long hours,” noted Brown. “It was quite an experience. If it were up to me I’d have skipped 2013, and gone from 2012 directly to 2014.”

Read more about the flood history here.