Oops! Tech error wipes out Alaska info

Cthippo · 21 Mar 2007

Associated Press said:

JUNEAU, Alaska - Perhaps you know that sinking feeling when a single keystroke accidentally destroys hours of work. Now imagine wiping out a disk drive containing information for an account worth $38 billion.

That's what happened to a computer technician reformatting a disk drive at the Alaska Department of Revenue. While doing routine maintenance work, the technician accidentally deleted applicant information for an oil-funded account — one of Alaska residents' biggest perks — and mistakenly reformatted the backup drive, as well.

There was still hope, until the department discovered its third line of defense, backup tapes, were unreadable.

"Nobody panicked, but we instantly went into planning for the worst-case scenario," said Permanent Fund Dividend Division Director Amy Skow. The July computer foul-up, which wiped out dividend distribution information for the fund, would end up costing the department more than $200,000.

Over the next few days, as the department, the division and consultants from Microsoft Corp. and Dell Inc. labored to retrieve the data, it became obvious the worst-case scenario was at hand.

Nine months worth of applicant information for the yearly payout from the Alaska Permanent Fund was gone: some 800,000 electronic images that had been painstakingly scanned into the system months earlier, the 2006 paper applications that people had either mailed in or filed over the counter, and supporting documentation such as birth certificates and proof of residence.

And the only backup was the paperwork itself — stored in more than 300 cardboard boxes.

"We had to bring that paper back to the scanning room, and send it through again, and quality control it, and then you have to have a way to link that paper to that person's file," Skow said.

Half a dozen seasonal workers came back to assist the regular division staff, and about 70 people working overtime and weekends re-entered all the lost data by the end of August.

"They were just ready, willing and able to chip in and, in fact, we needed all of them to chip in to get all the paperwork rescanned in a timely manner so that we could meet our obligations to the public," Skow said.

Last October and November, the department met its obligation to the public. A majority of the estimated 600,000 payments for last year's $1,106.96 individual dividends went out on schedule, including those for 28,000 applicants who were still under review when the computer disaster struck.

Former Revenue Commissioner Bill Corbus said no one was ever blamed for the incident.

"Everybody felt very bad about it and we all learned a lesson. There was no witch hunt," Corbus said.

According to department staff, they now have a proven and regularly tested backup and restore procedure.

The department is asking lawmakers to approve a supplemental budget request for $220,700 to cover the excess costs incurred during the six-week recovery effort, including about $128,400 in overtime and $71,800 for computer consultants.

The money would come from the permanent fund earnings, the money earmarked for the dividends. That means recipients could find their next check docked by about 37 cents.
Click to expand...

And you thought you screwed up bad at work

sui_winbolo · 21 Mar 2007

Uh, why couldn't they go past that single format. Recovery programs can go down several format layers and recover data.

Techno-Dann · 21 Mar 2007

It's probably more than just a single format. Knowing the government, "format" probably means a twenty or twenty-five pass wipe with gibberish, followed by a pass of zeroes to clean the drive.

Over the next few days, as the department, the division and consultants from Microsoft Corp. and Dell Inc. labored to retrieve the data, it became obvious the worst-case scenario was at hand.
Click to expand...

I would imagine that they did try whatever data-recovery programs they could find.

Anyway, this is the way things should be done. No eight-month investigation, no special investigator pulling off a witch hunt, no skewering someone random because he mistook a date under oath. They just sat down and solved the problem, without serious disruption.

Firehed · 21 Mar 2007

Not if you do a proper secure erase, which I can only assume people working for the government are going to be preforming.

sui_winbolo · 21 Mar 2007

Never thought about that, thanks for the insight.

Ramble · 21 Mar 2007

So it the fault of the guy? Not a 'tech' error.

DougEdey · 21 Mar 2007

This seems old...

...re-entered all the lost data by the end of August.
Click to expand...

cpemma · 21 Mar 2007

DougEdey said:

This seems old...
Click to expand...

Brought to you by husky mail.

Tulatin · 21 Mar 2007

isn't a $38 Million - then whittled down to $278,000 oopsie something that costs your ass it's job, though?

Log in or Sign up

Oops! Tech error wipes out Alaska info

Cthippo Can't mod my way out of a paper bag

sui_winbolo Giraffe_City

Techno-Dann Disgruntled kumquat

Firehed Why not? I own a domain to match.

sui_winbolo Giraffe_City

Ramble Ginger Nut

DougEdey I pwn all your storage

cpemma Ecky thump

Tulatin The Froggy Poster

Share This Page

Log in or Sign up

Oops! Tech error wipes out Alaska info

Cthippo Can't mod my way out of a paper bag

sui_winbolo Giraffe_City

Techno-Dann Disgruntled kumquat

Firehed Why not? I own a domain to match.

sui_winbolo Giraffe_City

Ramble Ginger Nut

DougEdey I pwn all your storage

cpemma Ecky thump

Tulatin The Froggy Poster

Share This Page

Useful Searches