Well - it would appear the rest of the cosmos follows the trends of this forum too and while backup has been a hot topic here for the last few days, the day I've had today is testament as to why you should.... So I got a call from a company this morning - they are not one of my clients, I have done a patch of consulting for them, and 3 months ago I did them a quote for a server installation as well as quoting for a back up strategy - they decide the server and backup strategy was a bit expensive and decided not to go ahead with the work. When someone calls you back after so long, you know its not because they want to take up your quote, they either want a re-quote for a different spec, or, there is a bit of an emergency, and they didn't know who else to call. Their server wouldnt boot (its not actually a server, they call it their server, but really its just a high spec desktop), so they asked me to go up and see them and see whats up, but they were saying that POST was saying there was no hard disk present - the office is only 5 minutes up the road from mine so I grabbed my site bag and off I trot. I get there, I try and cold boot the box, POST hangs for a while then tells me there is no drive, just as they had said - manager insistent its probably a loose connection in the box. Get a few tools out, open the box, all looks fine, gave everything a wiggle, still same - hangs on POST and doesnt detect disk. Things are starting to look not so good. One thing I have in my site bag is a caddie for just this kind of circumstance, so I get it out, take the disk out of the machine put it in the caddie and hook it up to my laptop. Drive isnt really making too much noise, the platters are spinning but not very fast at all, and thats it. The heads weren't moving, there was no seeking, just very slow idle spinning. Oopsie - it sounds buggered. I told them it's not to worry, I have some spare disks in the office, I'll just go grab one and we can chuck your last image on there and away you go.... .... .... the room went very quiet. THEY HAVE NO BACKUP. none, capish, nada. Baring in mind, this hard disk that has just gone, has the database for all their staff, payroll, students etc. etc. etc. - the machine basically runs a SQL database that does all their business' stuff - pretty much mission critical stuff. and they have no backup in place at all. Apparently 2 weeks ago someone was supposed to backup - but they couldnt be bothered to come in and do it from the office a few doors up, so they didnt. It was only today that they told us that they hadnt even been doing a windows backup. And its very hard to then find the words to use to express to someone how silly that is, when they are paying you for the privilege. So I have had to book the drive in for lab recovery - any idea how much that is? £150 for a 3 day evaluation, where they look at the drive, come back to you with a list of data they should be able to recover - then, for the actual recovery, I have been quoted £1000 - £3500, add to that my time from today, and your starting to wonder how they quantified our previous quote for a new server and backup strategy as being too much money. So yeah - if it's important. back. it. up. As - once it's gone, you can't back it up.
Well, what can I say as a fellow IT professional that can aid this situation... I know... HAH! They deserved it, and had it coming.
I have worked on the Air Conditioning in many large server rooms and I see daily backup of data. Last server room I worked in had about 300 Blade drives or something to that effect. The backup of data was imense to say the least.
I've been backing up since my Sinclair Spectrum tape days. I am surprised anyone even needs to be told that this is a good thing to do, least of all a business. Funny / tragic story!
It's things like this that make me so glad that we have a commvault backup system and a sun sl3000 tape library....a company that doesn't have a proper backup strategy in place almost deserve for these things to happen. Sent from Bittech Android app
And that, Ladies and Gentlemen, is why thinking purely about money with the idea "It'll never happen to me", leaves you in a crying heap in the corner. Backups are a good idea, even if it's just compressing stuff to another drive in the same PC, anything is a start.
Even a basic RAID would have saved them in this case, except they probably wouldn't have bothered to replace the failed drive until too late... (yes, I know RAID != backup, but it's a useful first step, I've never had to do a full recovery in anger due to a bad disk, because I've been able to get a replacement for the RAID and rebuild the data with no down time. But I do have decent, tested backups as well.)
been really lucky at recovering dead hard drives.. long as the board isn't fried you can usually get it running in a carriage long enough to recover a guys data.. seen that a lot with laptop drives but managed to get the data anyhow- usually including the recovery partition which I'll go for afterwards got a desktop seagate to run once by reflowing the board with a heat gun.. it wasn't dead dead- but it's id was coming back with a scrambled id on post.. those clean room guys make so much money.. wish someone would pass me half that for getting their data back xD
Had something similar happen to a local solicitors office. Went to do their end of year roll over and it all fell over. They went to do a restore to find someone had un-pluged the backup drive some 5 months before hand. One thing to have a backup it's another to actually make sure it's working!
Luckily, we have a backup strategy at work, plus at least 3 weekly off-site backups in my office. We're the CAD (Solidworks) department, and we're encouraged to back up our data to laptops & USB sticks so we can work overtime from home.
My brother is supposed to be running backup for our home work servers. He hasn't been by in about three months to do it. The sudden surge of stories like this on here is convincing me to take matters into my own hands.
Think it's like 70% of companies never fully recover from data loss. Don't quote me on that though. Is this company going to consider regular a more regular back up?
Well to add to this, its all well and good having a backup but for the love of god test it! We P2V'd our entire network a little while back, third party pretty much did everything with out documentation etc. Includes a very nice backup setup but again not well documented. Fast foward 7 months, SAN has major issues, causes corruption on the datastores that host the exchange mailbox stores. Attempt to restore from previous days backup, doesnt work. Attempt to restore from previous weeks backup, again doesnt work. Result, exchange has to stay in recovery mode for about a week while the mailboxstore was being fixed. Luckily we lost about 4 hours email between 4AM and 8AM. The sure amount of hell as my boss and his boss kinda gave up and left me to deal with it = priceless. We also found out, something our third party who designed the solution overlooked to tell us, Exchange 2003 is not support by MS when running on VMware. Now I am in charge of the backup. It happens when it should, gets taken offsite correctly and is tested regularly. The last point was a real pain for me to get sign off on as to do it properly with the way with have things setup takes me a total of 30 hours. As it stands you can tell me any single day between now and 28th Nov 2010 and I can restore any one of our servers to that point, inclusing single files. Next step for me is to convince managment to give me the time to get automated testing working! PLEASE TEST YOUR BACKUPS! Nims
Good one Nims, definitely worth remembering kids! Setting up a fake environment for testing helps, so you can do it whenever. Just keep the test VMs powered down when not in use, then click on button and voila, instant copied setup! Very easy to test.
I have a few servers in my office that I throw backups onto occasionally for testing. I have quite a few clients, so, obviously things like this cost them more so they aren't all willing to have their backups tested. But yes, you are right, testing is a must
Margo you'll be pleased I'm taking the advise and am probably about to pickup a HP microserver and some HDDs
