Hey folks, I'm currently working through some issues we're having with our backup. We have a structure of folders that contains approximately 2.4 million TIFF images roughly between 10 and 100k in size. These get backed up regularly with a Full backup on Monday nights and a Diff backup each evening for the rest of the week. The problem is that the throughput on the backup is terrible (60mb/minute to LTO2 tape) Initially we put this down to tape performance but now we've tried staging it to a backup-to-disc file and we STILL get awful throughput. I crunched some numbers with disk performance - quoted seek time on our drives is 4.7ms. 4.7x2400000=11280000ms=11280s = Almost 3.2 HOURS just in SEEKING?!!?!? This is not even allowing for possible fragmentation of files! (Although also not allowing for this being a 6 disc RAID10 as I'm not entirely familiar with the correlation between quantity of drives and seek time) With this massive potential bottleneck I suddenly thought, what about SSDs? On an enterprise scale they cost relatively little. I was thinking maybe 4 128gb SSDs in a RAID10 with a middle-of-the-road SATA RAID controller. At present the data is only ~40gb so space is not an issue. We would still stage the backup to a B2D file on a regular drive so that the tape can do a big sequential read. As I understand it SSD random read -> RAM (Backup software buffer) -> Sequential write on HDD makes this a good solution? What's everyone's thoughts on this?
Okay, here's the deal. First, your disk array. Since you have six drives in RAID 10, that means two identical three-disk RAID 0 arrays. Now, under the most ideal circumstances (read: hooked up to an awesome RAID controller card), this kind of array would give you up to six times the data throughput of a single drive; more likely your best-case scenario is three times. But it also means that in the worst case, your seek time actually triples (one drive has to seek, then the second has to seek, then the third has to seek, in sequence). Now, neither the tripled throughput or the tripled seek time are very likely to happen in most real-world scenarios - but over 2.4m files, yeah, both can happen. Second, your performance issue. Yes, the fact that you're on disks is a part of the problem - but you can only transfer as fast as your weakest link. If you switched to SSDs tomorrow and started backing up to tape, your backup times would still blow. Backing to disk would be faster, but then you're still limited by the performance of that disk. Honestly, you can only speed your backup times so much by improving the frontend before you need to do something about the backend too. Third, reliability. No true enterprise application uses MLC (multi-level cell) SSDs for active data storage, because each individual cell has a lifespan of about 10,000 writes, whereas SLC (single-level cell) has about ten times that lifespan. (Note that you can still read data from an SSD whose cells have "died" in this way, but if you can't write to them, you have to replace them anyway). SLC is also faster than MLC... and much more expensive. Fourth, data configuration. RAID 0 is much better for SSDs than for hard drives; there's basically no chance of data loss through hardware failure (no moving parts), and throughput increase is much more near linear because there aren't as many of those nasty overhead issues that get in the way... you know, like seek time. RAID 1, though, is basically useless for SSDs. Again, data loss through hardware failure basically doesn't happen, so backup is better done the traditional way than with mirroring. So where does all that leave you? First off, honestly, if I were in your shoes, I would do everything in my power to get the tape out of the equation. It will do absolutely nothing but slow everything else down, and SSDs are more than reliable enough to serve as backup storage anyway, especially if space is a minimal issue. If cost were not an issue and I were building the system from the ground up, here would be the setup I would run. For the regular data storage, get one of these babies. This is the OCZ Z-Drive, SLC version. It's basically an SSD that plugs straight into a PCI-Express slot - shows up on your computer just like any other drive. Fastest single drive on the market, bar none. 256GB and 512GB capacities - I'm guessing you would just need the 256GB one. I have no idea how much it costs, although I do know that the 256GB MLC version costs about US $1400. *ouch* If on the off-chance the computer the data goes on doesn't have a PCI-E slot, you can always make due with two or more of Intel's X25-E 64GB drives, and an awesome controller from Highpoint or Areca if you get more than two. Then for backup, set up a 2.5" hot-swap bay in the same computer that the Z-Drive is in and get a regular old 256GB MLC SSD like this Patriot Torqx. Yes, it has a 10,000-write lifespan, but as a backup drive, it would be fantastic. You have 40GB of data now, so that's 40GB of data written to it once per day. By the time the whole drive has been written to once, it's been six days; that means you can do 60,000 backups to the thing before it bites the dust, one backup per day. I'm guessing you'll want a higher-capacity drive anyway sometime before you hit that 164.38-year mark. Let us know if you have any questions. - Diosjenin -
What backup suite are you using? What's your budget? SSDs may be far more reliable than mechanical disks, but using RAID0 in a production solution is lunacy. Upgrading to SSD is still thinking inside the box a bit, I'd sack that lot off entirely and go for snapshotting and streaming that to tape. You're never going to get a nippy traditional backup with that many small files. You shouldn't have a problem streaming to LTO2 at 50+ MB/sec, its your millions of tiny files that are killing you and your problem is at the host side. Writing to the tape at 1MB/sec as you are now is going to result in some serious shoe-shining and it isn't good for your drive/tapes. Also, I'm curious, what's your RTO for this system?
Thanks for your input guys Backup software is Backup Exec. We've played with buffering and all the tape parameters until the cows come home, increased performance somewhat but it was still (obviously) pretty terrible. Budget... somewhere between a little, some, and a lot. We're open to suggestions. And as for RAID0, that's never, ever going to fly in a business environment. Even with super reliable drives. The problem is that no matter what sort of backup we do (snapshot, regular, B2D2T etc etc) we get completely boned by the sheer fact of how long it takes to read all these little files. Even if it's just checking the archive bit and ignoring them, it still has to do a seek (or 3) and check the file. I'm relatively certain that this is where a large percentage of our bottleneck lies and reading a lot of SSD reviews shows a 60x improvement (!!!) with random reads of small files. That's what drew me to them in the first place. I know we really need to get the data streaming, when it streams we achieve 600-1200mb/m to the tape which is totally sufficient. The question is, how on earth to get it streaming? Is the throughput of random reads from a RAID10 of 4 SSDs enough to get it to stream (using very large buffers on the tape if we have to) or do we need to do a backup to traditional disk first and stream it off that as 1 enormous file? This would probably make more sense for a restore as well, I guess. RTO... nothing spectacular. Only the freshest files (maybe the last week?) would need to be back ASAP and we're moving to DFS as well to help cover them.
10-20MB/s? That's all you need? One SSD could handle that, easy. Because the architecture of flash memory is such that index location of files makes no difference to access speed, random performance is essentially equivalent to sequential performance (given a good controller). Even one Intel SSD runs right up against the SATA II throughput barrier of 300MBps, random or sequential. Like I said earlier - your tape would then by far be your biggest bottleneck, and you could save a whole lot more time by backing up to another hotswapped SSD rather than tape. - Diosjenin -
A hotswapped SSD would be nice and probably really fast except that we have 3 weekly sets and a monthly set that gets archived for 2 years. That's a HELL of a lot of SSDs. At least when we 'retired' the 'media' I would end up with a pretty bitchin' gaming rig Besides that there's another couple of hundred gb in the backup...
You're going about this all the wrong way. Your problem isn't writing to tape, there's plenty of speed in LTO2 for what you need, your problem is at the source. I don't know a huge amount about the ins and outs of Backup Exec (I work mostly with NetBackup, with some splashes of TSM and CommVault, all of which are very different I'm sure, but bear with me). Does Backup Exec have anything (perhaps an additional license) that allows it to delve down below the filesystem itself and take a raw dump of a volume? This should ignore the fact that there's millions of small files and stream off to tape at full speed. Google tells me that perhaps Backup Exec System Recovery would do this. This would probably mean that incrementals during the week are a no-go, but with 40GB of data this shouldn't be an issue. What sort of storage hardware do you have at your disposal? Some hardware based copies/snapshots would do the trick, but would obviously cost a bundle if there's not already some kit available.
I was wondering if it had the capacity to do some sort of block-based backup or similar... I'll have to look into it further. Possibly not though We're going to hugely revise our storage of these files anyway so we're only backing up 1 month worth instead of 24 months... except I just found another 500,000 10kb files on that box just to make the problem worse. Storage hardware, very little. Everything here is DAS. EDIT: Just for a moment though, let's pretend the issue isn't backup, it's regular file access performance to these millions of little files. It's slow as hell (root of the problem, obviously) while everything else on the server is very speedy. How would you approach this?
Question time! Are you using internal disks or an external enclosure? What speed are the disks? How much more space is in the case/enclosure? How is it attached to the server? How much storage you you have in total, across the board? You say your budget is between a little and a lot, how much is a lot? If there's a fair bit of storage, just scattered about, maybe its time to start looking in to a SAN based solution to replace the lot. Something like a HDS AMS2000, a couple of Brocade 5100s and dual-port HBAs for all of your servers would have an entry fee of around £200k (ongoing support costs would vary wildly though, depending on what skills you have in-house), depending on how many hosts you have and how much raw space you need, but gives you all of the high availability goodness that comes with a proper SAN. There's iSCSI too if you want to save some pennies. If that's a bit too rich, throw more/faster spindles at it, and possibly a faster RAID controller. 10x 146GB 15k in RAID10 would give you a fair amount of iops, if that's not enough the SLC SSDs would be the next logical step. EDIT: Just a thought, you don't have some AV software that's attacking the files as its trying to back them up, do you?
