Backup Options - Multiple Servers to One Backup Server

[PendulousMind]

Hi all,

I am a dba for a large co and I have been tasked with creating a reasonable backup plan for our new servers. So far I have 1 Application, 3 DB and 1 Backup Servers in a rack in our data center. I am looking for an in expensive solution as to backing large amounts of data to this 1 backup server. I was hoping some of the network gurus could assist since I have never done any complex backup plans other than the backups I make for my db's. All of the research I have done points me to using the share networks for these tasks, however I am looking for something a little more complex and robust such as FCoE (Fibre Channel over Ethernet) since the data being backup is quite a large amount. Any help would be awesome.

[Nose Nuggets]

how much data, is really the key component here. and what OS's are you using?

assuming windows, if you want simple, you can make a scheduled task to make DB backups locally, and then a scheduled NTBackup job to move those files and other files you want backed up off that system to your backup server. you can setup NTBackup jobs to use a network location (\\backupserver\share or without a share you could do \\backupserver\c$\Backup folder)

it also depends what else is using the network; if you dont have any network overhead then gig ethernet should be pretty good for backups.

then there is also how often you are doing backups. what kind of retention and maximum lost data requirements you have.

finally, if his is seriously important stuff i would recommend some kind of offsite sollution in case the building burns to the ground. this could be something as simple as rotating two external hard drives or using a vendor like Iron Mountain to do a weekly tape pickup/dropoff

as for backup products, i have used every Symantec backup exec solution available from the last few years, from full blown backup exec with CPS to the rinky-dink backup exec products for sbs's. they all suck equally, and produce nothing but problems. just like every other piece of symantec software.

[PendulousMind]

how much data, is really the key component here. and what OS's are you using?

assuming windows, if you want simple, you can make a scheduled task to make DB backups locally, and then a scheduled NTBackup job to move those files and other files you want backed up off that system to your backup server. you can setup NTBackup jobs to use a network location (\\backupserver\share or without a share you could do \\backupserver\c$\Backup folder)

it also depends what else is using the network; if you dont have any network overhead then gig ethernet should be pretty good for backups.

then there is also how often you are doing backups. what kind of retention and maximum lost data requirements you have.

finally, if his is seriously important stuff i would recommend some kind of offsite sollution in case the building burns to the ground. this could be something as simple as rotating two external hard drives or using a vendor like Iron Mountain to do a weekly tape pickup/dropoff

as for backup products, i have used every Symantec backup exec solution available from the last few years, from full blown backup exec with CPS to the rinky-dink backup exec products for sbs's. they all suck equally, and produce nothing but problems. just like every other piece of symantec software.

Using Windows Server 2003 on all of the servers, Backing up multiple 20+ Gig DB's along with web directories and project code files. I would estimate around 75 to 100 Gigs all together.

Basically this will be done here in CA and we also have a similar setup in FL. CA backup server will handle CA Servers and same with FL. As far as an offsite location for back up I havent made it that far yet in my planning. Still in the planning stages for getting the data over in an efficient matter.

The App and DB servers are used through out the day but very lightly at night. So my window would preferably be at night. I am also looking to do this daily since majority of the data is updated on an ongoing daily basis and is crucial to the business.

I am comfortable with using the Windows Backup, however my real issue is how to go about it as far as share/ftp or something like FCoE. I hesitate to use the share/ftp only cause of the amount of data being backed up and was hoping that there was a better method.

[ldivinag]

dude...

dont you guys have the network admins to do this kind of stuff? unless you are wearing multiple hats.

cheapest and still dependable are the tape backups. lots of options.

dont forget to keep tapes OFF SITE. who cares if you make backups if your network room burns down... with the tapes in the same location.

[lazyworm]

You need to do some math.

* what's your backup frequency (daily? hourly? weekly? etc)
* what's your backup window (how many hours/min per time period)
* what's your backup policy? (incremental? full? some kind of fancy incremental algorithm?)
* what sort of data growth rate do you have? 3, 6, 12 and 24 months


After you have the above answered, then do the math. What sort of I/O can
the disks on your backup clients (the app and DB servers) support?
What sort of network bandwidth can they support? These numbers probably should be like
60-80% of the theoretical max minus the amount of normal traffic/transactions the machines
will be doing during the backup window.

Next, check to make sure your network can support this and the backup
server has enough cpu, disk I/O and network I/O to receive all this data
in a timely manner.

good luck.

[lazyworm]

forgot to add...

* what's your retention policy on the backed up data?

if you have 100G per day and you keep 30-days worth of data.
that means your storage (disk, tape, whatnot) needs to be
at least 3TB (plus growth)

[lazyworm]

weird double tap

[PendulousMind]

dude...

dont you guys have the network admins to do this kind of stuff? unless you are wearing multiple hats.

cheapest and still dependable are the tape backups. lots of options.

dont forget to keep tapes OFF SITE. who cares if you make backups if your network room burns down... with the tapes in the same location.

I wish, our IT handles most of the National development whereas the team I am on mostly handles local level db/web development. In a lot of cases, apps we develop will go national, but it has to be proven at the local level.

As far as the offsite issues I will be planning that part of it once we have the backup server up and running. I imagine it will handled by another data center in the area. The only downfall will be if the SHTF, but if that happens I doubt I would be concerned with it then.

[PendulousMind]

You need to do some math.

* what's your backup frequency (daily? hourly? weekly? etc)
* what's your backup window (how many hours/min per time period)
* what's your backup policy? (incremental? full? some kind of fancy incremental algorithm?)
* what sort of data growth rate do you have? 3, 6, 12 and 24 months


After you have the above answered, then do the math. What sort of I/O can
the disks on your backup clients (the app and DB servers) support?
What sort of network bandwidth can they support? These numbers probably should be like
60-80% of the theoretical max minus the amount of normal traffic/transactions the machines
will be doing during the backup window.

Next, check to make sure your network can support this and the backup
server has enough cpu, disk I/O and network I/O to receive all this data
in a timely manner.

good luck.

Apparently I am way over my head on this. I will do the best I can with the suggestions on this

forgot to add...

* what's your retention policy on the backed up data?

if you have 100G per day and you keep 30-days worth of data.
that means your storage (disk, tape, whatnot) needs to be
at least 3TB (plus growth)

It will be about 400G in total, daily growth wont be much more then .25% of that. Definitely a lot of factors to think about. I appreciate your help on this.

[ocabj]

dude...

dont you guys have the network admins to do this kind of stuff? unless you are wearing multiple hats.

cheapest and still dependable are the tape backups. lots of options.

dont forget to keep tapes OFF SITE. who cares if you make backups if your network room burns down... with the tapes in the same location.

Why would network admins handle backups? Network administrators handle the network. The Systems/Infrastructure Administrators should handle backups.

Tapes? Yes, we still use tapes, but that's the last resort for recovery. Snapshots and netbackup of systems are primary. Maybe transaction replay logs for DBs.

[karatebum]

get help from corp if you can. backups, doing them right, takes a couple of years of expiernce. just getting the thing set up with hardware if you've never done it before will take a half a day. backup exec has some really good features like a sql agent, open file support(uses windowz shadow copy thingy, and iteligent disaster recover(hell mary recovery where you boot off a cd, and it images itself from a tape.
anyways, my bill of material would be a LTO 5 juke box($6k) they take two magazines, one left one right 8 tapes each, you only need one LTO mag, especially 5th gen are epic low voltage scsi cable and controller 200 bucks, 20 tapes plus 1cleaner you keep in the mag, 1k, backup exec 800, each agent for the server is 150, sql agent is 4-500, advanced file open option, and iteligent diaster are now free with it. if you have a san, you can get a ndmp agent that will back that up(get latest snap shot direcotory, not working volume. is like 6-700
if you live in sac, i can show you some evening, it'll be enough to make you dangerous, and not good enough to be confident, it really takes a while with mentoring and self discovery to get up and going, and its gotta be right, you can screw the pooch by just selecting the "archive" check mark box(it deletes what has been backed up after backup is done.
i've neen doing it for 10 years, and didn't really get good until like year 7,

ps in this economy, a network admin is generic term these days, for a genralist, who can afford to just hire one network guy anmore and thats all he does? unless your over 15,000 people.

good luck, and again, if your in sac, i can show you enough to get started, but your skill set wont be enough to suceed, maybe skate by though.

ps a juke box, which will hold 16 tapes, only costs 1500 more then a single tape unit,

[PoofNoEyebrows]

Why not look into a simple NAS system and use rsync. Should be really easy to script and set on auto pilot. i know Western Digital has some 4TB models that are under 900.00. Tape is a pain. With rsync you should be able to setup a rotating incremental system for how ever far back you need to go.

[high_revs]

not as technical as the others but putting on a different type of hat as i deal with the ramifications sometimes in escalations....

how critical is this for the business pendelous mind? it sounds like it's pretty critical hence you used the word "expensive." (or did you mean inexpensive). putting the business hat on a little...

1. how critical is it that we recover asap in case of a meltdown? what is the cost per hour as the system is being recovered. that might help determine your methods for the backup solution - tapes or nas or san or something.

2. can you afford an enterprise type solution? i mean, sure you can use windows server backup options. but would they be robust enough for you?

3. at the minimum, grandfather model. 6 days increment, 1 day full.

4. is the mgmt willing to have a standby server that's replicated of the ones you're trying to backup? (goes back to question, can the business afford the downtime while you're recovering from tape). have you considered the recovery methods hence asking how long b4 mgmt says, we're going bankrupt here while this thing is down.

5. is it critical enough to have offline storage? i.e. store off site.

we had a case for some legacy systems in canada where the backup server had issues on more than one occasion. it's attached to an oracle database. because some logs couldn't be backed-up, they couldn't be cleaned up on the oracle server so disk space got full. and it brought the database down. this db housed an application cost 5 figures per hour of downtime.

just trying to put a different dimension in how this is approached as sometimes, it's not just the technical areas that need to be considered. the business purpose and cost can help drive what technical solution you should implement.

p.s. i feel for ya. as others said, it would be best to partner with a server admin who should be the one handling the backup server. if you have to backup to a network (and depending how big a network), consulting a network admin migth be needed as that's huge data you'll clog up the network, unless you resort to other methods. but this'll expand your resume. most dba's in our company get exposed to this because they partner with the infrastructure teams and it's enterprise wide. 11 years ago, our sql server dba was also learning about this and had to take care of the backup of the db's himself too. but we got help though from our sys admins.

[meaty-btz]

I will come out of left field and say:
If the data is mission critical and that large, a tape might be too slow for your window of opportunity to run the backup.

There are a ton of tools out there however, in my time I have learned some simple rules that have bearing on backups and the like.
1. Physical Servers will be the death of you. Virtualize, Virtualize, Virtualize. Host, ESX on the iron and all your servers should be virtual. This simplifies disaster recovery in amazing ways if you happen to lose some hardware. It also makes backing up the servers far simpler (single file backup) and restoration easier (single file recovery).
2. Your data should not be on your servers day to day servers as it were. Use a SAN (storage area network), period. DAS (direct attached storage) is a deadly sin in a real up to date data-center. NAS can work but is slower and less reliable. SAN is the way to go.
3.Use QUALITY Drives in your system. MTBF on a good drive is light years ahead of a cheap drive. Quality drives cost a ton for a very good reason. In a 6 year cycle we have had no quality drive fail, compared to our cheaper drives which die yearly.

Once you are at this point configuring a reliable and speedy backup is a snap! Disaster recovery is just as easy. Tapes are slow and used for archiving mostly. Tapes are surprisingly unreliable, but were the only game in town for a long time for cost effective large data archiving.

So, like owning a gun, its not just about backup. It is about preparing your environment in such a way as to aid disaster recovery when it happens. It WILL HAPPEN. I would be mortified if you are not already doing nightly database maint on your SQL db, part of which should be a "backup" export.

The point is this: backing up an entire server is dumb and a waste of time. ESX and VMware means you become hardware agnostic and your servers are one click backup and restore. Suddenly you have saved yourself all the hassle of backing up a Live Server and all that entails. Moving your critical data off the DAS onto a SAN allows for future grown and again, hardware agnostic environment. Backing up a SAN is simpler and more reliable. Hell you can have 2x SAN's configured with one as a LIVE SYNCED FAILOVER. That is, in nearly real-time your primary can drop and the servers will all kick over to the backup SAN. Your tertiary backup in the 2 SAN environment is the Clone Server runs the backup job at a preset time of day. It can now be nearly any window you wish because your backup won't occupy the primary active SAN in such a way as to degrade user experience. Tape becomes an excellent option again.

If you are in a real business environment your QOS requires nearly 100% uptime. Your "backup" environment should be built about the concept of UPTIME. A down server shouldn't slow your roll if you build correctly. Data loss becomes a thing of the past. Backup in the old meaning becomes a tertiary system designed for archiving and SHTF type situations. Everything else is designed to be failover tollerant and "live".

Lots of buzzwords here but I hope the meaning is clear. Backing Up in the classic sense is risky, slow, and a bad idea in the modern Data Center. Correctly building a modern data center hardware wise to be safe and fault tolerant is about having the proper infrastructure in place to negate the fear of data-loss excepting only the most aggregrious situations.

Otherwise you are about as well off with rsync or NT Backup as you are with the fancier backups copying cross the network to a NAS Box. I wouldn't trust such a system anywhere except at home and that is only because I can't afford to setup my home the way things are at work.