Network Guy's - WAN Capacity Planning question

[jmlivingston]

This is a question for you hard-core network guys that have done a lot of enterprise WAN work.

One of the locations on my WAN is starting to really hit their bandwidth limits pretty hard. Our 95th Percentile for transmit usage has grown to 89% of existing capacity and I'm starting to receive complaints about performance from our employees.

1st Question: How much headroom should I keep between the 95th Percentile and the max capacity of any given WAN circuit?

2nd Question: Where did you get that metric, and is it documented anywhere?

With the current economic environment I'll need to fully justify any requests to upgrade this circuit.

Thanks,
John

[bigmike82]

I've never done actual metrics on it, so I can't help you there.

Have you checked on what kind of traffic they're passing? Is it all legit, or are the employees clogging the tubes with youtube and such? Have you looked into using QoS to prioritize necessary traffic (email, for example)?

Before wanting to upgrade the circuit, I'd do everything possible to cheaply address the issue. Management will thank you for it.

[jmlivingston]

Yes, all traffic is legit and we are already using QoS on our WAN. As well we have WAN optimization appliances (Riverbed) at all of our locations.

Additional information: This is a fully-meshed MPLS network and each site has it's own internet connection so there's no internet traffic across the WAN.

[Mute]

Have you run your data through a bandwidth calculator yet?

[lazyworm]

I'm not sure if there is any hard fixed rules, except as you get closer to 100%,
you'd incur more overhead as there are more timeouts, drops,
resend etc. If these were my sites, this is what I'd do....

look for all available historical data and use it to estimate usage growth
over time. Project this out for X months. Where X should be slightly longer
than your contract for your pipes. OR whatever period that mgmt would
be comfortable with spending money on. Typically 1 - 2 years.

keep track of the trend over time and upgrade when usage is 4 months
out from getting maxed. ( 3 months for provisioning and 1 month for
paper work/padding) Where maxed is the a few % from the normal peak
of the usage cycle.

If your users are complaining. You're already out. It would only get worse
as you wait for new circuits to get dropped in and mgmt to say okay.

[lazyworm]

forgot to add... each site only has 1 link into the mesh?
If you have duals, then the % should be around 45%.

[jmlivingston]

look for all available historical data and use it to estimate usage growth
over time. Project this out for X months. Where X should be slightly longer
than your contract for your pipes. OR whatever period that mgmt would
be comfortable with spending money on. Typically 1 - 2 years.

Unfortunately we only have usage data for the last 4 months, prior to that we didn't have any tools that could monitor and capture the usage trends.

If your users are complaining. You're already out. It would only get worse
as you wait for new circuits to get dropped in and mgmt to say okay.

Yup, and this is exactly where I'm at today.

forgot to add... each site only has 1 link into the mesh?
If you have duals, then the % should be around 45%.

There are two separate MPLS meshes, each mesh using a different carrier. Our "primary" mesh handles data traffic and the "secondary" handles VoIP traffic. Each has the ability to fail over to the other in case of an outage, but bandwidth on the secondary network is minimal compared to what we do on the primary network.

[DiscoBayJoe]

There is no easy answer. It depends on your line of business and how time sensitive your applications are.

Quick rule of thumb -- Watch your interface Errors & Discards counters (I use SolarWinds Orion):

> If you are seeing Transmit Discards, upgrade the Circuit
> If you are seeing Receive Discards, upgrade the Device

A few here and there during peak are OK, but if they happen all day long, you'll benefit from the upgrade.

[DiscoBayJoe]

There are two separate MPLS meshes, each mesh using a different carrier. Our "primary" mesh handles data traffic and the "secondary" handles VoIP traffic. Each has the ability to fail over to the other in case of an outage, but bandwidth on the secondary network is minimal compared to what we do on the primary network.

You'd be much better off binding the two pipes into one larger pipe and implementing a well thought out QoS plan. Based on the rule peaks and averages, 1 + 1 really does equal 3

[odysseus]

Honestly, outside of QoS, application management (if you can find a glaring inefficiency which often can be seen), and other data management tools like acceleration, most have to just subscribe for more bandwidth. Some MPLS implementations I have seen can burst or can scale bandwidth (though these are expensive contracts) as needed by the provider.

As far as a figure, I have seen it vary depending on what has been standardized and agreed on as an organization, from 80% to "hey what's wrong I can't get this to work?". Industry used to hover back in the day as a gold standard somewhere around 80% utilization as your metric for normalized max util, but in this economic environment...

[lazyworm]

If I were in your shoes, I'd get mgmt buy off today before you pull any rabbit
out of the hat.

technically, sounds like routing some traffic over the other mesh is your best
bet. Other than that maybe route some traffic out the internet link via a vpn back to corporate?

Does this have to be a network only solution? If you have support from
the whole IT, can you move the services closer to the end user so the
bits doesn't have to go across the wan? Can you cache at the user's end?
Can you reschedule certain traffic off peak? e.g. backup, batch-typed work?

[lazyworm]

Unfortunately we only have usage data for the last 4 months, prior to that we didn't have any tools that could monitor and capture the usage trends.

Have you checked with your service provider as well? They might have some for billing purpose.

[odysseus]

Does this have to be a network only solution? If you have support from
the whole IT, can you move the services closer to the end user so the
bits doesn't have to go across the wan? Can you cache at the user's end?
Can you reschedule certain traffic off peak? e.g. backup, batch-typed work?

Absolutely the first hit. An assumption is that effort has been made to look for any inefficiencies that can cut down a lot of util if worked on. However also true is the classic issue of who owns it, fixes it, and pays for it out of their cost center. It can become a cage fight. But a traffic analysis should point the way pretty easily.

[five.five-six]

But a traffic analysis should point the way pretty easily.

and you will find who is watching you tube all day. I was taught to engeneer a network to run at 50%... but that was a long time ago

[lazyworm]

Absolutely the first hit. An assumption is that effort has been made to look for any inefficiencies that can cut down a lot of util if worked on. However also true is the classic issue of who owns it, fixes it, and pays for it out of their cost center. It can become a cage fight. But a traffic analysis should point the way pretty easily.

Politics never help in a technical situation. If everybody believe they're
equally f'd. Buy in usually happens much faster

edit: the users who complained... put them to good use, they're already on your side