Linux LVS (Piranha) IP load-balancing

[jmlivingston]

Any of you Linux guru's ever work with LVS/Piranha? I'm trying to put together a system for doing basic IP load-balancing in front of a pair of webservers but I've run into some snags.

Thanks,
John

[Pyrodyne]

Any of you Linux guru's ever work with LVS/Piranha? I'm trying to put together a system for doing basic IP load-balancing in front of a pair of webservers but I've run into some snags.

Thanks,
John

A little more information would be helpful. Are you attempting to load balance using different links, or attempting to distribute traffic on a single link? Are both webservers serving the same data, or do they serve different purposes?

[socalblue]

It's not hard & works fine for basic HTTP type stuff. Depending on the application requirements piranha may be all you need. Other options are ultramonkey & plain old round-robin DNS.

[jmlivingston]

This would be for a basic web-server farm, with 2 real servers (Windows IIS) sitting behind a VIP. Both servers will have the same content, with a back-end MS-SQL Enterprise Cluster housing the data. This is something I could easily do even with an old Cisco Local-Director, but I don't have one handy. If I can demonstrate that load-balancing will solve a) redundancy and b) capacity issues I might even get the budget to buy an ACE or perhaps an F5 BigIP. If this works really well? Might just stay with LVS and Piranha.

If either of you have actually done this using the NAT mode before, I've got no problems sharing the details with you but I'm a bit hesitant to post up all my current config information publicly. Just send me a PM with your contact info. It looks like all the LVS/Piranha is working correctly but iptables is tripping me up (Go figure, I'm a networking guy! Just that using iptables is nothing like the Cisco gear I work on all day.)

John

[nick]

Since you're running Windows, and your needs are basic, why not just use WNLB? No need for an extra box. And few things are easier to set up.

[jmlivingston]

We had it running for a long time and yanked it out a while back and went down from two servers to one.

WNLB can run in two different modes, in it's default mode it uses IP multicast which flooded our server VLAN every time our search engine did a crawl of the website. We worked through that issue and made the changes necessary to convert it to unicast mode which worked fine for a while.

About 2 months ago we migrated the app from Windows 2003 32bit physical servers to Win2008 64bit VM's. When this was done we had to move away from Unicast mode since it requires static arp and MAC entries in the switches to make it work properly (by MS's design ). This made us eliminate one of the servers. Our ESX servers run on giant IBM blade servers with integrated switches, plus the switch inside the ESX host itself, so it became to much to try and deal with this way. If we had to vmotion the system to one of our other chassis for some reason it'd break the WNLB because updating the static MAC entries in our core 6506's and the integrated blade-server switches would be a manual process.

All that ruled out WNLB for us. If we were running the Nexus 1000 switches inside our ESX servers that would change everything, but right now we're not.

So that brings me to the current situation. We're considering the purchase of a hardware load-balancer such as a Cisco ACE, we're working on getting a demo to test out but that hasn't happened yet. So right now I'm trying to do a proof-of-concept with Piranha, that LB can fix some of our concerns. If Piranha works out really well for us, we might just make it redundant and keep it around. We aren't looking for anything fancy, just to do a round-robin or least-connections type load balance across two webservers. I've got the Piranha box built and LVS appears to be fully functioning for a pilot, but something is broken and I'm pretty sure that it's iptables.

John

[Pyrodyne]

This would be for a basic web-server farm, with 2 real servers (Windows IIS) sitting behind a VIP. Both servers will have the same content, with a back-end MS-SQL Enterprise Cluster housing the data. This is something I could easily do even with an old Cisco Local-Director, but I don't have one handy. If I can demonstrate that load-balancing will solve a) redundancy and b) capacity issues I might even get the budget to buy an ACE or perhaps an F5 BigIP. If this works really well? Might just stay with LVS and Piranha.

If either of you have actually done this using the NAT mode before, I've got no problems sharing the details with you but I'm a bit hesitant to post up all my current config information publicly. Just send me a PM with your contact info. It looks like all the LVS/Piranha is working correctly but iptables is tripping me up (Go figure, I'm a networking guy! Just that using iptables is nothing like the Cisco gear I work on all day.)

John

This may help, even if it is quite old. Feel free to PM or obfuscate details in your configs. Iptables is usually pretty simple to get going once you nail down the right combinations.

[bigmike82]

I've never used pirhana, but I'm pretty decent with iptables these days...

If you want, PM me a sanitized config (no WAN IPs) and I'll take a look.

There are also some cool things to do with IPtables that will show you stats of what each chain does...so if you suspect a chain is dropping stuff it isn't supposed to, you'll see it (if your chains are set up properly).

[jmlivingston]

I ended up cross-posting this to another forum, so I've got a sanitized version now. It's all in a PDF file available here. In the meantime I'll take a look at that link from Pyrodyne.

John

[bigmike82]

Why do you think IPTables is an issue?

Does it work if you stop the IPTables service (service iptables stop)?

[jmlivingston]

Why do you think IPTables is an issue?

Does it work if you stop the IPTables service (service iptables stop)?

A tcpdump on eth0 (server-side network)shows the traffic going out to the real servers and returning, but a tcpdump on eth1 (client network) only shows the request and no replies. Since the return traffic is hitting the load-balancer I'm presuming that it's iptables causing my problems.

It does not work if iptables is stopped, but since iptables is doing the NAT that doesn't surprise me.

John

[Sig226]

This is my bread an butter.... That being said, I hate Piranaha...Too many small issues (which may not apply to your situation) and certainly a very low traffic mailing list.

Most of these kernel based loadbalancers simply manage IPVS groups---IPVS being the module that actually keeps the connection tracking table and handled the balancing of new connections.


If I were you---I would look into keepalived. It isn't update all that often---the last update being for IPV6, but it is solid. I can't name names---but one of the largest webhosting companies in the country uses an LVS setup a built over 6 years ago....At the time I left one pair of Dual 2.4Ghz Xeons w/ 2GB of RAM were handling over 40k active connections with over 60Mbit of sustained traffic to 50 back-end real servers. (Interrupt handling was the hardest part to work out)

Keepalived manages the VRRP IPs, Health checks, and IPVS definitions in one package. (Only IPVS and IPTables are required in additional keepalived) If you are a GUI guy though---you might not like it---there is no web interface.

Just my $0.02

If you do end up going that way, I'd look at IPTABLES FWMARKs in a NAT config with persistence.

Feel free to PM me with any questions...

[jmlivingston]

Thanks Sig, this looks really interesting. I've bookmarked this for future reference, it looks like keepalived is a pretty sophisticated load-balancing system. Not sure how my Windows team will handle having to use a linux command-line.

I'm in a holding pattern on the existing Piranha build, waiting for a pair of servers to begin testing with.

John

[lazyworm]

No experience with Piranha, but if you want to use a software load balancer,
check out HAproxy. http://haproxy.1wt.eu/

In my experience, it's light weight and solid.

[SEJeff]

You really don't need the overhead of Pihranah. Just use keepalived. Keepalived is an open sauce vrrp daemon. I've got a howto for a very _basic_ setup for failover of non-stateless services on my website:

http://www.digitalprognosis.com/opensource/scripts/keepalived/

That setup is for failover and not lb like you want, but read the docs and example configurations that come with keepalived. You can do everything you want with it.

@Sig226: Ironically, I know the admin who built the LVS / Keepalived setup for "that big isp 6 or so years ago". He is the one that made alot of ticketmaster use it and taught it to me. It is good stuff as you say.