HTML compression?

[command_liner]

Is HTML compression used in the configuration of vbulletin or in the server that dishes out the HTML?

I am thinking particularly of zstd and zlib. Zlib comression was novel 20 years ago, and zstd compression was novel 4 years ago. But it is old tech now. In practice zstd compression works much better -- more compression and way less CPU demand.

Can I get a test channel to browse with zstd compression turned on?

See https://www.vbulletin.com/docs/html/...ookieandheader

To be more clear, there are two control points. One is the generation of the HTML by vbulletin, and the other is the service of the generated HTML on some server. Only ONE compression method should be used. The best thing to do would be to use vbulletin to generate zstd-compressed content. The 2nd best would be to have the server recompress the content and send it out with zsd. 3rd would be to have the server do zlib (gzip or deflate) compression. Finally it may be possible only to have vbulletin do zlib compression.

[LBDamned]

I don't know the tech behind your question - but he's going cloud based, apparently... so, not sure that any of what you're trying to accomplish will matter...

https://www.calguns.net/calgunforum/announcement.php?f=64

[command_liner]

I don't know the tech behind your question - but he's going cloud based, apparently... so, not sure that any of what you're trying to accomplish will matter...

https://www.calguns.net/calgunforum/...ement.php?f=64

The cloud is just an abstraction of where the compute is done. Part of the problem of this site, and many others, is the sending out of way too much data traffic -- too many bytes. This imposes transmission costs on the sender, and slows down reciept.

Sure, there are way too many errors in the HTML here -- you can see them if you turn on the developer tools in Chrome and refresh the pages. Each one requires exception processing by the browser.

The main time sink is establishing the first socket connection and setting up the ssh. This is because the server is very busy. Why is it is busy? That may be because of socket throttling on purpose. Or it could just be due to a stupid misconfiguration of the site software.

I have solved millions of dollars worth of software problems by finding stupid configuration errors that nobody thought about. The hardest part of finding this sort of problem is finding and reading the documentation about the featuers of the existing system. Almost as hard is getting the people in charge to actually use the documention to configure the system correctly.

Turning on zstd might crush down the content sent to each browser by 10:1. That would produce, in effect, a 10:1 increase in speed for content delivery. This will have the add-on benefit of helping with the paging storm that is almost certainly causing those very slow https startups.

Take my word for it, you can have poorly configured software running in a big cloud infrastructure, and it will still perform poorly. OTOH, I have personally done super-provisioned cloud configuration to prove that there were some awful software faults in "mature" software. There is nothing quite like demonstrating a 100:1 reduction in startup time to somebody that insisted the software was "fine".

[LBDamned]

Software/server incompatibility... it's been mentioned more than once...

Supposedly owner is moving to cloud as resolution.

[command_liner]

Moving buggy software to the cloud just results in cloudy buggy software with another layer of irresponsibility/finger-pointing. But it so tempting when the snake oil people tell you it will solve all your problems.

[Kestryll]

The move to cloud based is primarily a financial one, it is cheaper than the self hosting.
The site issues are due to the forum running on 20 year old software that went EOL 7 years ago and has incompatibility issues with modern PHP and MySQL.

When we move to the cloud server we will also be updating the software to the latest version which has none of the conflicts and incompatibility issues.

[command_liner]

The move to cloud based is primarily a financial one, it is cheaper than the self hosting.
The site issues are due to the forum running on 20 year old software that went EOL 7 years ago and has incompatibility issues with modern PHP and MySQL.

When we move to the cloud server we will also be updating the software to the latest version which has none of the conflicts and incompatibility issues.

Good to know.

My point still stands: turn on server-side compression! Those of us out here on the edge of the internet need all the added bandwidth (virtual or real) that is available. HTML stream compression costs the provider almost nothing and has real benefits for the consumers. Indeed this is why Facebook spent a man-decade on the task of getting zstd compression working for HTML ... the benefits are real.

[SanDiego619]

Wouldn't compression reduce the amount of data being sent out, but increase CPU load?

[command_liner]

Wouldn't compression reduce the amount of data being sent out, but increase CPU load?

Data streaming out the wire involves page faults, which are very expensive. Zstd compression is written so it has few branch penalties and page faults. As a result, often the net result is reduced overall time (better hardware use) because there is less page faulting. That is becuse there is less data.

One pays the CPU cost of compression by avoiding the page faults of dealing with larger data volumes.

This is good research and a good result by Facebook. CPU is so fast today that one should think of CPU time-cost as zero at both the producer and consumer end. Disk IO (or big memory page faults) at the producer side have some cost, but the major cost is transmission setup and transmission of data.

Anything that can be done to reduce the IO and transmission cost is worth the effort. Need more convincing? The current Linux kernel uses zstd during swap! It costs less time to run the zstd compressor on the running image than it does to swap it out. Thus compressing pages before writing to disk and after writing to disk (on read-back) is a net win in time. The default compression in RPMs from RedHat is now zstd, and installation time is reduced for the same basic reason -- there is less IO.

All the static content here should be recompressed to zstd, and all the dynamic content should be sent out with zstd compression. Cost will go down and user satisfaction will go up.

Sometimes newer is better. This is one of those times.

[SanDiego619]

Data streaming out the wire involves page faults, which are very expensive. Zstd compression is written so it has few branch penalties and page faults. As a result, often the net result is reduced overall time (better hardware use) because there is less page faulting. That is becuse there is less data.

One pays the CPU cost of compression by avoiding the page faults of dealing with larger data volumes.

This is good research and a good result by Facebook. CPU is so fast today that one should think of CPU time-cost as zero at both the producer and consumer end. Disk IO (or big memory page faults) at the producer side have some cost, but the major cost is transmission setup and transmission of data.

Anything that can be done to reduce the IO and transmission cost is worth the effort. Need more convincing? The current Linux kernel uses zstd during swap! It costs less time to run the zstd compressor on the running image than it does to swap it out. Thus compressing pages before writing to disk and after writing to disk (on read-back) is a net win in time. The default compression in RPMs from RedHat is now zstd, and installation time is reduced for the same basic reason -- there is less IO.

All the static content here should be recompressed to zstd, and all the dynamic content should be sent out with zstd compression. Cost will go down and user satisfaction will go up.

Sometimes newer is better. This is one of those times.

That is interesting, thank you for the detailed information. I like that stuff, just don't know a lot about it.

[AlmostHeaven]

My biggest question is why the performance issues have been so inconsistent. Traffic levels do not seem to correlate with load times.

[command_liner]

Caution: Computer Science Ahead!

In a situation where there is high memory demand, especially where the system starts swapping, the peformance results are HIGHLY non-linear. Paging is almost as bad, especially in systems with lots of RAM installed. What happens is the page table does not fit in the processor cache, so every TLB/page miss results in processor cache misses. Add in swap, and things go backwards, but in a way that is wickedly non-linear.

If you toss in things like Java/Groovy, or C#, which have their own garbage collection delays on top of the underlying system's page fault storms and swapping, there are more combinations of slowness. Run this all on a virtual platform like VMWare vSphere, and the situation is even harder to understand, except that it randomly slows down by a factor of at least a thousand.

One of the killer issues I have run into is forking in these sorts of situations. That typically requires a system-wide kernel sync of pages... which takes a long time on systems with lots of memory. Another related, persistent problem is the configuration of overcommit, another memory control parameter. For consistent performance on saturated systems this needs to be turned off.

On Windows, the use of scanners like McAffee is a constant source of problems. I wrote a Perl library that operated the command-line version of InstallShield. This was used to generate customized encrypted installers for my customers. Making custom installers means manipulating thousands of files thousands of times. McAffee pretty much halts this form of automation, slowing processing down by a factor of 2 million when it gets overwhelmed.

I have seen these non-linear behaviors quite a bit. The problems are always similar in nature.

[Markinsac]

Some of the processes used are out of date and not supported by the current hosting company. Some calls might time out when numerous attempts are made in quick succession. The current plan to move from a site hosted to a cloud hosted software package will ensure that the software will always be at a stable and supported level.

I know several posts lately have indicated that the vendor will be performing the upgrade soon. The admins will have to pause any posting and updates for the time that the vendor needs for the data moves.

As stated by others, the problems don't follow a specific pattern that we are able to see - most of the issues seem to be with background processes and calls that might be timing out.

[command_liner]

Well, wishing for and pushing for zstd might have been a bridge too far. But this topic is not going away.
As it happens the intermediate provider Cloudflare is finishing up the 2nd try at turning on Brotli compression, the compression not quite as nice as zstd.
There is some good background at https://blog.cloudflare.com/this-is-brotli-from-origin/

A quick check shows that this browser (recent Chrome) is telling calguns.net it can use modern encoding.
Accept-Encoding:gzip, deflate, br, zstd

The response from calguns.net is right out of the 1990s
Content-Encoding:gzip

How about a move into this century?

[command_liner]

You have some Brotli turned on now. Good.
Work harder to get rid of that gzip and gif stuff. Switch to zstd and webp.