Wikipedia:Reference desk/Archives/Computing/2015 July 5

Computing desk
< July 4	<< Jun | July | Aug >>	July 6 >

Welcome to the Wikipedia Computing Reference Desk Archives
The page you are currently viewing is an archive page. While you can leave answers for any questions shown below, please ask new questions on one of the current reference desk pages.

July 5

Do computers compress their own information?

Are only data as files and data as streams compressed? Or do computers also compress the information in their microchips and RAM modules?--YX-1000A (talk) 15:49, 5 July 2015 (UTC)

Some operating systems compress pages of RAM as they're written off to the swap (which can make sense because the rate at which CPUs have become faster is much greater than the rate at which disks have) - see Virtual memory compression. Unfortunately that article mostly talks about how it worked in the 1990s (when the CPU:disk speed ratio wasn't nearly as high as it is now); I can't find concrete numbers for how well modern solutions perform on modern hardware. -- Finlay McWalterᚠTalk 16:53, 5 July 2015 (UTC)

Nearly everything can be compressed! Files, display output, audio, even bulk data in RAM, can all be compressed. Modern computers use all of these tricks, and more!

Some modern hardware is able to compress data totally transparently, meaning the user never asks for a compression operation or even knows that one had been used! For example, here's a product brief for Intel 520 Solid State Drives. All data committed to nonvolatile storage is losslessly compressed! Intelligent software and firmware decides when and where to apply the compression, based on heuristics about file type and compressibility. Data can be compressed at block-level, file-level, and so on.

Intel computers compress the bits that get drawn to screen: here's a whitepaper on Frame Buffer Compression, which can be lossless or lossy. This reduces the bandwidth needed to transfer millions of pixels per second to a monitor, enabling cheaper monitor hardware to advertise higher supported resolutions.

Modern Intel computers that run Apple's OS X Mavericks operating system use virtual memory compression, which can take advantage of CPU- and memory-controllers that have hardware implementations of data compression technology. Data can be compressed at the granularity of memory pages or larger blocks. This means that each trip to a "physical address" is actually a pass through a compressor or decompressor - physical addresses are still virtual addresses, even in most parts of the kernel! This can improve performance and (in tandem with smart system control software) can also lower power. You can inspect a software-only implementation of vm_compressor, which manages when pages should be compressed; and with some imagination, you can see how to replace the CPU-intensive parts of the data compaction with a "free" call into a hardware accelerator (provided by your hardware vendor) that compacts the data for you. Here's a whitepaper by some smart engineers at Nokia who applied the same trick to compress cache data in Linux for ARM.

You can bet that there are even more hardware-optimized data compression paths hiding all over your modern computer. Peripheral I/O, image data, display hardware, audio input and output, network traffic, all probably get compressed, decompressed, re-re-recompressed, a few dozen times before the user application software ever gets to the "bits." This is one reason why it is so laughable to hear audiophiles or graphics professionals extolling the virtues of uncompressed audio or video! They have no idea how many millions of transistors are inside their digitally-controlled microphone or speaker, nor how many times their picture has been "color-corrected" or "de-noised," nor how many digital signal processor pathways have been transited, unless they have detailed schematics of every single piece of proprietary hardware! And because these kinds of hardware can be made from commodity transistor processes, built on microscopic technologies, companies throw these types of compression ASICs everywhere, in all kinds of data pathways. For some applications, the designers take great care to ensure bit-identical, lossless reconstruction - for example, bulk compression of virtual memory must usually ensure bit-perfect reconstruction; but in many many applications, bit-identical reconstruction is not actually required or implemented.

Nimur (talk) 19:47, 5 July 2015 (UTC)