Cache performance measurement and metric

A CPU cache is a piece of hardware that reduces access time to data in memory by keeping some part of the frequently used data of the main memory in a 'cache' of smaller and faster memory.

The performance of a computer system depends on the performance of all individual units—which include execution units like integer, branch and floating point, I/O units, bus, caches and memory systems. The gap between processor speed and main memory speed has grown exponentially. Until 2001–05, CPU speed, as measured by clock frequency, grew annually by 55%, whereas memory speed only grew by 7%.^[1] This problem is known as the memory wall. The motivation for a cache and its hierarchy is to bridge this speed gap and overcome the memory wall.

The critical component in most high-performance computers is the cache. Since the cache exists to bridge the speed gap, its performance measurement and metrics are important in designing and choosing various parameters like cache size, associativity, replacement policy, etc. Cache performance depends on cache hits and cache misses, which are the factors that create constraints to system performance. Cache hits are the number of accesses to the cache that actually find that data in the cache, and cache misses are those accesses that don't find the block in the cache. These cache hits and misses contribute to the term average access time (AAT) also known as AMAT (average memory access time), which, as the name suggests, is the average time it takes to access the memory. This is one major metric for cache performance measurement, because this number becomes highly significant and critical as processor speed increases.

Another useful metric to test the performance is Power law of cache misses. It gives you the number of misses when you change the size of the cache, given that the number of misses for one of the cache sizes is known. Similarly, when you want to test the performance of the cache in terms of misses across different associativities, Stack distance profiling is used.