Wikipedia Easy Search

Information theory

Entropy Differential entropy Conditional entropy Joint entropy Mutual information Directed information Conditional mutual information Relative entropy Entropy rate Limiting density of discrete points
Asymptotic equipartition property Rate–distortion theory
Shannon's source coding theorem Channel capacity Noisy-channel coding theorem Shannon–Hartley theorem
v t e

Venn diagram showing additive and subtractive relationships various information measures associated with correlated variables $X$ and $Y$ . The area contained by both circles is the joint entropy $\mathrm {H} (X,Y)$ . The circle on the left (red and violet) is the individual entropy $\mathrm {H} (X)$ , with the red being the conditional entropy $\mathrm {H} (X|Y)$ . The circle on the right (blue and violet) is $\mathrm {H} (Y)$ , with the blue being $\mathrm {H} (Y|X)$ . The violet is the mutual information $\operatorname {I} (X;Y)$ .

In information theory, the conditional entropy quantifies the amount of information needed to describe the outcome of a random variable $Y$ given that the value of another random variable $X$ is known. Here, information is measured in shannons, nats, or hartleys. The entropy of $Y$ conditioned on $X$ is written as $\mathrm {H} (Y|X)$ .