The amount of entropy should be the same independently of how the process is regarded as being divided into parts. In this case, the entropy increases with the number of outcomes. If all the outcomes are equally likely, then entropy should be maximal (uncertainty is highest when all possible events are equiprobable). The measure should be unchanged if the outcomes x i are re-ordered. The measure should be continuous - i.e., changing the value of one of the probabilities by a very small amount should only change the entropy by a small amount. Information entropy is characterised by these desiderata: P( x i) = Pr( X= x i) is the probability mass function of X.

I( X) is the information content or self-information of X, which is itself a random variable and The information entropy of a discrete random variable X, that can take on possible values is Shannon in his 1948 paper 'A Mathematical Theory of Communication'. Įquivalently, the Shannon entropy is a measure of the average information content the recipient is missing when he does not know the value of the random variable. The entropy of English text is between 1.0 and 1.5 bits per letter. A long string of repeating characters has an entropy of 0, since every character is predictable. However, if the coin is not fair, then the uncertainty is lower (if asked to bet on the next outcome, we would bet preferentially on the most frequent result), and thus the Shannon entropy is lower. This also represents an absolute limit on the best possible lossless compression of any communication: treating a message as a series of symbols, the shortest number of bits necessary to transmit the message is the Shannon entropy in bits/symbol multiplied by the number of symbols in the original message.Ī fair coin has an entropy of one bit.

It is the minimum message length necessary to communicate information. It quantifies the information contained in a message, usually in bits or bits/symbol. In information theory, the Shannon entropy or information entropy is a measure of the uncertainty associated with a random variable. 7 Extending discrete entropy to the continuous case: differential entropy.5.4 Limitations of entropy as information content.5.1 Relationship to thermodynamic entropy.