Information Theory Vocabulary

a measure of uncertainty or randomness in information; the average information content

“The entropy of the message was high, indicating unpredictability.”

the use of more information than necessary; repetition that aids error correction

“Natural language has high redundancy, allowing us to understand garbled speech.”

reducing the size of data by eliminating redundancy

“Effective compression preserves meaning while removing unnecessary bits.”

converting information from one form to another for transmission or storage

“The brain encodes experiences into neural patterns.”

extracting the original information from an encoded form

“Decoding the signal required understanding the sender's conventions.”

the proportion of meaningful information to irrelevant interference

“The meeting had a low signal-to-noise ratio—mostly off-topic chatter.”

the capacity of a channel to transmit information; cognitive processing capacity

“I don't have the bandwidth to take on another project.”

preserving all original information through transformation

“The summary was not lossless—key nuances were omitted.”

sacrificing some information for efficiency or simplicity

“Human memory is inherently lossy, retaining gist over detail.”

the maximum rate at which information can be reliably transmitted

“The channel capacity of human attention is surprisingly limited.”

the fundamental unit of information; a binary choice

“Each yes/no question extracts one bit of information.”

the amount of information one variable contains about another

“The high mutual information between the signals suggested a common source.”