Information Theory Vocabulary

a measure of uncertainty or randomness in information; the average information content

“The entropy of the message was high, indicating unpredictability.”

Origin: Greek entropia (transformation); adopted by Shannon from thermodynamics

the use of more information than necessary; repetition that aids error correction

“Natural language has high redundancy, allowing us to understand garbled speech.”

Origin: Latin redundare (to overflow)

reducing the size of data by eliminating redundancy

“Effective compression preserves meaning while removing unnecessary bits.”

Origin: Latin comprimere (to press together)

converting information from one form to another for transmission or storage

“The brain encodes experiences into neural patterns.”

Origin: Greek en- (in) + Latin codex (book, system)

extracting the original information from an encoded form

“Decoding the signal required understanding the sender's conventions.”

Origin: Latin de- (reverse) + codex

the proportion of meaningful information to irrelevant interference

“The meeting had a low signal-to-noise ratio—mostly off-topic chatter.”

Origin: Technical term from electrical engineering

the capacity of a channel to transmit information; cognitive processing capacity

“I don't have the bandwidth to take on another project.”

Origin: From radio frequency band + width

preserving all original information through transformation

“The summary was not lossless—key nuances were omitted.”

Origin: English loss + -less (without)

sacrificing some information for efficiency or simplicity

“Human memory is inherently lossy, retaining gist over detail.”

Origin: English loss + -y (characterized by)

the maximum rate at which information can be reliably transmitted

“The channel capacity of human attention is surprisingly limited.”

Origin: Technical term from Shannon's 1948 paper

the fundamental unit of information; a binary choice

“Each yes/no question extracts one bit of information.”

Origin: Portmanteau of 'binary digit' coined by John Tukey

the amount of information one variable contains about another

“The high mutual information between the signals suggested a common source.”

Origin: Technical term from information theory