Entropy and Information

How to read this page: This article maps the topic from beginner to expert across six levels � Remembering, Understanding, Applying, Analyzing, Evaluating, and Creating. Scan the headings to see the full scope, then read from wherever your knowledge starts to feel uncertain. Learn more about how BloomWiki works ?

Entropy and Information are the foundational concepts of modern communication, discovered by Claude Shannon in 1948. While we often think of information as "Meaning," Shannon realized that information is actually about the reduction of "Uncertainty." Entropy is the measure of how much "Surprise" is in a message. The more unpredictable a message is, the more information it contains. This field is the "DNA of the Digital Age"—it tells us the absolute limits of how much we can compress a file, how fast we can send data over the internet, and how to keep secrets using cryptography.

Remembering

• Information Theory — The mathematical study of the quantification, storage, and communication of information.
• Claude Shannon — The "Father of Information Theory" who defined the mathematical bit.
• Entropy (H) — The measure of uncertainty or randomness in a set of data.
• Bit — The basic unit of information (a Binary Digit), representing a choice between two equally likely options (0 or 1).
• Redundancy — The fraction of a message that is unnecessary and can be removed without losing information.
• Source — The entity that generates a message (like a person or a computer).
• Channel — The medium used to send information (like a wire, air, or fiber-optic cable).
• Noise — Random interference that can corrupt a message during transmission.
• Surprise — The mathematical property where rare events provide more information than common ones.
• Binary Logarithm (log₂) — The mathematical tool used to calculate the number of bits needed to represent a choice.

Understanding

Information and entropy are understood through Uncertainty and Predictability.

1. Information as "Surprise": Shannon realized that if you already know what a message is going to say, it contains **zero** information.

• If I tell you "The sun will rise tomorrow," that is a 100% certain event. Surprise = 0, Information = 0 bits.
• If I flip a coin, there is a 50/50 chance. Surprise = 1, Information = 1 bit.
• Information is the "Resolution of Uncertainty."

2. Entropy (The Measure of Chaos): Entropy (H) calculates the average "Surprise" of a whole system.

• If a coin is weighted and always lands on Heads, its entropy is 0.
• If a coin is fair, its entropy is at its maximum (1 bit).
• In language, some letters (like 'E' in English) are very common. They have low entropy. Rare letters (like 'Z' or 'Q') have high entropy because they are "Surprising."

3. The Absolute Limit: Shannon proved that Entropy is the "Speed Limit" of communication.

• You can never compress a file smaller than its entropy without losing data.
• You can never send data over a noisy wire faster than the "Channel Capacity."

The Bit: Before Shannon, "Information" was a vague idea. He turned it into a physical substance—the bit—that could be measured, counted, and sold like electricity or water.

Applying

Modeling 'The Entropy Calculation' (Measuring the information in a choice): <syntaxhighlight lang="python"> import math

def calculate_entropy(probabilities):

   """
   H = -sum(p * log2(p))
   Measures the 'Uncertainty' in bits.
   """
   entropy = 0
   for p in probabilities:
       if p > 0:
           entropy -= p * math.log2(p)
   return round(entropy, 2)

1. Fair Coin (50/50)

print(f"Fair Coin Entropy: {calculate_entropy([0.5, 0.5])} bits")

1. Weighted Coin (90/10)

print(f"Weighted Coin Entropy: {calculate_entropy([0.9, 0.1])} bits")

1. Note: The weighted coin is more predictable, so it has LESS information.

</syntaxhighlight>

Information Landmarks

A Mathematical Theory of Communication (1948) → Shannon's paper that is considered the "Magna Carta" of the information age.

The Enigma Machine → While Shannon worked on theory, Alan Turing worked on "Information" in the context of codebreaking during WWII.

Deep Space Communication → NASA uses information theory to send photos from Pluto using a tiny, weak signal that is almost entirely covered by "Noise."

DNA as Code → Biologists realized that DNA is an information system with an entropy that can be measured just like a computer file.

Analyzing

Data vs. Information

Feature	Data	Information
Nature	Raw symbols (A, B, C...)	The resolution of uncertainty
Measure	Count (e.g., 1000 letters)	Entropy (e.g., 200 bits)
Goal	Storage	Communication
Redundancy	High (Many unnecessary parts)	Low (Pure surprise)

The Concept of "Redundancy": Analyzing why we can still read a sentence if we remove the vowels ("Ths s n nfrmtn thry rtcl"). English is about 50% redundant. Information theory allows us to "Squeeze out" this redundancy to make files smaller.

Evaluating

Evaluating information theory:

1. Meaning vs. Data: Does Shannon's theory miss the "Meaning" of a message? (Shannon himself said yes—his theory only cares about the *transmission* of symbols, not what they mean to humans).
2. Efficiency: How close are we to the "Shannon Limit" in our 5G and fiber-optic networks? (We are currently at about 99% of the theoretical maximum).
3. Physics: Is information a fundamental part of the universe, like matter and energy? (The "It from Bit" philosophy).
4. Chaos: If the universe is moving toward maximum entropy (The Heat Death), is it also moving toward "Maximum Information"?

Creating

Future Frontiers:

1. Semantic Communication: Designing new theories that don't just send "Bits," but send "Meanings" and "Concepts" to save even more bandwidth.
2. DNA Hard Drives: Using information theory to store the world's data in synthetic DNA, which can last for 10,000 years.
3. Neural Information Theory: Mapping how the human brain "Compresses" sensory data so we don't get overwhelmed.
4. Quantum Entropy: Developing the math of information for quantum states that can be "0 and 1 at the same time."