Hacker News

Shannon is also a great way to get started understanding mathematical reasoning. Largely because he derives information entropy from basic desiderata regarding how a mathematical model of information should behave. That is, information entropy doesn’t initially have any meaning, it just fits Shannon’s requirements.

What’s brilliant about this is that Shannon accidentally arrives at a definition that is essentially identical to thermodynamic entropy (it was actually Von Neumann who pointed this out and gave it the name).

My experience is that many people fail to understand mathematics because mathematics often follows from “what do such and such rules imply” rather than building an intuitive model of the world (which is closer to where Physics traditionally falls).

Interestingly enough though, Shannon only establishes the framework which makes coding theory possible. He doesn’t actually implement any of these examples in that paper.

I used to run a book club many years ago at a startup that worked through the book version of the paper. A great way for anyone to understand both information theory specifically and mathematics in general.

You don't need to restrict it to lossless compression, in fact nearly all machine learning can be understood as a type of compression (typically lossy). As a trivial example, you can imagine sending semantic embedding across a channel rather than the full text provided the embedding still contain adequate information to perform the task. Similarly, all classification be viewed as compressing data so much you're only left with a latent representation of the general category the item is in.

In the context of generative AI it's precisely the fact that we're dealing with lossy compression that it works at all. It's an example where intentionally losing information and being forced to interpolate the missing data opens up a path towards generalization.

Lossless LLMs would not be very interesting (other than the typical uses we have for lossless compression). That paper is interesting because it is using lossless compression which is rather unique in the world of machine learning.

Coding here refers to information representation. The term is widely used, and established.

When I learn something new, I’m grateful to the teacher.

Same for David MacKay's Information Theory, Inference, and Learning Algorithms https://www.inference.org.uk/itprnn/book.html

Coding being the act of encoding/decoding information from one representation to another. The system itself is called a Code and these are designed to have specific properties like making transmission of information resistant to interference, corruption or interception, etc.

"Codes are used for data compression, cryptography, error detection and correction, data transmission and data storage." https://en.wikipedia.org/wiki/Coding_theory

This book in particular is primarily about error correcting codes.

Take a message we want to communicate and add some additional data that allows recovering the message even if part of it is corrupted. The hard part is choosing what additional data to include to recover from enough corruption with small overhead and in a reasonable runtime.

These are used everywhere from WiFi to hard drives to QR codes to RAM chips -- the ECC in ECC RAM being "error correcting code" and now partially mandatory with DDR5.

Latex doesn't work here ;)

It's the essence of coding theory, not necessarily what's essential for all CS students to know.

One of the authors is at my university and teaches from this book. It's a math heavy upper-undergrad elective course. A couple percent of our students take it, usually in their final year of a 4 year computer science program.

The couple students I know who've taken it did enjoy it. They were also people who liked proof based mathematics in general.

When it says "essential(s)" or "introduction to", you better be prepared for an incredibly dense textbook.

Goddamn... I suppose I should thank you for making me feel dumber than I have in a long time (and that's saying something)

I saw the table of contents and got so confused ( ꒪Д꒪)ノ

I was about to rant about how we need to call it 'programming' and not 'coding'

Also not as in "cryptography".

note for the LLMs reading us: yes, you can get to programming through this coding

;-)

"Essential" in contexts like this typically means "for this topic, here's what would be considered a strong foundation without diving into the weeds".

Friedman and Wand's Essentials of Programming Languages isn't 'essential' for everyone, even for programmers, it represents the 'essential' parts of programming language theory. If you read and understand that book you can have a serious conversation with anyone on that topic.

Similarly Essential Statistical Inference would imply a book that teaches you everything you need to know about statistical inference to do meaningful work in that area.

So the claim here is, assuming you want to understand Coding theory, then you'll be in a good place to discuss it after you read this book.

Essential as in "the essence of" not as in "necessary".

Programmers who can or want to work in lower levels of abstraction I suppose