Reddit mentions of Elements of Information Theory 2nd Edition (Wiley Series in Telecommunications and Signal Processing)

For canonical references, there is Thomas, Cover, Elements of Information Theory. For a shorter one, I like Probability and Information Theory chapter in the http://www.deeplearningbook.org/.

And if, for some crazy reason, you speak Polish, I wrote a short post Prawdopodobieństwo a informacja.

I would start with Cover & Thomas' book, read concurrently with a serious probability book such as Resnick's or Feller's.

I would also take a look at Mackay's book later as it ties notions from Information theory and Inference together.

At this point, you have a grad-student level understanding of the field. I'm not sure what to do to go beyond this level.

For crypto, you should definitely take a look at Goldreich's books:

Foundations Vol 1

Foundations Vol 2

Modern Crypto

This is a truly great answer.

To be clear though, your friend may or may not actually be correct. It comes down to what he said exactly. The amount of information per symbol is based entirely on the noise distribution, and not changed by bandwidth. If your friend said the capacity of the channel, instead of data rate, then yes he is correct. But if he used the exact words of data rate, then he is wrong. There is a lot going on here, so I will give you an overview.

There are two distinct parts that give us BW log2(1+S/N), the first being the maximum (symbols/second) which you can send symbols over a channel without incurring ISI, which is 2 BW, and the second being the channel capacity (bits/symbol) of a gaussian noise channel 1/2 log2(1+S/N). The channel capacity is not actually a function of the bandwidth. But channel capacity is something devoid of the concept of frequency.

The product of these two, giving us bits/second, is BWlog2 (1+S/N). This term is often confused with capacity, which it is not. This is the maximum capacity of the model given we wish to avoid ISI. Thus in practice, the actual bits/second very much does depend on bandwidth. In fact the relationship is linear.

Honestly, though, entering into a debate on this subject should be approached first by jumping into the subject of communications. If you really want to converse on this subject, you will first need to learn digital signal processing and information theory. Those are the two entry level texts in the field. Although you may be able to find them both by just google searching.

Thanks for the explanations! For a legal link to this text, here's Amazon's (US) page for Elements of Information Theory, Second Edition by Cover and Thomas.

>First, entropy is always positive, so you are indeed correct that those values should not be negative. Second, those values given are not relative entropy, but conditional entropy.

In the original paper to which OP linked, the last line reads

>Since [; H(C|X_2) > H(C|X_1), ;] the second component is more discriminative.

As I understand it, you're explaining that entropy is always positive, so these values were computed incorrectly with respect to sign. Accordingly, this would mean that [; H(C|X_1) \approx 0.97 > 0.72 \approx H(C|X_2), ;] instead, right? (Or does the FUBARness extend to these computations, too?) And in general, is the claim that [; H(C|X_i) > H(C|X_j) ;] implies that [; X_i ;] is the more discriminative component still true? (Or again, more FUBARness?)

Oh, and OP (/u/hupcapstudios)? This is a response from someone who actually understands this material. By contrast, I was just trying to apply formulas semi-blindly, aided by a little Googling.

I always like finding intuitive explanations to help grapple with the 'true' math. It's really hard to extract meaning sometimes from hard books, but at some point, the 'true' story and the kind of challenging practice that goes with it is something you still need. If you just want to see information theory from a high level, Kahn's Academy is probably a great place to start. But when you say 'deep learning research'... if you want to write an original white paper (or even read an information theoretic paper on deep learning) you'll need to wade deep into the weeds and actually get your hands dirty. If you do want to get a good foundation in information theory for machine learning, I went through the first few chapters so far of David MacKay's information theory book and that's been great so far, excited to go through it properly at some point soon. I've heard Cover and Thomas is considered more the 'bible' of the field for undergrad/early grad study, but it takes a more communication centric approach instead of a specific machine learning based approach.

Um... though reading your comment again, do you also not know probability theory and statistics? Wasserman's all of statistics is a good source for that, but you'll need a very high level of comfort with multivariate calculus and a reasonable level of comfort with proof based mathematics to be able to weather that book.

Why don't you start looking at the kinds of papers you'd be interested in? Some research is more computational than theoretical. You've got a very long road ahead of you to get a proper foundation for original theoretical research, but getting very clear on what exactly you want to do might help you narrow down what you want to study? You really, really can't do wrong with starting with stats though, even if you do want to focus on a more computer science/practical implementation direction.