Information Theory, Inference and Learning Algorithms

I work in that area :) The skills needed will vary a lot. Hence my concern about studying H&H. It's a good idea, as real analysis is the foundation. But it will take too much of your time to get to something useful. Probably you should try to learn more applied material in parallel and let both threads merge in the future. For maximum likelihood, you need to learn convex optimization right after real analysis. The canonical reference is [1], but there's also a very simple and pragmatic linear algebra textbook by the same author that also covers some of the optimization basics [2]. This might be a good entry point, certainly easier than Spivak or H&H. There's also [3,4], which you probably know about. These are great and emphasize the modeling part. Maximum likelihood (via EM) is in the appendix, and you don't really need to know a lot of math to get going.…

Some books from a few different domains which may be of the type OP is asking for: 1. The Trivium: The Liberal Arts of Logic, Grammar, and Rhetoric : Understanding the Nature and Function of Language by Sister Miriam Joseph. 2. Introduction to Logic by Irving Copi. 3. Computer Systems: A Programmer's Perspective by Randall Bryant and others. 5. Prolog Programming by Ivan Bratko. 6. Information Theory, Inference and Learning Algorithms by David MacKay 7. Geometry by Brennan. 8. Understanding Poetry by Robert Penn Warren and Cleanth Brookes

You can find video of MacKay's lectures covering parts of the book online. I'm more of a reader but I really enjoyed them. And the book is excellent.

I second this as well! ITILA is not just outstanding but offers a timeless perspective to statistical and uncertainty modeling, not to mention information theory which is often understudied in this field (including by myself). As the parent poster says, this field moves fast but this book will give a solid grounding. Even though the treatment on neural networks is short, the beginning chapters are worthwhile. The chapter one random variables and probability is one of the best introductions to probabilistic modeling which I’ve seen.

second the Mackay book recommendation! It is a lovely text with a ton of examples, good reference for a senior course on applied probability.

It is a little old given how quickly the field moves, but 'Information Theory, Inference and Learning Algorithms' has a chapter on neural networks. It is an outstanding book: a labour of love from a very smart person. The exercises are varied, explanations are great, there's a sprinkling of humour, and connections drawn between multiple fields of study. Moreover, it is freely available from the author's website: http://www.inference.org.uk/itprnn/book.html Have you considered giving Goodfellow another shot, but trying to re-derive the results therein as a form of exercise? I think that would likely be one of the faster methods to bring yourself reasonably up to date with the field.

The Law - Frédéric Bastiat. http://bastiat.org/en/the_law.html Capitalism - A Treatise on Economics - George Reisman. http://www.capitalism.net/Capitalism/CAPITALISM_Internet.pdf The Works of Lysander Spooner. http://lysanderspooner.org/node/2 The Machinery of Freedom David Friedman. - http://daviddfriedman.com/The_Machinery_of_Freedom_.pdf Democratizing Innovation - Eric Von Hippel. http://web.mit.edu/evhippel/www/democ1.htm The Sources of Innovation - Eric Von Hippel. http://web.mit.edu/evhippel/www/sources.htm Introduction to Information Retrieval - http://nlp.stanford.edu/IR-book/html/htmledition/irbook.html Information Theory, Inference, and Learning Algorithms - David MacKay. http://www.inference.phy.cam.ac.uk/itprnn/book.html Introduction to Cybernetics - William Ross Ashby. http://pespmc1.vub.ac.be/books/IntroCyb.pdf "Man-Computer Symbiosis" and "The Computer As A Communication Device" - J.C.R. Licklider. http://memex.org/licklider.pdf Design For A Brain - William Ross Ashby. https://archive.org/details/designforbrainor00ashb The Writings of Douglas Engelbart. http://www.dougengelbart.org/library/library.html…

David Mackay's Information Theory, Inference and Learning algorithms doesn't cover everything, but it ties up a lot of important topics. http://www.inference.phy.cam.ac.uk/mackay/itila/

i second the David MacKay recommendation. its a real pleasure to read. for me personally, his coverage of the fundamentals of information theory was amazing. the proofs are easy to follow and explained well so you can match the intuition to formula. i'd also like to recommend Boyd & Vanderberghe's book on Convex Optimization (available here http://www.stanford.edu/~boyd/cvxbook/). This is maybe a little less fun but very deep and still well written and 'illustrated'. My background is in ML and I came to realize over time that a lot of ML is just statistics and optimization theory rehashed by computer scientist.

Hi - I was in a similar situation a year ago (starting a Master's in ML, feeling crappy at math). What worked for me was: - Studying is doing problems. Reading is not studying. When studying, read the minimum amount necessary to complete the problem. Do all problems CLOSED-book: no notes or reading while doing problems. - A problem is not understood until you can do it on a blank piece of paper two days after reading the relevant material (so do problems several times to check). - Study machine learning problems that use the math. The examples in math text books are pretty useless to you - do ML problems directly. When you don't get it, look the math up quickly in a reference book (A.E.M. by Kreyszig works), as much as needed to complete the problem, and move on. - Implement regularly. Once you've done the problems concerning an algorithm or technique, implement it. Make sure your implementation is correct, and spend some effort making it quick. Read source code for other people's implementations. This method is effective (for me) because I like programming and machine learning, I don't like math textbooks or the problems within, and I can't remember what I haven't solved.…

You could read Gilbert Strang's Linear Algebra books, there are also video lectures available... Gelman's Bayesian Data Analysis is a good statistical intro with a Bayesian bend, maybe Freedman's Statistical models, read some general machine learning intro book, such as Alpaydin or Bishop for a more probabilistic view. For NLP especially, you'll need some information theory. Maybe start with MacKay's Information Theory, Inference, etc. (it is freely available online), very entertaining and accessible book and you can follow the pointers there for more details. For general data mining and machine learning you need some basics of optimization, pick up any standard book on convex optimization to get an idea of the basic problems and methods. that's it at a first try :)

I'll second this recommendation - I bought the printed copy, and I'm constantly going to it for reference. The fact that it's available for free is just an added bonus. I would also suggest Elements of Statistical Learning: http://www-stat.stanford.edu/~tibs/ElemStatLearn/ As well as Duda, Hart, and Stork's Pattern Classification: http://rii.ricoh.com/~stork/DHS.html

Information Theory, Inference and Learning Algorithms by David MacKay is excellent, and freely available online. http://www.inference.phy.cam.ac.uk/mackay/itila/book.html